Bypassing Directory Ownership Check in PostgreSQL 16.6 with Secure z/OS NFS (AT-TLS)

On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:

I'm currently running PostgreSQL version 16.6 inside a Docker container
(base image: UBI 9), using Docker Compose. The PostgreSQL data directory
is mounted from an NFS volume hosted on a z/OS NFS server.

The environment has a few constraints:

- It’s a highly secure and access-controlled setup.
- Due to platform restrictions on z/OS, the mounted NFS directory cannot
  be owned by the PostgreSQL user (e.g., `postgres`) inside the container.
- As a result, PostgreSQL fails to start because of the directory
  ownership validation check.

It is not a good idea to have a mount point be the data directory.
The proper solution is to create the data directory inside the
mount point. That way, the permissions of the data directory don't
have to be the same as the permissions of the mount point.

Yours,
Laurenz Albe

Hi Laurenz

Thanks for the reply,

If I am not mistaken, below is my understanding of your suggestion.

Suppose that My mount point on the NFS server is say /nfs-mount/postgres/
and you are suggesting to have a data directory as say
/nfs-mount/postgres/db or something like that ?
and assign this value to the PGDATA ?

If that is the case, then when and who should be creating the directory DB
?

Please correct me if I am wrong about the understanding.

Thanks,
Amol

On Mon, Jul 14, 2025 at 5:50 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:

On Mon, 2025-07-14 at 11:19 +0530, Amol Inamdar wrote:

I'm currently running PostgreSQL version 16.6 inside a Docker container
(base image: UBI 9), using Docker Compose. The PostgreSQL data directory
is mounted from an NFS volume hosted on a z/OS NFS server.

The environment has a few constraints:

- It’s a highly secure and access-controlled setup.
- Due to platform restrictions on z/OS, the mounted NFS directory cannot
be owned by the PostgreSQL user (e.g., `postgres`) inside the

container.

- As a result, PostgreSQL fails to start because of the directory
ownership validation check.

It is not a good idea to have a mount point be the data directory.
The proper solution is to create the data directory inside the
mount point. That way, the permissions of the data directory don't
have to be the same as the permissions of the mount point.

Yours,
Laurenz Albe

--
-regards
Amol

On Mon, 2025-07-14 at 17:59 +0530, Amol Inamdar wrote:

If I am not mistaken, below is my understanding of your suggestion. 

Suppose that My mount point on the NFS server is say /nfs-mount/postgres/ 
and you are suggesting to have a data directory as say /nfs-mount/postgres/db or something like that ? 
and assign this value to the PGDATA ? 

If that is the case, then when and who should be creating the directory DB ? 

Please correct me if I am wrong about the understanding.

You understood me perfectly well.

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

Alternatively, the root user could create the data directory with the
correct ownership and permissions prior to running "initdb".

Yours,
Laurenz Albe

Thanks Laurenz,

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

This exactly is the problem we are facing, to give you a summary,
our NFS server is enabled with AT-TLS authentication
and we are accessing the server via a proxy server (Haproxy).
This acts as our NFS client and it is configured with the
required client certificates.

The outcome of above configuration is that any directory created
in the NFS mount is always owned by the user in the certificates
and if that user isn't present in the proxy container it is marked
as nobody:nogroup, we tried various things like
created the user similar to postgres user so that the users ids match but
always ended up giving error “data directory “/var/lib” has wrong ownership

Hence, we thought of skipping this check (Directory owner and postgres user
validation) and
wanted to understand the implication of the same.

Thanks,
Amol,

On Mon, Jul 14, 2025 at 6:14 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:

On Mon, 2025-07-14 at 17:59 +0530, Amol Inamdar wrote:

If I am not mistaken, below is my understanding of your suggestion.

Suppose that My mount point on the NFS server is say

/nfs-mount/postgres/

and you are suggesting to have a data directory as say

/nfs-mount/postgres/db or something like that ?

and assign this value to the PGDATA ?

If that is the case, then when and who should be creating the directory

DB ?

Please correct me if I am wrong about the understanding.

You understood me perfectly well.

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

Alternatively, the root user could create the data directory with the
correct ownership and permissions prior to running "initdb".

Yours,
Laurenz Albe

--
-regards
Amol

Laurenz Albe <laurenz.albe@cybertec.at> writes:

It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:

/messages/by-id/41BFAB7C.5040108@joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check. It is there
for very good reasons.

regards, tom lane

On Mon, 2025-07-14 at 18:32 +0530, Amol Inamdar wrote:

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly safe.

This exactly is the problem we are facing, to give you a summary, 
our NFS server is enabled with AT-TLS authentication
and we are accessing the server via a proxy server (Haproxy). 
This acts as our NFS client and it is configured with the 
required client certificates.

The outcome of above configuration is that any directory created 
in the NFS mount is always owned by the user in the certificates 
and if that user isn't present in the proxy container it is marked 
as nobody:nogroup, we tried various things like
created the user similar to postgres user so that the users ids match but 
always ended up giving error  “data directory “/var/lib” has wrong ownership 

Hence, we thought of skipping this check (Directory owner and postgres user validation) and 
wanted to understand the implication of the same.

No; don't.

Simply mount the directory once, create a subdirectory with the
appropriate ownership and permissions, and there you go.
Problem solved.

Yours,
Laurenz Albe

On 2025-07-14 10:07:20 -0400, Tom Lane wrote:

Laurenz Albe <laurenz.albe@cybertec.at> writes:

It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:

/messages/by-id/41BFAB7C.5040108@joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Be careful: There are two different directorys involved in a mount
point. The one in the parent filesystem and the one in the mounted file
system.

E.g.:

# mkdir /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,0 Inode: 317 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

So here we have inode 317 on device 254,0 and unsurprisingly it belongs
to root and is only writable by root.

Now let's mount a filesystem:

# mount /dev/vgroot/demo /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,13 Inode: 2 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

This is now inode 2 on device 254,13 (i.e. the root directory on
/dev/vgroot/demo). Since I just created that it also belongs to root.
Let's change that:

# chown postgres: /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,13 Inode: 2 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 108/postgres) Gid: ( 114/postgres)

Ok, now it belongs to postgres and postgres could create files,
subdirectories, etc.

But when I unmount it:

# umount /mnt/demo
# stat /mnt/demo
File: /mnt/demo
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 254,0 Inode: 317 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Lo and behold: The original Inode is back and unchanged.

The user postgres couldn't write to that directory if it tried to.

That said, there are of course some remaining failure modes:

1) An admin might notice the wrong permissions and "fix" them without
investigating the cause
2) Some automated procedure (running as root) might "fix" the
permissions at startup (like the initdb in the example) or during an
upgrade.
3) use your imagination ;-).

hjp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

"Peter J. Holzer" <hjp-pgsql@hjp.at> writes:

On 2025-07-14 10:07:20 -0400, Tom Lane wrote:

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory.

Be careful: There are two different directorys involved in a mount
point. The one in the parent filesystem and the one in the mounted file
system.

True, and the safety requirement really is only that the parent
filesystem's mount-point directory not be writable by us.
But normal practice is that both directories are root-owned,
or at least owned by highly privileged users.

(I have a vague idea that there are system-level security hazards,
not specific to Postgres, if mount-point directories are publicly
writable. Don't feel like researching that though.)

regards, tom lane

I am not sure whether PostgreSQL depends on system call `fsyncdata` to
sync data to disk. If yes, then I don't think it's safe to use NFS.
When `fsyncdata` returns success, it doesn't mean the data has
really been synced to disk. But if PostgreSQL crashes right after
it returns success to clients. Eventually it breaks the D (Durability) of
ACID.

Benjamin

On Mon, Jul 14, 2025 at 7:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

On Mon, Jul 14, 2025 at 12:02 PM Benjamin Wang <benjamin.ahrtr@gmail.com>
wrote:

I am not sure whether PostgreSQL depends on system call `fsyncdata` to
sync data to disk.

https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-SYNC-METHOD

David J.

Benjamin Wang <benjamin.ahrtr@gmail.com> writes:

I am not sure whether PostgreSQL depends on system call `fsyncdata` to
sync data to disk. If yes, then I don't think it's safe to use NFS.

Well, that's a whole other discussion. The point about mount
directories applies to any sort of dismountable storage.

(But I too would not use Postgres-over-NFS for any critical data.
Too many moving parts. It's tough enough to ensure crash safety
with local storage.)

regards, tom lane

On Mon, 2025-07-14 at 14:30 -0400, Tom Lane wrote:

(I have a vague idea that there are system-level security hazards,
not specific to Postgres, if mount-point directories are publicly
writable.  Don't feel like researching that though.)

Well, if you are using an ext? file system, there is a lost+found
directory where fsck places links to orphaned inodes.
If the PostgreSQL user owns the mount point and wants to use
"initdb" to create a data directory in it, the program will fail
and complain that the directory is not empty. The danger is great
that the user removes the lost+found directory to solve the problem.

True, one could re-create it with "mklost+found", but if a DBA
is uneducated enough to remove the directory in the first place,
the risk is high that he wouldn't think of creating it again,
which is a problem if the file system ever becomes corrupted.

All this doesn't apply to NFS, but it is yet another reason
(apart from the safety of a subdirectory that doesn't exist
on the file system underlying the mount point) why we should
continue to recommend that the data directory be not a mount point.

Yours,
Laurenz Albe

Thanks Laurenz.

On Mon, Jul 14, 2025 at 8:11 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:

On Mon, 2025-07-14 at 18:32 +0530, Amol Inamdar wrote:

The data directory can either be created by "initdb", in which case
the mount point must allow the PostgreSQL user to create a directory.
You could set the group of the mount point to the group of the
PostgreSQL user and use permissions 1770, which should be perfectly

safe.

This exactly is the problem we are facing, to give you a summary,
our NFS server is enabled with AT-TLS authentication
and we are accessing the server via a proxy server (Haproxy).
This acts as our NFS client and it is configured with the
required client certificates.

The outcome of above configuration is that any directory created
in the NFS mount is always owned by the user in the certificates
and if that user isn't present in the proxy container it is marked
as nobody:nogroup, we tried various things like
created the user similar to postgres user so that the users ids match

but

always ended up giving error “data directory “/var/lib” has wrong

ownership

Hence, we thought of skipping this check (Directory owner and postgres

user validation) and

wanted to understand the implication of the same.

No; don't.

Simply mount the directory once, create a subdirectory with the
appropriate ownership and permissions, and there you go.
Problem solved.

Yours,
Laurenz Albe

--
-regards
Amol

Thanks Tom and Laurenz for the explanation.
Let me try out a few things and get back to you if needed.

Thanks,
Amol

On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Laurenz Albe <laurenz.albe@cybertec.at> writes:

It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:

/messages/by-id/41BFAB7C.5040108@joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check. It is there
for very good reasons.

regards, tom lane

--
-regards
Amol

Hi All,

I would like to rephrase the question a little bit, below is how our setup
going to be

1. NFS mount point is for /nfs-mount/postgres (and permissions locked
down so that Postgres cannot create directories in here)
2. Postgres data directory is /nfs-mount/postgres/db
3.

With secured NFS + AT-TLS setup Postgres will be able to write to data
directory but not parent dir, however the file ownership information
Postgres sees from the stat() call will not match the Postgres user in the
container (even though the AT-TLS strict access control will ensure only
the Posgres user can read/write to this directory)

Considering the above scenario/setup, what is the danger of removing the
ownership check in miscinit.c checkDataDir() function ?

On Tue, Jul 15, 2025 at 5:06 PM Amol Inamdar <amol.aai@gmail.com> wrote:

Thanks Tom and Laurenz for the explanation.
Let me try out a few things and get back to you if needed.

Thanks,
Amol

On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Laurenz Albe <laurenz.albe@cybertec.at> writes:

It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:

/messages/by-id/41BFAB7C.5040108@joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check. It is there
for very good reasons.

regards, tom lane

--
-regards
Amol

--
-regards
Amol

Quoting Tom's earlier email:
"(But I too *would not use Postgres-over-NFS for any critical data*.
Too many moving parts. It's tough enough to ensure crash safety
with local storage.)"

You're going through a lot of security effort to implement a Worst Practice.

On Wed, Jul 16, 2025 at 9:25 AM Amol Inamdar <amol.aai@gmail.com> wrote:

Hi All,

I would like to rephrase the question a little bit, below is how our setup
going to be

1. NFS mount point is for /nfs-mount/postgres (and permissions locked
down so that Postgres cannot create directories in here)
2. Postgres data directory is /nfs-mount/postgres/db
3.

With secured NFS + AT-TLS setup Postgres will be able to write to data
directory but not parent dir, however the file ownership information
Postgres sees from the stat() call will not match the Postgres user in the
container (even though the AT-TLS strict access control will ensure only
the Posgres user can read/write to this directory)

Considering the above scenario/setup, what is the danger of removing the
ownership check in miscinit.c checkDataDir() function ?

On Tue, Jul 15, 2025 at 5:06 PM Amol Inamdar <amol.aai@gmail.com> wrote:

Thanks Tom and Laurenz for the explanation.
Let me try out a few things and get back to you if needed.

Thanks,
Amol

On Mon, Jul 14, 2025 at 7:37 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Laurenz Albe <laurenz.albe@cybertec.at> writes:

It is not a good idea to have a mount point be the data directory.

^^^ This. ^^^

That is primarily for safety reasons: if for some reason the
filesystem gets dismounted, or hasn't come on-line yet during
a reboot, you do not want Postgres to be able to write on the
underlying mount-point directory. There is a sobering tale
in this old thread:

/messages/by-id/41BFAB7C.5040108@joeconway.com

Now it didn't help any that they were using a start script that
would automatically run initdb if it didn't see a data directory
where expected. But even without that, you are in for a world of
hurt if the mount drops while the server is running and the server
has any ability to write on the underlying storage; it will think
whatever it was able to write is safely down on disk. To prevent
that, the server must not have write permissions on the mount
point, which dictates making a separate data directory (with
different ownership/permissions) just below the mount.

Do not bypass that ownership/permissions check. It is there
for very good reasons.

regards, tom lane

--
-regards
Amol

--
-regards
Amol

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

On Wed, 2025-07-16 at 18:54 +0530, Amol Inamdar wrote:

I would like to rephrase the question a little bit, below is how our setup going to be 
   1. NFS mount point is for /nfs-mount/postgres (and permissions locked down so
that Postgres cannot create directories in here)
   2. Postgres data directory is /nfs-mount/postgres/db
   3. With secured NFS + AT-TLS setup Postgres will be able to write to data directory
but not parent dir, however the file ownership information Postgres sees from the
 stat() call will not match the Postgres user in the container (even though the
AT-TLS strict access control will ensure only the Posgres user can read/write to
this directory)
Considering the above scenario/setup, what is the danger of removing the ownership check
in miscinit.c checkDataDir() function ? 

The danger is that somebody else than the PostgreSQL user has permissions on
the data directory. You will argue that that somebody is root, and root has
these permissions anyway.

But there is another reason why PostgreSQL insists that the PostgreSQL user
owns the data directory: at startup, the postmaster checks if the data
directory belongs to the current user and fails if not. This is a protection
against starting the postmaster with the wrong user.

There are certainly ways to do it differently, but I'd argue that they would
be more complicated, and the current simple solution is robust.

If you pre-create the data directory with the appropriate permissions,
what keeps you from giving ownership to the correct user too?

Yours,
Laurenz Albe

On Wed, Jul 16, 2025 at 9:25 AM Amol Inamdar <amol.aai@gmail.com> wrote:

1. NFS mount point is for /nfs-mount/postgres (and permissions locked
down so that Postgres cannot create directories in here)
2. Postgres data directory is /nfs-mount/postgres/db
3.

With secured NFS + AT-TLS setup Postgres will be able to write to data
directory but not parent dir, however the file ownership information
Postgres sees from the stat() call will not match the Postgres user in the
container (even though the AT-TLS strict access control will ensure only
the Posgres user can read/write to this directory)

This thread is fascinating. It's like combining two of the most annoying

technologies in the world, NFS and SELinux, into something worse than
either of them.

Many people use Docker, and NFS, and Postgres all the time. Stop trying to
push on a string. Conform your process to Postgres' fairly minimal and
sane requirements, rather than the other way around.

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

@Laurenz Albe <laurenz.albe@cybertec.at>
If you pre-create the data directory with the appropriate permissions,
what keeps you from giving ownership to the correct user too?

Our NFS server is not a regular linux based server,
it's on zOS (Mainframes) with AT-TLS security enabled,
hence it doesn't allow changing of ownership.

Basically, we have tried everything we could
to change the directory ownership to match with the postgres user
and that as of now looks impossible, unless we make changes in the
environment.

To summarize*, we are not able to change the ownership of the data
directory *
*due to the Mainframe NFS server limitations when enabled with AT-TLS
security *
*Hence we wanted to check if bypassing this check is ok if it could be
assured *
*that only the postgres user can write here (NFS-AT-TLS ensures that). *

I wouldn't get into details of explaining why changing ownership is not
possible,
as that would take this discussion to another context, hence avoiding.

Thanks in advance

On Wed, Jul 16, 2025 at 9:18 PM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:

On Wed, 2025-07-16 at 18:54 +0530, Amol Inamdar wrote:

I would like to rephrase the question a little bit, below is how our

setup going to be

1. NFS mount point is for /nfs-mount/postgres (and permissions locked

down so

that Postgres cannot create directories in here)
2. Postgres data directory is /nfs-mount/postgres/db
3. With secured NFS + AT-TLS setup Postgres will be able to write to

data directory

but not parent dir, however the file ownership information

Postgres sees from the

stat() call will not match the Postgres user in the container

(even though the

AT-TLS strict access control will ensure only the Posgres user can

read/write to

this directory)
Considering the above scenario/setup, what is the danger of removing the

ownership check

in miscinit.c checkDataDir() function ?

The danger is that somebody else than the PostgreSQL user has permissions
on
the data directory. You will argue that that somebody is root, and root
has
these permissions anyway.

But there is another reason why PostgreSQL insists that the PostgreSQL user
owns the data directory: at startup, the postmaster checks if the data
directory belongs to the current user and fails if not. This is a
protection
against starting the postmaster with the wrong user.

There are certainly ways to do it differently, but I'd argue that they
would
be more complicated, and the current simple solution is robust.

If you pre-create the data directory with the appropriate permissions,
what keeps you from giving ownership to the correct user too?

Yours,
Laurenz Albe

--
-regards
Amol