BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

Started by PG Bug reporting formalmost 3 years ago6 messagesbugs
Jump to latest
#1PG Bug reporting form
noreply@postgresql.org

The following bug has been logged on the website:

Bug reference: 17938
Logged by: Onkobu Tanaake
Email address: onkobu@onkobutanaake.de
PostgreSQL version: 15.3
Operating system: Gentoo
Description:

After upgrade from 14 to 15 from time to time this error occurs. A snapshot
of a responsive database looks like this:

$> ls -la /dev/shm/PostgreSQL.*
-rw------- 1 postgres postgres 1048576 May 22 19:18
/dev/shm/PostgreSQL.2644390168
-rw------- 1 postgres postgres 26176 May 22 19:14
/dev/shm/PostgreSQL.3338605124

$> lsof | grep PostgreSQL
postgres 16944 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16945 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16946 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16948 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16949 postgres mem REG 0,49
1048576
3 /dev/shm/PostgreSQL.2644390168
postgres 16949 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16950 postgres mem REG 0,49
1048576
3 /dev/shm/PostgreSQL.2644390168
postgres 16950 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124
postgres 16958 postgres mem REG 0,49
1048576
3 /dev/shm/PostgreSQL.2644390168
postgres 16958 postgres mem REG 0,49
26176
2 /dev/shm/PostgreSQL.3338605124

$> df -h
Filesystem Size Used Avail Use% Mounted on
[…]
shm 32G 1.1M 32G 1% /dev/shm
[…]

Config (postgresql.conf) is as follows:

max_wal_size = 2GB
min_wal_size = 1GB
effective_cache_size = 8GB
log_timezone = 'Europe/Berlin'
datestyle = 'iso, mdy'
timezone = 'Europe/Berlin'
default_text_search_config = 'pg_catalog.english'
plperl.on_init = 'use utf8; use re; package utf8; require
"utf8_heavy.pl";'

I don't use shared_memory_type and also not dynamic_shared_memory_type so
they're default (mmap for the first, posix for the second). Got the settings
from https://pgtune.leopard.in.ua/ for a conservative use on a 32GByte 8
core/ 16 threads machine.

Building goes along the lines of an ebuild
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-db/postgresql/postgresql-15.3.ebuild#n117.
I enabled icu, kerberos, nls, pam, ssl, threads, uuid, xml and zlib.

The server does not run any systemd but is pure Init-V. So it is very
unlikely that a systemd process causes this.Also no container in use. Init
process directly runs postgres server. Kernel is 5.15.80 and was the same
for version 14. I also don't log in with the postgres-user which is also
impossible (disabled in /etc/shadow). WIthin 7 days this happens very
likely/ at least once. How can I track this down further?

#2Michael Paquier
michael@paquier.xyz
In reply to: PG Bug reporting form (#1)
Re: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

On Mon, May 22, 2023 at 06:10:43PM +0000, PG Bug reporting form wrote:

The server does not run any systemd but is pure Init-V. So it is very
unlikely that a systemd process causes this.Also no container in use. Init
process directly runs postgres server. Kernel is 5.15.80 and was the same
for version 14. I also don't log in with the postgres-user which is also
impossible (disabled in /etc/shadow). WIthin 7 days this happens very
likely/ at least once. How can I track this down further?>

Are you sure that your system does not run systemd and enable
RemoveIPC? Even if the PostgreSQL service is not running with it, you
would be able to see see issue as long as the parameter is enabled:
/messages/by-id/BACA8550-4F3D-41D0-A0E1-5BF307BCD648@elevated-dev.com
--
Michael

#3Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Michael Paquier (#2)
Re: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

At Tue, 23 May 2023 07:43:11 +0900, Michael Paquier <michael@paquier.xyz> wrote in

On Mon, May 22, 2023 at 06:10:43PM +0000, PG Bug reporting form wrote:

The server does not run any systemd but is pure Init-V. So it is very
unlikely that a systemd process causes this.Also no container in use. Init
process directly runs postgres server. Kernel is 5.15.80 and was the same
for version 14. I also don't log in with the postgres-user which is also
impossible (disabled in /etc/shadow). WIthin 7 days this happens very
likely/ at least once. How can I track this down further?>

Are you sure that your system does not run systemd and enable
RemoveIPC? Even if the PostgreSQL service is not running with it, you
would be able to see see issue as long as the parameter is enabled:
/messages/by-id/BACA8550-4F3D-41D0-A0E1-5BF307BCD648@elevated-dev.com

The OP says:

I don't use shared_memory_type and also not dynamic_shared_memory_type so
they're default (mmap for the first, posix for the second). Got the settings

It looks like the OP might be misunderstanding
dynamic_shared_memory_type. Outside of Windows, it first prefers
posix, then sysv and finally mmap. Thus it is quite usual to use posix
for DSM segments. Actually posix is defaultly used for
dynamic_shared_memory_type on my Rocky 9 box.

I can't pinpoint why the "No such file or directory" is comnig up,
though.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#4Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Kyotaro Horiguchi (#3)
Re: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

At Wed, 24 May 2023 09:57:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

dynamic_shared_memory_type. Outside of Windows, it first prefers
posix, then sysv and finally mmap. Thus it is quite usual to use posix
for DSM segments. Actually posix is defaultly used for

It is written in the documentation.

https://www.postgresql.org/docs/15/runtime-config-resource.html#GUC-DYNAMIC-SHARED-MEMORY-TYPE

The use of the mmap option, which is not the default on any
platform, is generally discouraged because the operating system may
write modified pages back to disk repeatedly, increasing system I/O
load; however, it may be useful for debugging, when the pg_dynshmem
directory is stored on a RAM disk, or when other shared memory
facilities are not available.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

#5Onkobu
onkobu@onkobutanaake.de
In reply to: Kyotaro Horiguchi (#4)
Re: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

I finally found the source: a broken test in a script that initiates a
chroot environment. For some reason the author decided to write this:

test -L /dev/shm && rm /dev/shm && mkdir /dev/shm

Instead of re-creating a symlink based /dev/shm as true mountpoint it
always did so.

Since the issue came up from time to time without any clear relation I
scanned scripts on the machine that require root permissions – not so
may to dig through.

This bogus check with immediate action was replaced with a bail out of
the script in case this would be true. No bug in PostgreSQL.

Am 24.05.23 um 03:09 schrieb Kyotaro Horiguchi:

At Wed, 24 May 2023 09:57:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

dynamic_shared_memory_type. Outside of Windows, it first prefers
posix, then sysv and finally mmap. Thus it is quite usual to use posix
for DSM segments. Actually posix is defaultly used for

It is written in the documentation.

https://www.postgresql.org/docs/15/runtime-config-resource.html#GUC-DYNAMIC-SHARED-MEMORY-TYPE

The use of the mmap option, which is not the default on any
platform, is generally discouraged because the operating system may
write modified pages back to disk repeatedly, increasing system I/O
load; however, it may be useful for debugging, when the pg_dynshmem
directory is stored on a RAM disk, or when other shared memory
facilities are not available.

regards.

--
------
Web: http://onkobutanaake.de - mit RSS-Feed
Verschlüsselte/ signierte Elektropost (PGP/GPG) an diese Adresse bevorzugt,
öffentlicher Schlüssel http://onkobutanaake.de/onkobutanaake.asc
-
Encrypted/ signed electro-mail (PGP/GPG) to this address preferred,
public key http://onkobutanaake.de/onkobutanaake.asc

Fingerprint 6261 2530 89DA 93E3 2D9F A0C2 B660 481D 9F77 A942

Attachments:

OpenPGP_0xB660481D9F77A942.ascapplication/pgp-keys; name=OpenPGP_0xB660481D9F77A942.ascDownload
#6Michael Paquier
michael@paquier.xyz
In reply to: Onkobu (#5)
Re: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

On Mon, May 29, 2023 at 04:56:09PM +0200, Onkobu wrote:

This bogus check with immediate action was replaced with a bail out of the
script in case this would be true. No bug in PostgreSQL.

Cool. Thanks for the feedback!
--
Michael