scorpionfly needs more semaphores

Thomas Munro <thomas.munro@gmail.com> writes:

We're seeing occasional failures like this:
running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
FATAL: could not create semaphores: No space left on device
2019-09-13 12:11:26.882 PDT [64926] DETAIL: Failed system call was
semget(5728001, 17, 03600).

I think you should switch to using "named" POSIX semaphores by
building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
squillion little files under /tmp and mmap() them), or increase the
number of SysV semaphores you can create with sysctl[1], or finish
writing your operating system[2] so you can switch to "unnamed" POSIX
semaphores :-)

I'd recommend the second option. Since the discussion in [1],
we've fixed our docs for OpenBSD to say

In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl,
for example:
# sysctl kern.seminfo.semmni=100
To make these settings persist over reboots, modify /etc/sysctl.conf.
You will usually want to increase kern.seminfo.semmni and
kern.seminfo.semmns, as OpenBSD's default settings for these are
uncomfortably small.

Scorpionfly also seems to be having problems with its git repo breaking on
a regular basis. I have no idea what's up with that.

regards, tom lane

Thus said Tom Lane <tgl@sss.pgh.pa.us> on Wed, 18 Sep 2019 00:33:19 -0400

Thomas Munro <thomas.munro@gmail.com> writes:

We're seeing occasional failures like this:
running bootstrap script ... 2019-09-13 12:11:26.882 PDT [64926]
FATAL: could not create semaphores: No space left on device
2019-09-13 12:11:26.882 PDT [64926] DETAIL: Failed system call was
semget(5728001, 17, 03600).

I think you should switch to using "named" POSIX semaphores by
building with USE_NAMED_POSIX_SEMAPHORES (then it'll create a
squillion little files under /tmp and mmap() them), or increase the
number of SysV semaphores you can create with sysctl[1], or finish
writing your operating system[2] so you can switch to "unnamed" POSIX
semaphores :-)

I'd recommend the second option. Since the discussion in [1],
we've fixed our docs for OpenBSD to say

In OpenBSD 3.3 and later, IPC parameters can be adjusted using sysctl,
for example:
# sysctl kern.seminfo.semmni=100
To make these settings persist over reboots, modify /etc/sysctl.conf.
You will usually want to increase kern.seminfo.semmni and
kern.seminfo.semmns, as OpenBSD's default settings for these are
uncomfortably small.

Thanks, Thomas and Tom for reaching out to me. I certainly don't want to
recompile my kernel, as I basically run -current OpenBSD via snapshots.

That said, I've made the adjustment to the sysctl:

$ sysctl | ag kern.seminfo.semmni
kern.seminfo.semmni=100

Scorpionfly also seems to be having problems with its git repo breaking on
a regular basis. I have no idea what's up with that.

That is a mystery to me as well. 9.4 stable seems to be the branch with
the most problems:
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&amp;br=REL9_4_STABLE

My cronjobs:
0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl --verbose
0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl --run-all

I'm willing to make more tweaks to prevent these false positives, so
feel free to continue monitoring to see how things work out over the
next several builds.

On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:

$ sysctl | ag kern.seminfo.semmni
kern.seminfo.semmni=100

It still seems to be happening. Perhaps you need to increase semmns too?

Scorpionfly also seems to be having problems with its git repo breaking on
a regular basis. I have no idea what's up with that.

That is a mystery to me as well. 9.4 stable seems to be the branch with
the most problems:
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=scorpionfly&amp;br=REL9_4_STABLE

My cronjobs:
0 */6 * * * cd /home/pgbuilder/bin/REL10 && ./run_build.pl --verbose
0 */12 * * * cd /home/pgbuilder/bin/REL10 && ./run_branches.pl --run-all

I think you need just the run_branches.pl entry.

BTW I'm sorry for my flippant tone about sem_init() earlier, which
someone pointed out to me was not great cross-project open source.
What I really meant to say was: if you have OpenBSD developer
contacts, it would be cool if you could highlight that issue as
something that would make the PostgreSQL-on-OpenBSD experience nicer
for end users (and I suspect other multi-process software too). On
Linux and FreeBSD we now use sem_init()
(PREFERRED_SEMAPHORES=UNNAMED_POSIX) so users never need to worry
about configuring that kernel resource. On at least AIX, we still
have to use SysV, but there the default limits are high enough that
(according to our manual) no adjustment is required. Also, as I
speculated in that other thread: based on a quick peek at the
implementation, you might get better performance on very large busy
PostgreSQL clusters from our cache line padded sem_init() array than
you do with your more densely packed SysV semas (I could be totally
wrong about that, but we saw an effect like that on some other
operating system).

--
Thomas Munro
https://enterprisedb.com

On 2019-09-23 00:29, Thomas Munro wrote:

On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:

$ sysctl | ag kern.seminfo.semmni
kern.seminfo.semmni=100

It still seems to be happening. Perhaps you need to increase semmns too?

I have on my OpenBSD 6.5 /etc/sysctl.conf the following:

kern.seminfo.semmni=200
kern.seminfo.semmns=4000

and it seems to work fine.

/Mikael

On Mon Sep 23, 2019 at 8:34 PM Mikael Kjellström wrote:

On 2019-09-23 00:29, Thomas Munro wrote:

On Wed, Sep 18, 2019 at 4:55 PM jungle boogie <jungleboogie0@gmail.com> wrote:

$ sysctl | ag kern.seminfo.semmni
kern.seminfo.semmni=100

It still seems to be happening. Perhaps you need to increase semmns too?

I have on my OpenBSD 6.5 /etc/sysctl.conf the following:

kern.seminfo.semmni=200
kern.seminfo.semmns=4000

and it seems to work fine.

Thanks! I've made these adjustments and we'll see what happens.

Thomas,

No offense taken with your input from last week. Your additional input this week
is welcomed and I'll pass along to some folks.

Thank you all for continuing to monitor Postgresql building on OpenBSD.