occasional ECPG failures on dikkop (FreeBSD)
Hi,
about a month ago dikkop started reporting occasional failures in ECPG
tests. I'm not very familiar with ecpg, and I've been unable to figure
this out so far, so I wonder if others might know ...
The failures seem to happen maybe ~5% of the runs, but only when it's
through the buildfarm client. I've been unable to reproduce the issue,
even when trying to use exactly the same options etc.
Two failures from master:
*
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dikkop&dt=2026-04-07%2011%3A00%3A39
*
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dikkop&dt=2026-05-04%2010%3A00%3A10
However, it seems to affect older branches too, all the way back to
REL_14_STABLE.
The failures started to appear ~30 days ago, which aligns with the
machine being upgraded from FreeBSD 14.1 to 14.4. (It might have been
running 14.3, not sure.)
The failures look like this:
ok 64 - thread/prep 732 ms
not ok 65 - thread/alloc 65 ms
# (test process was terminated by signal 11: Segmentation fault)
ok 66 - thread/descriptor 136 ms
so the problem seems to be in thread/alloc. But the log says this:
$ grep 'signal 11' /var/log/messages
Apr 1 21:53:30 generic kernel: pid 27622 (thread_implicit), jid 0, uid
1001: exited on signal 11 (core dumped)
Apr 1 23:54:19 generic kernel: pid 50594 (alloc), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 2 20:19:57 generic kernel: pid 53415 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 4 02:07:58 generic kernel: pid 48615 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 7 12:58:20 generic kernel: pid 17092 (alloc), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 9 13:21:47 generic kernel: pid 65784 (alloc), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 10 18:20:17 generic kernel: pid 67540 (thread_implicit), jid 0, uid
1001: exited on signal 11 (core dumped)
Apr 22 16:29:29 generic kernel: pid 10941 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 22 20:29:47 generic kernel: pid 32964 (thread_implicit), jid 0, uid
1001: exited on signal 11 (core dumped)
Apr 22 23:34:54 generic kernel: pid 43109 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 29 19:24:49 generic kernel: pid 81996 (thread), jid 0, uid 1001:
exited on signal 11 (core dumped)
Apr 30 10:58:42 generic kernel: pid 65438 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
May 3 22:15:57 generic kernel: pid 21640 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
May 4 12:08:15 generic kernel: pid 98832 (alloc), jid 0, uid 1001:
exited on signal 11 (core dumped)
May 5 12:04:33 generic kernel: pid 65140 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
May 5 13:05:45 generic kernel: pid 12122 (prep), jid 0, uid 1001:
exited on signal 11 (core dumped)
So there is plenty of segfaults in the other ecpg tests, it seems.
Sadly, I haven't found any core files. I'll try to look again after the
next failure.
Any ideas? I don't see similar failures on other machines.
regards
--
Tomas Vondra
Tomas Vondra <tomas@vondra.me> writes:
about a month ago dikkop started reporting occasional failures in ECPG
tests. I'm not very familiar with ecpg, and I've been unable to figure
this out so far, so I wonder if others might know ...
There's some related discussion in the ECPGprepared_statement()
thread:
/messages/by-id/75460b3c-255d-47eb-b889-d99de38e6758@gmail.com
I have no idea whether the bug that Andrew just identified explains
dikkop's problem, though.
regards, tom lane
On 5/6/26 16:00, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
about a month ago dikkop started reporting occasional failures in ECPG
tests. I'm not very familiar with ecpg, and I've been unable to figure
this out so far, so I wonder if others might know ...There's some related discussion in the ECPGprepared_statement()
thread:/messages/by-id/75460b3c-255d-47eb-b889-d99de38e6758@gmail.com
I have no idea whether the bug that Andrew just identified explains
dikkop's problem, though.regards, tom lane
Ah, I haven't noticed the last couple messagess, mentioning dikkop. Not
sure if it's the same issue - I guess we'll have to wait if the failures
go away once it gets fixed.
Still, it seems like a long-standing issue. It's curious dikkop started
failing only very recently, after the OS upgrade.
thanks
--
Tomas Vondra
Tomas Vondra <tomas@vondra.me> writes:
Ah, I haven't noticed the last couple messagess, mentioning dikkop. Not
sure if it's the same issue - I guess we'll have to wait if the failures
go away once it gets fixed.
Still, it seems like a long-standing issue. It's curious dikkop started
failing only very recently, after the OS upgrade.
Yeah. Given that these are threading tests, I'm suspecting some
change in thread scheduling behavior in this latest FBSD release,
which somehow made it easier to hit a pre-existing issue.
regards, tom lane
On 5/6/26 16:15, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
Ah, I haven't noticed the last couple messagess, mentioning dikkop. Not
sure if it's the same issue - I guess we'll have to wait if the failures
go away once it gets fixed.Still, it seems like a long-standing issue. It's curious dikkop started
failing only very recently, after the OS upgrade.Yeah. Given that these are threading tests, I'm suspecting some
change in thread scheduling behavior in this latest FBSD release,
which somehow made it easier to hit a pre-existing issue.
The failures on dikkop continued, and I got annoyed by that. So I spent
a bit of time investigating this today. I managed to reproduce the
crashes in a way that allowed me to grab backtraces from the core files
(which we don't seem to get in the buildfarm). And all of them generally
looked like this:
--------------------------------------------------------------------
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0 __vfprintf (fp=fp@entry=0x889dab00, locale=locale@entry=0x841dfcf0
<__xlocale_global_locale>, serrno=0, fmt0=fmt0@entry=0x20087a
"Connection: %d", ap=...) at /usr/src/lib/libc/stdio/vfprintf.c:477
warning: 477 /usr/src/lib/libc/stdio/vfprintf.c: No such file or directory
[Current thread is 1 (LWP 933707)]
(gdb) bt
#0 __vfprintf (fp=fp@entry=0x889dab00, locale=locale@entry=0x841dfcf0
<__xlocale_global_locale>, serrno=0, fmt0=fmt0@entry=0x20087a
"Connection: %d", ap=...) at /usr/src/lib/libc/stdio/vfprintf.c:477
#1 0x000000008411d434 in vsprintf_l (locale=0x841dfcf0
<__xlocale_global_locale>, ap=..., str=<optimized out>, fmt=<optimized
out>) at /usr/src/lib/libc/stdio/vsprintf.c:61
#2 vsprintf (str=0x889daefc "", fmt=<optimized out>, ap=...) at
/usr/src/lib/libc/stdio/vsprintf.c:68
#3 0x00000000841123f4 in sprintf (str=0x0, str@entry=0x889daefc "",
fmt=0x841dfcf0 <__xlocale_global_locale> "") at
/usr/src/lib/libc/stdio/sprintf.c:56
#4 0x0000000000210ed8 in fn (arg=0x0) at prep.pgc:39
#5 0x00000000847844fc in thread_start (curthread=0x63044546d010) at
/usr/src/lib/libthr/thread/thr_create.c:291
#6 0x0000000000000000 in ?? ()
--------------------------------------------------------------------
Which is a crash in snprintf, or rather vfprintf on this line:
decpt_len = (decimal_point[1] == '\0' ? 1 : strlen(decimal_point));
That's a bit bizarre, because I thought snprintf is meant to be thread
safe. After staring at this for a while and getting nowhere, I decided
to make sure that's actually true, by doing this:
--------------------------------------------------------------------
#include <stdio.h>
#include <pthread.h>
void *fn(void *arg) {
char buf[128];
for(int i=0; i<10000; i++)
sprintf(buf, "%d", 1);
return NULL;
}
int main() {
pthread_t th[100];
for(int i=0; i<100; i++)
pthread_create(&th[i], NULL, fn, NULL);
for(int i=0; i<100; i++)
pthread_join(th[i], NULL);
return 0;
}
--------------------------------------------------------------------
And after running that a in a loop for a couple seconds ...
$ while /usr/bin/true; do ./test; done;
Segmentation fault (core dumped)
The backtrace is exactly the one I posted earlier. Turns out snprintf()
is not as thread-safe as I thought.
AFAIK the culprit it the line right before the crash:
decimal_point = localeconv_l(locale)->decimal_point;
/* The overwhelmingly common case is decpt_len == 1. */
decpt_len = (decimal_point[1] == '\0' ? 1 : strlen(decimal_point));
The localeconv_t() rebuilds some sort of cache, protected by an atomic
flag. But it uses atomic_store_int(), which is a "relaxed" variant:
From "atomic" manpage:
... The first, or relaxed variant, performs the operation without
imposing any ordering constraints on accesses to other memory
locations. ...
So localeconv_t() updates some fields (e.g. "decimal_point" pointer),
and sets the flag. But there's no guarantee in which order will other
threads see these updates. A thread may see the new "flag" value, but
not the decimal_point value. That seems ... not great?
I suppose this only affects architectures like ARM/AArch64, which is
what dikkop runs on (rpi5). Which would explain why we don't see this on
the x86_64 animals. opaleye didn't hit it either, but it has not run for
months, and it didn't dikkop only started hitting it ~3 months ago.
As for a fix, it seems enough to populate the cache before starting the
threads, e.g. by doing a sprintf().
regards
--
Tomas Vondra
Tomas Vondra <tomas@vondra.me> writes:
On 5/6/26 16:15, Tom Lane wrote:
Yeah. Given that these are threading tests, I'm suspecting some
change in thread scheduling behavior in this latest FBSD release,
which somehow made it easier to hit a pre-existing issue.
The backtrace is exactly the one I posted earlier. Turns out snprintf()
is not as thread-safe as I thought.
Nice detective work! I assume you've filed a FreeBSD bug?
regards, tom lane
On 6/30/26 03:43, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
On 5/6/26 16:15, Tom Lane wrote:
Yeah. Given that these are threading tests, I'm suspecting some
change in thread scheduling behavior in this latest FBSD release,
which somehow made it easier to hit a pre-existing issue.The backtrace is exactly the one I posted earlier. Turns out snprintf()
is not as thread-safe as I thought.Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.
regards
--
Tomas Vondra
Tomas Vondra <tomas@vondra.me> writes:
On 6/30/26 03:43, Tom Lane wrote:
Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.
OK. I do have one, so can file this if the bureaucracy seems excessive.
regards, tom lane
On 6/30/26 16:03, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
On 6/30/26 03:43, Tom Lane wrote:
Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.OK. I do have one, so can file this if the bureaucracy seems excessive.
Please do. There was no response so far, and I don't know when to expect
one. Their page mentions "reasonable amount of time", but I have no idea
if that's 24h or a week.
--
Tomas Vondra
On 6/30/26 17:25, Tomas Vondra wrote:
On 6/30/26 16:03, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
On 6/30/26 03:43, Tom Lane wrote:
Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.OK. I do have one, so can file this if the bureaucracy seems excessive.
Please do. There was no response so far, and I don't know when to expect
one. Their page mentions "reasonable amount of time", but I have no idea
if that's 24h or a week.
Nevermind, I just got the account created, so I'll file the bug.
--
Tomas Vondra
On 6/30/26 18:33, Tomas Vondra wrote:
On 6/30/26 17:25, Tomas Vondra wrote:
On 6/30/26 16:03, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
On 6/30/26 03:43, Tom Lane wrote:
Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.OK. I do have one, so can file this if the bureaucracy seems excessive.
Please do. There was no response so far, and I don't know when to expect
one. Their page mentions "reasonable amount of time", but I have no idea
if that's 24h or a week.Nevermind, I just got the account created, so I'll file the bug.
Here's a link: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296410
--
Tomas Vondra
On 6/30/26 19:00, Tomas Vondra wrote:
On 6/30/26 18:33, Tomas Vondra wrote:
On 6/30/26 17:25, Tomas Vondra wrote:
On 6/30/26 16:03, Tom Lane wrote:
Tomas Vondra <tomas@vondra.me> writes:
On 6/30/26 03:43, Tom Lane wrote:
Nice detective work! I assume you've filed a FreeBSD bug?
Not yet, I need to create an account for their bugzilla, which requires
mailing maintainers etc. Working on it.OK. I do have one, so can file this if the bureaucracy seems excessive.
Please do. There was no response so far, and I don't know when to expect
one. Their page mentions "reasonable amount of time", but I have no idea
if that's 24h or a week.Nevermind, I just got the account created, so I'll file the bug.
Here's a link: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296410
It seems the FreeBSD locales might be a bit more borked than I thought,
and just changing the one atomic call may not be a sufficient fix (it'd
likely fix our tests, but maybe not other cases).
It seems like a rare issue, so it probably won't be very high on the
TODO list. I'm sure the FreeBSD folks have enough other work.
I wonder if we should just tweak our tests to do a localeconv_t in the
main function, as a workaround. That should make the failures go away.
It's not a huge issue, but the failures from my animal annoy me.
regards
--
Tomas Vondra