BUG #19609: Server crashes when executing a JIT-compiled SQL function on s390x
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253320psql -h localhost -U postgresBuilt from patchset v2 (message #2), August 05, 2026 at 11:38 AM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253320_2 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253320_2 && git checkout t253320_2Patchset v2 (message #2) is on t253320_2
The following bug has been logged on the website:
Bug reference: 19609
Logged by: Artem Zarubin
Email address: a.prototype7@gmail.com
PostgreSQL version: 19beta2
Operating system: Ubuntu 24.04.4 LTS (s390x)
Description:
Hello, I found a reproducible server crash when an SQL-language function is
executed with LLVM JIT enabled on s390x.
Tested commit:
bad: 0392fb900eb89f52988cccd33046443c39c70d1c, PostgreSQL 19devel
The server was built with:
./configure \
--prefix=/home/test/pginstall-vanilla18 \
--enable-debug \
--enable-cassert \
--with-llvm \
LLVM_CONFIG=/usr/bin/llvm-config-18 \
CLANG=/usr/bin/clang-18
Environment:
Architecture: s390x
OS: Ubuntu 24.04.4 LTS
Kernel: Linux 6.8.0-136-generic
GCC: 13.3.0
LLVM: 18.1.3
The following parameters are used by the attached TAP test:
restart_after_crash = on
jit = on
Minimal SQL script to reproduce:
CREATE FUNCTION type_text(oid) RETURNS text
LANGUAGE sql STABLE
AS $$
SELECT typname::text
FROM pg_catalog.pg_type
WHERE oid = $1
$$;
SET jit = on;
SET jit_above_cost = 0;
SET jit_inline_above_cost = -1;
SET jit_optimize_above_cost = -1;
SET jit_expressions = on;
SET jit_tuple_deforming = off;
SELECT count(*)
FROM (
SELECT oid
FROM pg_catalog.pg_type
ORDER BY oid
LIMIT 7
) AS t
WHERE type_text(t.oid) = 'int2vector';
Expected result:
count
-------
1
(1 row)
Actual result:
server closed the connection unexpectedly
This probably means the server terminated abnormally before or while
processing the request.
connection to server was lost
The server log contains:
LOG: client backend (PID ...) was terminated by signal 11: Segmentation
fault
DETAIL: Failed process was running: SELECT count(*) ...
Complete backtrace available from the core dump produced by the minimal
reproducer:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000002aa3d0f29bc in cstring_to_text (s=0x0) at varlena.c:186
#1 name_text (fcinfo=<optimized out>) at varlena.c:2709
#2 0x000003ff9ae36126 in ?? ()
Backtrace stopped: frame did not save the PC
si_signo = 11 (SIGSEGV)
si_code = 1 (SEGV_MAPERR)
si_addr = 0x0
r1 = 0x0
r11 = 0x0
pc = 0x2aa3d0f29bc <name_text+60>
The frame above name_text() contains JIT-generated code without unwind
information. name_text() receives a NULL C-string pointer and crashes in
cstring_to_text().
With jit=off, the same query completes successfully and returns 1. LLVM
inlining and PostgreSQL's expensive-query JIT optimization tier are not
required: the crash is also reproduced with jit_inline_above_cost and
jit_optimize_above_cost set to -1.
The attached patch adds the reproducer as
src/test/modules/test_misc/t/014_jit_s390x.pl. From the PostgreSQL source
tree, I ran it against the installed build with:
PG_INSTALL=/home/test/pginstall-vanilla18
PATH="$PG_INSTALL/bin:$PATH" \
PERL5LIB="$PWD/src/test/perl" \
PG_REGRESS="$PWD/src/test/regress/pg_regress" \
prove -v src/test/modules/test_misc/t/014_jit_s390x.pl
The TAP test fails as follows because the backend crashes:
not ok 1 - JIT-compiled SQL function expression does not crash the backend
got: '2'
expected: '0'
not ok 2 - JIT-compiled SQL function expression returns expected row
got: ''
expected: '1'
The crash reproduced on every run of the final minimal test on this machine,
including runs against newly initialized test clusters.
---
Best regards,
Artem Zarubin
Postgres Professional: https://postgrespro.com/
Hello,
I am attaching the TAP test mentioned in the bug report.
The patch adds the self-contained reproducer as
src/test/modules/test_misc/t/014_jit_s390x.pl and registers it in the Meson
test list.
Best regards,
Artem Zarubin
Postgres Professional: https://postgrespro.com/
вт, 4 авг. 2026 г. в 19:28, PG Bug reporting form <noreply@postgresql.org>:
Show quoted text
The following bug has been logged on the website:
Bug reference: 19609
Logged by: Artem Zarubin
Email address: a.prototype7@gmail.com
PostgreSQL version: 19beta2
Operating system: Ubuntu 24.04.4 LTS (s390x)
Description:Hello, I found a reproducible server crash when an SQL-language function is
executed with LLVM JIT enabled on s390x.Tested commit:
bad: 0392fb900eb89f52988cccd33046443c39c70d1c, PostgreSQL 19devel
The server was built with:
./configure \
--prefix=/home/test/pginstall-vanilla18 \
--enable-debug \
--enable-cassert \
--with-llvm \
LLVM_CONFIG=/usr/bin/llvm-config-18 \
CLANG=/usr/bin/clang-18Environment:
Architecture: s390x
OS: Ubuntu 24.04.4 LTS
Kernel: Linux 6.8.0-136-generic
GCC: 13.3.0
LLVM: 18.1.3The following parameters are used by the attached TAP test:
restart_after_crash = on
jit = onMinimal SQL script to reproduce:
CREATE FUNCTION type_text(oid) RETURNS text
LANGUAGE sql STABLE
AS $$
SELECT typname::text
FROM pg_catalog.pg_type
WHERE oid = $1
$$;SET jit = on;
SET jit_above_cost = 0;
SET jit_inline_above_cost = -1;
SET jit_optimize_above_cost = -1;
SET jit_expressions = on;
SET jit_tuple_deforming = off;SELECT count(*)
FROM (
SELECT oid
FROM pg_catalog.pg_type
ORDER BY oid
LIMIT 7
) AS t
WHERE type_text(t.oid) = 'int2vector';Expected result:
count
-------
1
(1 row)Actual result:
server closed the connection unexpectedly
This probably means the server terminated abnormally before or while
processing the request.
connection to server was lostThe server log contains:
LOG: client backend (PID ...) was terminated by signal 11: Segmentation
fault
DETAIL: Failed process was running: SELECT count(*) ...Complete backtrace available from the core dump produced by the minimal
reproducer:Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000002aa3d0f29bc in cstring_to_text (s=0x0) at varlena.c:186
#1 name_text (fcinfo=<optimized out>) at varlena.c:2709
#2 0x000003ff9ae36126 in ?? ()
Backtrace stopped: frame did not save the PCsi_signo = 11 (SIGSEGV)
si_code = 1 (SEGV_MAPERR)
si_addr = 0x0r1 = 0x0
r11 = 0x0
pc = 0x2aa3d0f29bc <name_text+60>The frame above name_text() contains JIT-generated code without unwind
information. name_text() receives a NULL C-string pointer and crashes in
cstring_to_text().With jit=off, the same query completes successfully and returns 1. LLVM
inlining and PostgreSQL's expensive-query JIT optimization tier are not
required: the crash is also reproduced with jit_inline_above_cost and
jit_optimize_above_cost set to -1.The attached patch adds the reproducer as
src/test/modules/test_misc/t/014_jit_s390x.pl. From the PostgreSQL source
tree, I ran it against the installed build with:PG_INSTALL=/home/test/pginstall-vanilla18
PATH="$PG_INSTALL/bin:$PATH" \
PERL5LIB="$PWD/src/test/perl" \
PG_REGRESS="$PWD/src/test/regress/pg_regress" \
prove -v src/test/modules/test_misc/t/014_jit_s390x.plThe TAP test fails as follows because the backend crashes:
not ok 1 - JIT-compiled SQL function expression does not crash the
backend
got: '2'
expected: '0'
not ok 2 - JIT-compiled SQL function expression returns expected row
got: ''
expected: '1'The crash reproduced on every run of the final minimal test on this
machine,
including runs against newly initialized test clusters.---
Best regards,
Artem Zarubin
Postgres Professional: https://postgrespro.com/
=?UTF-8?B?0JDRgNGC0ZHQvCDQl9Cw0YDRg9Cx0LjQvQ==?= <a.prototype7@gmail.com> writes:
I am attaching the TAP test mentioned in the bug report.
If the same test doesn't fail on other arches (x86, arm, etc) then
this is almost certainly an LLVM bug, and you should be reporting
it to them.
regards, tom lane
Hi, Tom and Artem!
I looked at this on qemu-system-s390x (TCG),
Ubuntu 24.04, with an unpatched PG JIT emitter.
Arch test
---------------
Same PG sources and JIT GUCs (jit=on, jit_above_cost=0,
jit_inline_above_cost=-1, …, jit_dump_bitcode=off):
s390x TCG, LLVM 16.0.6 / 18.1.3 / 19.1.1 — crash
("server closed the connection" on clean builds)
x86_64, LLVM 18.1.3 and 20.1.8 — OK (8/8; EXPLAIN shows
JIT Functions: 3, Inlining false)
So the failure is arch-specific in the sense Tom meant. Caveat:
my s390x crashes are under TCG, not a real Z box (the original
report was on hardware). qemu-user did not reproduce, full-system
TCG did.
Repro
-----
CREATE FUNCTION type_text(oid) RETURNS text
LANGUAGE sql STABLE
AS $$ SELECT typname::text FROM pg_catalog.pg_type
WHERE oid = $1 $$;
SELECT type_text(22) FROM generate_series(1,7) g;
-- TAP form with LIMIT 7 / count(*) fails the same way
-- generate_series(1,6) / LIMIT 6 is fine; jit=off and
-- allowing inlining are fine
With a diagnostic build, name_text sees a NULL byref Datum with
isnull=false on the 7th call. jit_dump_bitcode=on often hides
the bug.
Where it breaks (short)
-----------------------
Rough IR:
store i64 %d, ptr inttoptr (i64 <op->resvalue>)
call i64 @name_text(ptr inttoptr (i64 <fcinfo>))
Slot content on call #7 is still good (C read). Reloading
tts_values in IR does not help. A C helper that writes through
the resvalue pointer can make name_text see a good Datum. The IR
store via baked inttoptr does not. On call #7 I also saw live
ExprState/fcinfo addresses differ from earlier calls — so the
JIT is using a stronger "freeze absolute addresses at first
compile" contract than the interpreter.
I tried a tiny standalone clang test on the same s390x TCG guest
(store / store+call through compile-time inttoptr to a stable
heap cell, 8 times). That did *not* fail. So I do not yet have
a reduced LLVM testcase, a naive inttoptr store is not enough.
As an academic check only: if the JIT loads resvalue / resnull /
fcinfo from the live ExprState each step (IR GEP/load from
state->steps[opno]…) instead of baking them as l_ptr_const /
inttoptr, the repro stopped failing on my harness. I am not
proposing that as a patch — just a data point that the failure
is tied to the baked-address model on this target.
-------
Please treat QEMU TCG as a possible confounder
until someone confirms on real Z (original reporter: hardware,
LLVM 18.1.3).
I am filing this mainly as investigation notes. If any of it
helps with the LLVM report or with later PG work, good.
ср, 5 авг. 2026 г. в 19:15, Tom Lane <tgl@sss.pgh.pa.us>:
=?UTF-8?B?0JDRgNGC0ZHQvCDQl9Cw0YDRg9Cx0LjQvQ==?= <a.prototype7@gmail.com>
writes:I am attaching the TAP test mentioned in the bug report.
If the same test doesn't fail on other arches (x86, arm, etc) then
this is almost certainly an LLVM bug, and you should be reporting
it to them.regards, tom lane
--
Regards,
Rachitskiy Andrey
On Wed, Aug 5, 2026 at 3:50 PM PG Bug reporting form
<noreply@postgresql.org> wrote:
Architecture: s390x
OS: Ubuntu 24.04.4 LTS
Kernel: Linux 6.8.0-136-generic
GCC: 13.3.0
LLVM: 18.1.3
FWIW, I've found that clang 18 and early minors of clang 19 have other
kinds of broken code generation on this architecture.
--
John Naylor
Amazon Web Services