LLVM JIT: any JIT-compiled query crashes (SIGILL) on a libLLVM 19 + ASAN build

Hi Andres,

Just to let you know — the CI run for this commitfest entry shows the
same crash independently on master as well, so this may not be an RPR
(Reported-Problem Reproduction) issue specific to the patch.

The identical crash occurs on a standalone test against master.

Thanks,
Henson

2026년 6월 10일 (수) 오전 11:09, Henson Choi <assam258@gmail.com>님이 작성:

(Reported-Problem Reproduction) -> (Row Pattern Recognition)

2026년 6월 10일 (수) 오전 11:42, Henson Choi <assam258@gmail.com>님이 작성:

Hi,

On Tue Jun 9, 2026 at 11:09 PM -03, Henson Choi wrote:

While looking into Andres Freund's note that cfbot is failing with crashes
inside the JIT on the Row Pattern Recognition patch [1], I found that the
crash is not specific to that patch at all: on the CI's AddressSanitizer
build with LLVM 19, any query that is pushed through the LLVM JIT code
generator crashes the backend with SIGILL. It reproduces on plain master
with a trivial aggregate, so I am reporting it as its own issue, separate
from that feature.

[ ... ]

I investigated this issue a bit and I may found the root cause.

The meson build passes get_option('c_args') directly to the clang
command that generates llvmjit_types.bc and other bitcode files used for
inlining. When building with -fsanitize=address, the sanitizer
instrumentation may change struct layouts in the generated LLVM IR.

This causes a mismatch between the field indices the JIT code expects
(defined via FIELDNO_* macros, e.g., FIELDNO_EXPRSTATE_PARENT = 11) and
the actual struct layout in the bitcode. The result is an assertion
failure when accessing struct fields:

Assertion failed: (indexValid(N)), function getElementType
LLVMStructGetTypeAtIndex(StructTy=..., i=11)

IIUC the SIGILL crash in decodeDiscriminator() reported initially was
likely a secondary effect of this struct layout corruption.

I think that the fix is to filter out sanitizer flags when generating
bitcode for the JIT code, see the attached diff.

With this fix, JIT works correctly under ASAN + LLVM 19 on my machine.
Can you please also test it on your side?

I'm also wondering if this happens only with LLVM 19 or other versions
too.

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

Hi Matheus,

Thanks for digging into this, and for the patch!

I think that the fix is to filter out sanitizer flags when generating
bitcode for the JIT code [...]
With this fix, JIT works correctly under ASAN + LLVM 19 on my machine.

Confirmed here too: with your filter applied the crash is gone and the JIT
runs normally under ASAN. Filtering the sanitizer flags out of the
bitcode is the right fix.

the sanitizer instrumentation may change struct layouts in the generated
LLVM IR [...] FIELDNO_EXPRSTATE_PARENT = 11 [...]

One nit: on libLLVM 20.1.8 the bitcode struct layout is identical with and
without -fsanitize=address (e.g. %struct.ExprState, index 11 stays a
pointer), so it isn't a FIELDNO/layout mismatch here. In short, the crash
needs debug info (-ggdb) and sanitizer instrumentation to both land in the
JIT bitcode: the SIGILL is in decodeDiscriminator(), i.e. the instrumented
IR going through the debug-info path. Your fix keeps the debug info but
drops the instrumentation, and that alone stops it -- so the
instrumentation is the trigger. The LLVM 19 assertion is likely the same
cause surfacing differently.

I'm also wondering if this happens only with LLVM 19 or other versions
too.

Not LLVM 19 only -- I reproduced the same SIGILL on libLLVM 20.1.8.

v2 series attached, folding in your fix:

0001 Add a "jit" regression test (renamed/minimized from "jit_crash").
jit is off by default now, so this turns it on to push a trivial
query through the JIT provider.

0002 Your meson fix, with an added warning() so a sanitizer build knows
its JIT code won't be instrumented. (Author: Matheus Alcantara.)

0003 Same for autoconf: filter sanitizer flags from BITCODE_CFLAGS/
CXXFLAGS with a configure warning, plus -g under --enable-debug so
the bitcode keeps debug info. The -g part is a judgment call --
autoconf just rebuilds BITCODE_CFLAGS from a whitelist that
doesn't include -g -- so feel free to keep or drop it.

Tested on both build systems with an ASAN backend: the jit test crashes
before and passes after, JIT stays functional (pg_jit_available() = t,
EXPLAIN ANALYZE shows functions compiled), and the warning fires.

Thanks,
Henson

On Thu Jun 11, 2026 at 10:48 PM -03, Henson Choi wrote:

I think that the fix is to filter out sanitizer flags when generating
bitcode for the JIT code [...]
With this fix, JIT works correctly under ASAN + LLVM 19 on my machine.

Confirmed here too: with your filter applied the crash is gone and the JIT
runs normally under ASAN. Filtering the sanitizer flags out of the
bitcode is the right fix.

Thanks for confirming.

the sanitizer instrumentation may change struct layouts in the generated
LLVM IR [...] FIELDNO_EXPRSTATE_PARENT = 11 [...]

One nit: on libLLVM 20.1.8 the bitcode struct layout is identical with and
without -fsanitize=address (e.g. %struct.ExprState, index 11 stays a
pointer), so it isn't a FIELDNO/layout mismatch here. In short, the crash
needs debug info (-ggdb) and sanitizer instrumentation to both land in the
JIT bitcode: the SIGILL is in decodeDiscriminator(), i.e. the instrumented
IR going through the debug-info path. Your fix keeps the debug info but
drops the instrumentation, and that alone stops it -- so the
instrumentation is the trigger. The LLVM 19 assertion is likely the same
cause surfacing differently.

Ok, it make sense.

I'm also wondering if this happens only with LLVM 19 or other versions
too.

Not LLVM 19 only -- I reproduced the same SIGILL on libLLVM 20.1.8.

v2 series attached, folding in your fix:

0001 Add a "jit" regression test (renamed/minimized from "jit_crash").
jit is off by default now, so this turns it on to push a trivial
query through the JIT provider.

0002 Your meson fix, with an added warning() so a sanitizer build knows
its JIT code won't be instrumented. (Author: Matheus Alcantara.)

0003 Same for autoconf: filter sanitizer flags from BITCODE_CFLAGS/
CXXFLAGS with a configure warning, plus -g under --enable-debug so
the bitcode keeps debug info. The -g part is a judgment call --
autoconf just rebuilds BITCODE_CFLAGS from a whitelist that
doesn't include -g -- so feel free to keep or drop it.

Tested on both build systems with an ASAN backend: the jit test crashes
before and passes after, JIT stays functional (pg_jit_available() = t,
EXPLAIN ANALYZE shows functions compiled), and the warning fires.

Thanks. I think that the patches looks good and I also think that it's
good to have a JIT test case since it's off by default. I'm just
wondering if the the test patch should be 0003 instead 0001 since it
will break CI if committed before the meson and autoconf changes.

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

Hi Matheus,

I think that the patches looks good and I also think that it's
good to have a JIT test case since it's off by default. I'm just
wondering if the the test patch should be 0003 instead 0001 since it
will break CI if committed before the meson and autoconf changes.

Agreed on both counts. The series started out to show that the JIT
crash seen under Row Pattern Recognition was not RPR's fault but a
generic ASAN + JIT bitcode + debug-info problem, but since JIT is off
by default a dedicated test is worth having on its own, and I think it
makes a good seed to grow further coverage on.

And you're right about the ordering: if the test goes in first it turns
JIT on while the bitcode is still instrumented, so any commit between the
test and the build-system fixes would crash CI. I've reordered the
series so the fixes land first:

0001 meson: exclude sanitizer flags from the JIT bitcode.
(Author: Matheus Alcantara.)

0002 autoconf: same filtering for BITCODE_CFLAGS/CXXFLAGS, plus -g
under --enable-debug so the bitcode keeps its debug info.

0003 Add the "jit" regression test, which turns JIT on to push a
trivial query through the provider. This is the minimal case
that crashes the backend with ASAN enabled, and it passes again
once 0001/0002 are applied.

That way the build-system fixes precede the test at every point in the
history, so CI stays green wherever it is bisected.

Thanks!
Henson

Hi hackers,

On an AddressSanitizer build, every JIT-compiled query crashes or
hangs the backend while the JIT generates code for it, even "SELECT 1"
with jit = on and jit_above_cost = 0. On Linux the backend dies of
SIGILL; on macOS it loops forever inside LLVM.

The JIT takes the LLVM types of PostgreSQL's structs and scalars from
the global variables in llvmjit_types.c, which is compiled to bitcode
with the same flags. To add a redzone after each variable,
AddressSanitizer wraps it in a struct of type { T, [N x i8] }: the
declared type T at offset zero, followed by the redzone. So
llvm_pg_var_type() hands that struct type to the JIT instead of T,
and, for example, a GEP for a field of ExprContext indexes a struct
with two elements, which makes the IR invalid.

v4, attached, leaves the build systems alone and fixes this where the
types are read:

0001 The jit regression test. It turns jit on and sets
jit_above_cost to zero, so that "SELECT 1" is JIT-compiled.
0002 llvm_pg_var_type() raises an error for a type that is not of
the expected form (a scalar or a named struct) instead of
handing it to the JIT.
0003 When llvm_pg_var_type() finds a struct whose second element
is a redzone, it returns the type of the first element, T.
0004 An alternative to 0003: exclude the variables in
llvmjit_types.c from AddressSanitizer instrumentation with a
clang attribute pragma, so that they are never wrapped.

Only one of 0003 and 0004 is needed; whichever is better can be
picked. The trade-off is:

0003 Leaves llvmjit_types.c alone and changes only llvmjit.c, but
relies on the shape of the struct that AddressSanitizer
builds ({ T, [N x i8] }). If that shape changes, it falls
back to the error from 0002.
0004 Keeps the wrapping from happening at all, so it does not
depend on how AddressSanitizer works inside, but adds a
clang-specific pragma to llvmjit_types.c.

0001 comes first only to show what each of the following patches
changes. Committed alone, it breaks CI's AddressSanitizer build, so
it should go in together with 0002 and 0003 (or 0004), or after them.

Testing (the jit test, run by pg_regress against a running server):

macOS arm64 Rocky Linux 10 x86_64
Apple clang ASan gcc 14 ASan
LLVM 21.1.7, meson LLVM 21.1.8, meson
0001 hangs inside LLVM SIGILL
0001+0002 ERROR ERROR
0001+0002+0003 ok ok
0001+0002+0004 ok ok

It of course works with both 0003 and 0004 applied as well: the
variables are then never wrapped, so the code in 0003 is just not
reached.

Best regards,
Henson

On Mon, Oct 26, 2026 Henson Choi wrote:

v4, attached, leaves the build systems alone and fixes this where the
types are read:

Hello Henson,

I've been reviewing the LLVM code recently and it led me to your work,
thank you for digging into this area of the code.

I tested your v4 patches on Debian 13 to match the CI image's
distribution with gcc 14.2, meson, -O2 -ggdb -fsanitize=address
-fno-sanitize-recover=all, cassert, against both trixie's LLVM 19.1.7
and LLVM 21.1.8 from trixie-backports. On x86_64, each row is the jit
regression test plus a heavier query (jit_above_cost = 0,
jit_optimize_above_cost = 0, an aggregate over generate_series(1,
100000)):

LLVM 19.1.7 LLVM 21.1.8
0001 SIGILL SIGSEGV in
llvm::DINode::getFlagString()
0001+0002 ERROR (TypeSizeT ...) ERROR (TypeSizeT ...)
0001+0002+0003 ok, correct result ok, correct result
0001+0002+0004 ok, correct result ok, correct result

So on LLVM 21 the unpatched symptom is a SEGV in the debug-info code
rather than SIGILL, which fits your reading that the invalid IR is what
reaches it.

Both 0003 and 0004 work here. I'd lean towards 0004: it keeps the types
from being wrapped in the first place, and llvmjit_types.c is only ever
compiled by clang. 0002 is worth keeping either way, as a cheap check
that turns a crash in LLVM into an ERROR.

The same crash also shows up on aarch64 (Graviton, LLVM 19 and 21,
SIGILL with both), and both 0003 and 0004 fix it there too.

FWIW I've got a JIT-inlining fix with a pageinspect test [1]/messages/by-id/auSUW7eDMqZfEF80@floki that is
the second test in the tree to turn JIT on, and CI's ASAN job crashes
in it for exactly this reason, so I'd be glad to see this go in.

best.

-greg

[1]: /messages/by-id/auSUW7eDMqZfEF80@floki