Re: BUG #16696: Backend crash in llvmjit

Hi

On Mon, Nov 2, 2020, at 13:45, Dmitry Marakasov wrote:

* PG Bug reporting form (noreply@postgresql.org) wrote:

Environment details:
- FreeBSD 12.1 amd64
- PostgreSQL 13.0 (built from FreeBSD ports)
- llvm-10.0.1 (build from FreeBSD ports)

My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
the system, and PostgreSQL uses llvm9:

ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

I'll check once I'm home whether I can reproduce in my environment.

Regards,

Andres

* Andres Freund (andres@anarazel.de) wrote:

On Mon, Nov 2, 2020, at 13:45, Dmitry Marakasov wrote:

* PG Bug reporting form (noreply@postgresql.org) wrote:

Environment details:
- FreeBSD 12.1 amd64
- PostgreSQL 13.0 (built from FreeBSD ports)
- llvm-10.0.1 (build from FreeBSD ports)

My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
the system, and PostgreSQL uses llvm9:

ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

It doesn't give too much apart from the function name. Here's gdb
output:

#0 0x0000000810f5c70b in evalexpr_0_2 ()
No symbol table info available.
#1 0x000000080bfded8d in ExecRunCompiledExpr (state=0x80b18a850, econtext=0x801bc2a80, isNull=0x7fffffffddd7) at llvmjit_expr.c:2424
cstate = 0x80b4a9fe0
func = 0x810f5c120 <evalexpr_0_2>
#2 0x00000000007dd70b in ExecEvalExprSwitchContext (state=0x80b18a850, econtext=0x801bc2a80, isNull=0x7fffffffddd7) at ../../../src/include/executor/executor.h:313
retDatum = 140737488346496
oldContext = 0x801bc2000
#3 0x00000000007dd669 in ExecProject (projInfo=0x80b18a848) at ../../../src/include/executor/executor.h:347
econtext = 0x801bc2a80
state = 0x80b18a850
slot = 0x801bc3ec0
isnull = false
#4 0x00000000007dd42f in ExecResult (pstate=0x801bc2970) at nodeResult.c:136
node = 0x801bc2970
outerTupleSlot = 0x801bc31e0
outerPlan = 0x801bc2b10
econtext = 0x801bc2a80
#5 0x00000000007a1035 in ExecProcNodeFirst (node=0x801bc2970) at execProcnode.c:450
No locals.
#6 0x00000000007b5e22 in ExecProcNode (node=0x801bc2970) at ../../../src/include/executor/executor.h:245
No locals.
#7 0x00000000007b5a4c in fetch_input_tuple (aggstate=0x801bc2398) at nodeAgg.c:589
slot = 0x7a9ef7 <ReScanExprContext+39>
#8 0x00000000007b5678 in agg_retrieve_direct (aggstate=0x801bc2398) at nodeAgg.c:2356
node = 0x801bb5dd0
econtext = 0x801bc28e0
tmpcontext = 0x801bc27c0
peragg = 0x80b4aa6f0
pergroups = 0x80b4aa990
outerslot = 0x1a
firstSlot = 0x80b4aa000
result = 0xffff0b4aa12c
hasGroupingSets = false
numGroupingSets = 1
currentSet = 32767
nextSetSize = 0
numReset = 1
i = 1
#9 0x00000000007b250e in ExecAgg (pstate=0x801bc2398) at nodeAgg.c:2171
node = 0x801bc2398
result = 0x0
#10 0x00000000007a1035 in ExecProcNodeFirst (node=0x801bc2398) at execProcnode.c:450
No locals.
#11 0x0000000000799db2 in ExecProcNode (node=0x801bc2398) at ../../../src/include/executor/executor.h:245
No locals.
#12 0x0000000000796161 in ExecutePlan (estate=0x801bc2118, planstate=0x801bc2398, use_parallel_mode=false, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
dest=0x80b565f00, execute_once=true) at execMain.c:1646
slot = 0x80b565f50
current_tuple_count = 0
#13 0x000000000079602e in standard_ExecutorRun (queryDesc=0x80220e918, direction=ForwardScanDirection, count=0, execute_once=true) at execMain.c:364
estate = 0x801bc2118
operation = CMD_SELECT
dest = 0x80b565f00
sendTuples = true
oldcontext = 0x80220e800
#14 0x0000000000795e84 in ExecutorRun (queryDesc=0x80220e918, direction=ForwardScanDirection, count=0, execute_once=true) at execMain.c:308
No locals.
#15 0x00000000009f4928 in PortalRunSelect (portal=0x80b0c3118, forward=true, count=0, dest=0x80b565f00) at pquery.c:912
queryDesc = 0x80220e918
direction = ForwardScanDirection
nprocessed = 294333
#16 0x00000000009f438d in PortalRun (portal=0x80b0c3118, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x80b565f00, altdest=0x80b565f00, qc=0x7fffffffe3a0) at pquery.c:756
_save_exception_stack = 0x7fffffffe580
_save_context_stack = 0x0
_local_sigjmp_buf = {{_sjb = {10437281, -4571079317099439115, 140737488347448, 140737488347776, 140737488350016, 0, 140737488350024, 140737488350056, -64641, 2, 4294967297, 34359738368}}}
_do_rethrow = false
result = false
nprocessed = 34395449568
saveTopTransactionResourceOwner = 0x80b0ae3a0
saveTopTransactionContext = 0x80b0cb000
saveActivePortal = 0x0
saveResourceOwner = 0x80b0ae3a0
savePortalContext = 0x0
saveMemoryContext = 0x80b0cb000
#17 0x00000000009efc10 in exec_simple_query (
query_string=0x801b87118 "SELECT COUNT(*) FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"...) at postgres.c:1239
snapshot_set = true
per_parsetree_context = 0x0
plantree_list = 0x80b565eb0
parsetree = 0x801b8ecd0
commandTag = CMDTAG_SELECT
qc = {commandTag = CMDTAG_UNKNOWN, nprocessed = 0}
querytree_list = 0x801bb9160
portal = 0x80b0c3118
receiver = 0x80b565f00
format = 0
parsetree_item__state = {l = 0x801b8ed00, i = 0}
dest = DestRemote
oldcontext = 0x80b0cb000
parsetree_list = 0x801b8ed00
parsetree_item = 0x801b8ed18
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str = "\340\343\377\377\377\177\000\000\022S\201\000\000\000\000\000\377q\t\000\000\000\000\000\365\033'h\356\001\000"
#18 0x00000000009eeec9 in PostgresMain (argc=1, argv=0x80b0ac730, dbname=0x80b0ac6a8 "repology", username=0x80b0ac688 "repology") at postgres.c:4315
query_string = 0x801b87118 "SELECT COUNT(*) FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"...
firstchar = 81
input_message = {
data = 0x801b87118 "SELECT COUNT(*) FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"..., len = 495, maxlen = 1024, cursor = 495}
local_sigjmp_buf = {{_sjb = {10414768, 3, 140737488348136, 140737488348704, 140737488350016, 0, 140737488350024, 140737488350056, 34388575103, 0, 0, 1}}}
send_ready_for_query = false
disable_idle_in_transaction_timeout = false
#19 0x000000000092bcb7 in BackendRun (port=0x80b0a5000) at postmaster.c:4536
av = 0x80b0ac730
maxac = 2
ac = 1
i = 1
#20 0x000000000092b090 in BackendStartup (port=0x80b0a5000) at postmaster.c:4220
bn = 0x801b97510
pid = 0
#21 0x000000000092a073 in ServerLoop () at postmaster.c:1739
port = 0x80b0a5000
i = 2
rmask = {__fds_bits = {256, 0 <repeats 15 times>}}
selres = 1
now = 1604363041
readmask = {__fds_bits = {448, 0 <repeats 15 times>}}
nSockets = 9
last_lockfile_recheck_time = 1604363041
last_touch_time = 1604363041
#22 0x0000000000927d52 in PostmasterMain (argc=3, argv=0x7fffffffeb48) at postmaster.c:1412
opt = -1
status = 0
userDoption = 0x800c881a0 "/var/db/postgres/data13"
listen_addr_saved = true
i = 64
output_config_variable = 0x0
#23 0x0000000000819bcf in main (argc=3, argv=0x7fffffffeb48) at main.c:210
do_check_root = true

Disassembly:

Dump of assembler code for function evalexpr_0_2:
0x0000000810f5c120 <+0>: push %rbp
0x0000000810f5c121 <+1>: mov %rsp,%rbp
0x0000000810f5c124 <+4>: push %r15
0x0000000810f5c126 <+6>: push %r14
0x0000000810f5c128 <+8>: push %r13
0x0000000810f5c12a <+10>: push %r12
0x0000000810f5c12c <+12>: push %rbx
0x0000000810f5c12d <+13>: sub $0x48,%rsp
0x0000000810f5c131 <+17>: mov %rdx,-0x68(%rbp)
0x0000000810f5c135 <+21>: mov %rdi,%r12
0x0000000810f5c138 <+24>: movabs $0x80b18a855,%r13
0x0000000810f5c142 <+34>: mov 0x18(%rsi),%rax
0x0000000810f5c146 <+38>: mov 0x10(%rdi),%rcx
0x0000000810f5c14a <+42>: mov 0x18(%rax),%r14
0x0000000810f5c14e <+46>: mov 0x20(%rax),%r15
0x0000000810f5c152 <+50>: mov 0x18(%rcx),%rbx
0x0000000810f5c156 <+54>: mov 0x20(%rcx),%rdx
0x0000000810f5c15a <+58>: mov (%r14),%rax
0x0000000810f5c15d <+61>: mov (%r15),%cl
0x0000000810f5c160 <+64>: mov %rax,0xf3(%r13)
0x0000000810f5c167 <+71>: mov %cl,0xfb(%r13)
0x0000000810f5c16e <+78>: movb $0x1,0x0(%r13)
0x0000000810f5c173 <+83>: cmp $0x1,%cl
0x0000000810f5c176 <+86>: mov %rdx,-0x30(%rbp)
0x0000000810f5c17a <+90>: je 0x810f5c1b4 <evalexpr_0_2+148>
0x0000000810f5c17c <+92>: cmpb $0x1,0x10b(%r13)
0x0000000810f5c184 <+100>: je 0x810f5c1b4 <evalexpr_0_2+148>
0x0000000810f5c186 <+102>: movb $0x0,0xef(%r13)
0x0000000810f5c18e <+110>: lea 0xd3(%r13),%rdi
0x0000000810f5c195 <+117>: movabs $0xaa0b40,%rax
0x0000000810f5c19f <+127>: callq *%rax
0x0000000810f5c1a1 <+129>: mov -0x30(%rbp),%rdx
0x0000000810f5c1a5 <+133>: mov 0xef(%r13),%cl
0x0000000810f5c1ac <+140>: mov %rax,0x3(%r13)
0x0000000810f5c1b0 <+144>: mov %cl,0x0(%r13)
0x0000000810f5c1b4 <+148>: mov 0x8(%r12),%rax
0x0000000810f5c1b9 <+153>: mov 0x5(%r12),%cl
0x0000000810f5c1be <+158>: mov %cl,(%rdx)
0x0000000810f5c1c0 <+160>: test %cl,%cl
0x0000000810f5c1c2 <+162>: jne 0x810f5c1da <evalexpr_0_2+186>
0x0000000810f5c1c4 <+164>: cmpb $0x1,(%rax)
0x0000000810f5c1c7 <+167>: jne 0x810f5c1d7 <evalexpr_0_2+183>
0x0000000810f5c1c9 <+169>: cmpb $0x3,0x1(%rax)
0x0000000810f5c1cd <+173>: jne 0x810f5c1d7 <evalexpr_0_2+183>
0x0000000810f5c1cf <+175>: mov 0x2(%rax),%rax
0x0000000810f5c1d3 <+179>: add $0x22,%rax
0x0000000810f5c1d7 <+183>: mov %rax,(%rbx)
0x0000000810f5c1da <+186>: mov (%r14),%rax
0x0000000810f5c1dd <+189>: mov (%r15),%cl
0x0000000810f5c1e0 <+192>: mov %rax,0x5a3(%r13)
0x0000000810f5c1e7 <+199>: mov %cl,0x5ab(%r13)
0x0000000810f5c1ee <+206>: movb $0x1,0x0(%r13)
0x0000000810f5c1f3 <+211>: cmp $0x1,%cl
0x0000000810f5c1f6 <+214>: je 0x810f5c230 <evalexpr_0_2+272>
0x0000000810f5c1f8 <+216>: cmpb $0x1,0x5bb(%r13)
0x0000000810f5c200 <+224>: je 0x810f5c230 <evalexpr_0_2+272>
0x0000000810f5c202 <+226>: movb $0x0,0x59f(%r13)
0x0000000810f5c20a <+234>: lea 0x583(%r13),%rdi
0x0000000810f5c211 <+241>: movabs $0xaa0b40,%rax
0x0000000810f5c21b <+251>: callq *%rax
0x0000000810f5c21d <+253>: mov -0x30(%rbp),%rdx
0x0000000810f5c221 <+257>: mov 0x59f(%r13),%cl
0x0000000810f5c228 <+264>: mov %rax,0x3(%r13)
0x0000000810f5c22c <+268>: mov %cl,0x0(%r13)
0x0000000810f5c230 <+272>: mov 0x8(%r12),%rax
0x0000000810f5c235 <+277>: mov 0x5(%r12),%cl
0x0000000810f5c23a <+282>: mov %cl,0x1(%rdx)
0x0000000810f5c23d <+285>: test %cl,%cl
0x0000000810f5c23f <+287>: jne 0x810f5c258 <evalexpr_0_2+312>
0x0000000810f5c241 <+289>: cmpb $0x1,(%rax)
0x0000000810f5c244 <+292>: jne 0x810f5c254 <evalexpr_0_2+308>
0x0000000810f5c246 <+294>: cmpb $0x3,0x1(%rax)
0x0000000810f5c24a <+298>: jne 0x810f5c254 <evalexpr_0_2+308>
0x0000000810f5c24c <+300>: mov 0x2(%rax),%rax
0x0000000810f5c250 <+304>: add $0x22,%rax
0x0000000810f5c254 <+308>: mov %rax,0x8(%rbx)
0x0000000810f5c258 <+312>: mov (%r14),%rax
0x0000000810f5c25b <+315>: mov (%r15),%cl
0x0000000810f5c25e <+318>: mov %rax,0x643(%r13)
0x0000000810f5c265 <+325>: mov %cl,0x64b(%r13)
0x0000000810f5c26c <+332>: movb $0x1,0x0(%r13)
0x0000000810f5c271 <+337>: cmp $0x1,%cl
0x0000000810f5c274 <+340>: je 0x810f5c2ae <evalexpr_0_2+398>
0x0000000810f5c276 <+342>: cmpb $0x1,0x65b(%r13)
0x0000000810f5c27e <+350>: je 0x810f5c2ae <evalexpr_0_2+398>
0x0000000810f5c280 <+352>: movb $0x0,0x63f(%r13)
0x0000000810f5c288 <+360>: lea 0x623(%r13),%rdi
0x0000000810f5c28f <+367>: movabs $0xaa0b40,%rax
0x0000000810f5c299 <+377>: callq *%rax
0x0000000810f5c29b <+379>: mov -0x30(%rbp),%rdx
0x0000000810f5c29f <+383>: mov 0x63f(%r13),%cl
0x0000000810f5c2a6 <+390>: mov %rax,0x3(%r13)
0x0000000810f5c2aa <+394>: mov %cl,0x0(%r13)
0x0000000810f5c2ae <+398>: mov 0x8(%r12),%rax
0x0000000810f5c2b3 <+403>: mov 0x5(%r12),%cl
0x0000000810f5c2b8 <+408>: mov %cl,0x2(%rdx)
0x0000000810f5c2bb <+411>: test %cl,%cl
0x0000000810f5c2bd <+413>: jne 0x810f5c2d6 <evalexpr_0_2+438>
0x0000000810f5c2bf <+415>: cmpb $0x1,(%rax)
0x0000000810f5c2c2 <+418>: jne 0x810f5c2d2 <evalexpr_0_2+434>
0x0000000810f5c2c4 <+420>: cmpb $0x3,0x1(%rax)
0x0000000810f5c2c8 <+424>: jne 0x810f5c2d2 <evalexpr_0_2+434>
0x0000000810f5c2ca <+426>: mov 0x2(%rax),%rax
0x0000000810f5c2ce <+430>: add $0x22,%rax
0x0000000810f5c2d2 <+434>: mov %rax,0x10(%rbx)
0x0000000810f5c2d6 <+438>: mov (%r14),%rax
0x0000000810f5c2d9 <+441>: mov (%r15),%cl
0x0000000810f5c2dc <+444>: mov %rax,0x6e3(%r13)
0x0000000810f5c2e3 <+451>: mov %cl,0x6eb(%r13)
0x0000000810f5c2ea <+458>: movb $0x1,0x0(%r13)
0x0000000810f5c2ef <+463>: cmp $0x1,%cl
0x0000000810f5c2f2 <+466>: je 0x810f5c32c <evalexpr_0_2+524>
0x0000000810f5c2f4 <+468>: cmpb $0x1,0x6fb(%r13)
0x0000000810f5c2fc <+476>: je 0x810f5c32c <evalexpr_0_2+524>
0x0000000810f5c2fe <+478>: movb $0x0,0x6df(%r13)
0x0000000810f5c306 <+486>: lea 0x6c3(%r13),%rdi
0x0000000810f5c30d <+493>: movabs $0xaa0b40,%rax
0x0000000810f5c317 <+503>: callq *%rax
0x0000000810f5c319 <+505>: mov -0x30(%rbp),%rdx
0x0000000810f5c31d <+509>: mov 0x6df(%r13),%cl
0x0000000810f5c324 <+516>: mov %rax,0x3(%r13)
0x0000000810f5c328 <+520>: mov %cl,0x0(%r13)
0x0000000810f5c32c <+524>: mov 0x8(%r12),%rax
0x0000000810f5c331 <+529>: mov 0x5(%r12),%cl
0x0000000810f5c336 <+534>: mov %cl,0x3(%rdx)
0x0000000810f5c339 <+537>: test %cl,%cl
0x0000000810f5c33b <+539>: jne 0x810f5c354 <evalexpr_0_2+564>
0x0000000810f5c33d <+541>: cmpb $0x1,(%rax)
0x0000000810f5c340 <+544>: jne 0x810f5c350 <evalexpr_0_2+560>
0x0000000810f5c342 <+546>: cmpb $0x3,0x1(%rax)
0x0000000810f5c346 <+550>: jne 0x810f5c350 <evalexpr_0_2+560>
0x0000000810f5c348 <+552>: mov 0x2(%rax),%rax
0x0000000810f5c34c <+556>: add $0x22,%rax
0x0000000810f5c350 <+560>: mov %rax,0x18(%rbx)
0x0000000810f5c354 <+564>: mov (%r14),%rax
0x0000000810f5c357 <+567>: mov (%r15),%cl
0x0000000810f5c35a <+570>: mov %rax,0x783(%r13)
0x0000000810f5c361 <+577>: mov %cl,0x78b(%r13)
0x0000000810f5c368 <+584>: movb $0x1,0x0(%r13)
0x0000000810f5c36d <+589>: cmp $0x1,%cl
0x0000000810f5c370 <+592>: je 0x810f5c3aa <evalexpr_0_2+650>
0x0000000810f5c372 <+594>: cmpb $0x1,0x79b(%r13)
0x0000000810f5c37a <+602>: je 0x810f5c3aa <evalexpr_0_2+650>
0x0000000810f5c37c <+604>: movb $0x0,0x77f(%r13)
0x0000000810f5c384 <+612>: lea 0x763(%r13),%rdi
0x0000000810f5c38b <+619>: movabs $0xaa0b40,%rax
0x0000000810f5c395 <+629>: callq *%rax
0x0000000810f5c397 <+631>: mov -0x30(%rbp),%rdx
0x0000000810f5c39b <+635>: mov 0x77f(%r13),%cl
0x0000000810f5c3a2 <+642>: mov %rax,0x3(%r13)
0x0000000810f5c3a6 <+646>: mov %cl,0x0(%r13)
0x0000000810f5c3aa <+650>: mov 0x8(%r12),%rax
0x0000000810f5c3af <+655>: mov 0x5(%r12),%cl
0x0000000810f5c3b4 <+660>: mov %cl,0x4(%rdx)
0x0000000810f5c3b7 <+663>: test %cl,%cl
0x0000000810f5c3b9 <+665>: jne 0x810f5c3d2 <evalexpr_0_2+690>
0x0000000810f5c3bb <+667>: cmpb $0x1,(%rax)
0x0000000810f5c3be <+670>: jne 0x810f5c3ce <evalexpr_0_2+686>
0x0000000810f5c3c0 <+672>: cmpb $0x3,0x1(%rax)
0x0000000810f5c3c4 <+676>: jne 0x810f5c3ce <evalexpr_0_2+686>
0x0000000810f5c3c6 <+678>: mov 0x2(%rax),%rax
0x0000000810f5c3ca <+682>: add $0x22,%rax
0x0000000810f5c3ce <+686>: mov %rax,0x20(%rbx)
0x0000000810f5c3d2 <+690>: mov (%r14),%rax
0x0000000810f5c3d5 <+693>: mov (%r15),%cl
0x0000000810f5c3d8 <+696>: mov %rax,0x823(%r13)
0x0000000810f5c3df <+703>: mov %cl,0x82b(%r13)
0x0000000810f5c3e6 <+710>: movb $0x1,0x0(%r13)
0x0000000810f5c3eb <+715>: cmp $0x1,%cl
0x0000000810f5c3ee <+718>: je 0x810f5c428 <evalexpr_0_2+776>
0x0000000810f5c3f0 <+720>: cmpb $0x1,0x83b(%r13)
0x0000000810f5c3f8 <+728>: je 0x810f5c428 <evalexpr_0_2+776>
0x0000000810f5c3fa <+730>: movb $0x0,0x81f(%r13)
0x0000000810f5c402 <+738>: lea 0x803(%r13),%rdi
0x0000000810f5c409 <+745>: movabs $0xaa0b40,%rax
0x0000000810f5c413 <+755>: callq *%rax
0x0000000810f5c415 <+757>: mov -0x30(%rbp),%rdx
0x0000000810f5c419 <+761>: mov 0x81f(%r13),%cl
0x0000000810f5c420 <+768>: mov %rax,0x3(%r13)
0x0000000810f5c424 <+772>: mov %cl,0x0(%r13)
0x0000000810f5c428 <+776>: mov 0x8(%r12),%rax
0x0000000810f5c42d <+781>: mov 0x5(%r12),%cl
0x0000000810f5c432 <+786>: mov %cl,0x5(%rdx)
0x0000000810f5c435 <+789>: test %cl,%cl
0x0000000810f5c437 <+791>: jne 0x810f5c450 <evalexpr_0_2+816>
0x0000000810f5c439 <+793>: cmpb $0x1,(%rax)
0x0000000810f5c43c <+796>: jne 0x810f5c44c <evalexpr_0_2+812>
0x0000000810f5c43e <+798>: cmpb $0x3,0x1(%rax)
0x0000000810f5c442 <+802>: jne 0x810f5c44c <evalexpr_0_2+812>
0x0000000810f5c444 <+804>: mov 0x2(%rax),%rax
0x0000000810f5c448 <+808>: add $0x22,%rax
0x0000000810f5c44c <+812>: mov %rax,0x28(%rbx)
0x0000000810f5c450 <+816>: mov (%r14),%rax
0x0000000810f5c453 <+819>: mov (%r15),%cl
0x0000000810f5c456 <+822>: mov %rax,0x10d3(%r13)
0x0000000810f5c45d <+829>: mov %cl,0x10db(%r13)
0x0000000810f5c464 <+836>: movb $0x1,0x0(%r13)
0x0000000810f5c469 <+841>: cmp $0x1,%cl
0x0000000810f5c46c <+844>: je 0x810f5c4a6 <evalexpr_0_2+902>
0x0000000810f5c46e <+846>: cmpb $0x1,0x10eb(%r13)
0x0000000810f5c476 <+854>: je 0x810f5c4a6 <evalexpr_0_2+902>
0x0000000810f5c478 <+856>: movb $0x0,0x10cf(%r13)
0x0000000810f5c480 <+864>: lea 0x10b3(%r13),%rdi
0x0000000810f5c487 <+871>: movabs $0xaa0b40,%rax
0x0000000810f5c491 <+881>: callq *%rax
0x0000000810f5c493 <+883>: mov -0x30(%rbp),%rdx
0x0000000810f5c497 <+887>: mov 0x10cf(%r13),%cl
0x0000000810f5c49e <+894>: mov %rax,0x3(%r13)
0x0000000810f5c4a2 <+898>: mov %cl,0x0(%r13)
0x0000000810f5c4a6 <+902>: mov 0x8(%r12),%rax
0x0000000810f5c4ab <+907>: mov 0x5(%r12),%cl
0x0000000810f5c4b0 <+912>: mov %cl,0x6(%rdx)
0x0000000810f5c4b3 <+915>: test %cl,%cl
0x0000000810f5c4b5 <+917>: jne 0x810f5c4ce <evalexpr_0_2+942>
0x0000000810f5c4b7 <+919>: cmpb $0x1,(%rax)
0x0000000810f5c4ba <+922>: jne 0x810f5c4ca <evalexpr_0_2+938>
0x0000000810f5c4bc <+924>: cmpb $0x3,0x1(%rax)
0x0000000810f5c4c0 <+928>: jne 0x810f5c4ca <evalexpr_0_2+938>
0x0000000810f5c4c2 <+930>: mov 0x2(%rax),%rax
0x0000000810f5c4c6 <+934>: add $0x22,%rax
0x0000000810f5c4ca <+938>: mov %rax,0x30(%rbx)
0x0000000810f5c4ce <+942>: mov (%r14),%rax
0x0000000810f5c4d1 <+945>: mov (%r15),%cl
0x0000000810f5c4d4 <+948>: mov %rax,0x1173(%r13)
0x0000000810f5c4db <+955>: mov %cl,0x117b(%r13)
0x0000000810f5c4e2 <+962>: movb $0x1,0x0(%r13)
0x0000000810f5c4e7 <+967>: cmp $0x1,%cl
0x0000000810f5c4ea <+970>: je 0x810f5c524 <evalexpr_0_2+1028>
0x0000000810f5c4ec <+972>: cmpb $0x1,0x118b(%r13)
0x0000000810f5c4f4 <+980>: je 0x810f5c524 <evalexpr_0_2+1028>
0x0000000810f5c4f6 <+982>: movb $0x0,0x116f(%r13)
0x0000000810f5c4fe <+990>: lea 0x1153(%r13),%rdi
0x0000000810f5c505 <+997>: movabs $0xaa0b40,%rax
0x0000000810f5c50f <+1007>: callq *%rax
0x0000000810f5c511 <+1009>: mov -0x30(%rbp),%rdx
0x0000000810f5c515 <+1013>: mov 0x116f(%r13),%cl
0x0000000810f5c51c <+1020>: mov %rax,0x3(%r13)
0x0000000810f5c520 <+1024>: mov %cl,0x0(%r13)
0x0000000810f5c524 <+1028>: mov 0x8(%r12),%rax
0x0000000810f5c529 <+1033>: mov 0x5(%r12),%cl
0x0000000810f5c52e <+1038>: mov %cl,0x7(%rdx)
0x0000000810f5c531 <+1041>: test %cl,%cl
0x0000000810f5c533 <+1043>: jne 0x810f5c54c <evalexpr_0_2+1068>
0x0000000810f5c535 <+1045>: cmpb $0x1,(%rax)
0x0000000810f5c538 <+1048>: jne 0x810f5c548 <evalexpr_0_2+1064>
0x0000000810f5c53a <+1050>: cmpb $0x3,0x1(%rax)
0x0000000810f5c53e <+1054>: jne 0x810f5c548 <evalexpr_0_2+1064>
0x0000000810f5c540 <+1056>: mov 0x2(%rax),%rax
0x0000000810f5c544 <+1060>: add $0x22,%rax
0x0000000810f5c548 <+1064>: mov %rax,0x38(%rbx)
0x0000000810f5c54c <+1068>: mov (%r14),%rax
0x0000000810f5c54f <+1071>: mov (%r15),%cl
0x0000000810f5c552 <+1074>: mov %rax,0x1213(%r13)
0x0000000810f5c559 <+1081>: mov %cl,0x121b(%r13)
0x0000000810f5c560 <+1088>: movb $0x1,0x0(%r13)
0x0000000810f5c565 <+1093>: cmp $0x1,%cl
0x0000000810f5c568 <+1096>: je 0x810f5c5a2 <evalexpr_0_2+1154>
0x0000000810f5c56a <+1098>: cmpb $0x1,0x122b(%r13)
0x0000000810f5c572 <+1106>: je 0x810f5c5a2 <evalexpr_0_2+1154>
0x0000000810f5c574 <+1108>: movb $0x0,0x120f(%r13)
0x0000000810f5c57c <+1116>: lea 0x11f3(%r13),%rdi
0x0000000810f5c583 <+1123>: movabs $0xaa0b40,%rax
0x0000000810f5c58d <+1133>: callq *%rax
0x0000000810f5c58f <+1135>: mov -0x30(%rbp),%rdx
0x0000000810f5c593 <+1139>: mov 0x120f(%r13),%cl
0x0000000810f5c59a <+1146>: mov %rax,0x3(%r13)
0x0000000810f5c59e <+1150>: mov %cl,0x0(%r13)
0x0000000810f5c5a2 <+1154>: mov 0x8(%r12),%rax
0x0000000810f5c5a7 <+1159>: mov 0x5(%r12),%cl
0x0000000810f5c5ac <+1164>: mov %cl,0x8(%rdx)
0x0000000810f5c5af <+1167>: test %cl,%cl
0x0000000810f5c5b1 <+1169>: jne 0x810f5c5ca <evalexpr_0_2+1194>
0x0000000810f5c5b3 <+1171>: cmpb $0x1,(%rax)
0x0000000810f5c5b6 <+1174>: jne 0x810f5c5c6 <evalexpr_0_2+1190>
0x0000000810f5c5b8 <+1176>: cmpb $0x3,0x1(%rax)
0x0000000810f5c5bc <+1180>: jne 0x810f5c5c6 <evalexpr_0_2+1190>
0x0000000810f5c5be <+1182>: mov 0x2(%rax),%rax
0x0000000810f5c5c2 <+1186>: add $0x22,%rax
0x0000000810f5c5c6 <+1190>: mov %rax,0x40(%rbx)
0x0000000810f5c5ca <+1194>: mov (%r14),%rax
0x0000000810f5c5cd <+1197>: mov (%r15),%cl
0x0000000810f5c5d0 <+1200>: mov %rax,0x12b3(%r13)
0x0000000810f5c5d7 <+1207>: mov %cl,0x12bb(%r13)
0x0000000810f5c5de <+1214>: movb $0x1,0x0(%r13)
0x0000000810f5c5e3 <+1219>: cmp $0x1,%cl
0x0000000810f5c5e6 <+1222>: je 0x810f5c620 <evalexpr_0_2+1280>
0x0000000810f5c5e8 <+1224>: cmpb $0x1,0x12cb(%r13)
0x0000000810f5c5f0 <+1232>: je 0x810f5c620 <evalexpr_0_2+1280>
0x0000000810f5c5f2 <+1234>: movb $0x0,0x12af(%r13)
0x0000000810f5c5fa <+1242>: lea 0x1293(%r13),%rdi
0x0000000810f5c601 <+1249>: movabs $0xaa0b40,%rax
0x0000000810f5c60b <+1259>: callq *%rax
0x0000000810f5c60d <+1261>: mov -0x30(%rbp),%rdx
0x0000000810f5c611 <+1265>: mov 0x12af(%r13),%cl
0x0000000810f5c618 <+1272>: mov %rax,0x3(%r13)
0x0000000810f5c61c <+1276>: mov %cl,0x0(%r13)
0x0000000810f5c620 <+1280>: mov 0x8(%r12),%rax
0x0000000810f5c625 <+1285>: mov 0x5(%r12),%cl
0x0000000810f5c62a <+1290>: mov %cl,0x9(%rdx)
0x0000000810f5c62d <+1293>: test %cl,%cl
0x0000000810f5c62f <+1295>: jne 0x810f5c648 <evalexpr_0_2+1320>
0x0000000810f5c631 <+1297>: cmpb $0x1,(%rax)
0x0000000810f5c634 <+1300>: jne 0x810f5c644 <evalexpr_0_2+1316>
0x0000000810f5c636 <+1302>: cmpb $0x3,0x1(%rax)
0x0000000810f5c63a <+1306>: jne 0x810f5c644 <evalexpr_0_2+1316>
0x0000000810f5c63c <+1308>: mov 0x2(%rax),%rax
0x0000000810f5c640 <+1312>: add $0x22,%rax
0x0000000810f5c644 <+1316>: mov %rax,0x48(%rbx)
0x0000000810f5c648 <+1320>: mov %rbx,-0x58(%rbp)
0x0000000810f5c64c <+1324>: mov (%r14),%rax
0x0000000810f5c64f <+1327>: mov (%r15),%cl
0x0000000810f5c652 <+1330>: mov %rax,0x1353(%r13)
0x0000000810f5c659 <+1337>: mov %cl,0x135b(%r13)
0x0000000810f5c660 <+1344>: movb $0x1,0x0(%r13)
0x0000000810f5c665 <+1349>: cmp $0x1,%cl
0x0000000810f5c668 <+1352>: je 0x810f5c7e3 <evalexpr_0_2+1731>
0x0000000810f5c66e <+1358>: cmpb $0x1,0x136b(%r13)
0x0000000810f5c676 <+1366>: je 0x810f5c7e3 <evalexpr_0_2+1731>
0x0000000810f5c67c <+1372>: movb $0x0,0x134f(%r13)
0x0000000810f5c684 <+1380>: lea 0x1333(%r13),%rdi
0x0000000810f5c68b <+1387>: movabs $0xaa0b40,%rax
0x0000000810f5c695 <+1397>: callq *%rax
0x0000000810f5c697 <+1399>: mov -0x30(%rbp),%rdx
0x0000000810f5c69b <+1403>: mov 0x134f(%r13),%bl
0x0000000810f5c6a2 <+1410>: mov %rax,0x3(%r13)
0x0000000810f5c6a6 <+1414>: mov %bl,0x0(%r13)
0x0000000810f5c6aa <+1418>: cmp $0x1,%bl
0x0000000810f5c6ad <+1421>: je 0x810f5c7e3 <evalexpr_0_2+1731>
0x0000000810f5c6b3 <+1427>: mov 0x3(%r13),%rax
0x0000000810f5c6b7 <+1431>: mov %rax,0x13f3(%r13)
0x0000000810f5c6be <+1438>: movb $0x0,0x13fb(%r13)
0x0000000810f5c6c6 <+1446>: lea 0x13d3(%r13),%rdi
0x0000000810f5c6cd <+1453>: movabs $0xb7cfc0,%rax
0x0000000810f5c6d7 <+1463>: callq *%rax
0x0000000810f5c6d9 <+1465>: mov -0x30(%rbp),%rdx
0x0000000810f5c6dd <+1469>: test %rax,%rax
0x0000000810f5c6e0 <+1472>: je 0x810f5c7e3 <evalexpr_0_2+1731>
0x0000000810f5c6e6 <+1478>: mov %rax,0x1493(%r13)
0x0000000810f5c6ed <+1485>: mov %bl,0x149b(%r13)
0x0000000810f5c6f4 <+1492>: movb $0x0,0x148f(%r13)
0x0000000810f5c6fc <+1500>: movzbl (%rax),%esi
0x0000000810f5c6ff <+1503>: movabs $0xc85a04,%rcx
0x0000000810f5c709 <+1513>: mov (%rcx),%edi
0x0000000810f5c70b <+1515>: mov 0x1b178bc(%rip),%rcx # 0x812a73fce
0x0000000810f5c712 <+1522>: mov %fs:0x0,%rdx
0x0000000810f5c71b <+1531>: mov (%rdx,%rcx,1),%rbx
0x0000000810f5c71f <+1535>: cmp %esi,%edi
0x0000000810f5c721 <+1537>: mov %rax,-0x60(%rbp)
0x0000000810f5c725 <+1541>: mov %edi,-0x48(%rbp)
0x0000000810f5c728 <+1544>: mov %rbx,-0x40(%rbp)
0x0000000810f5c72c <+1548>: jle 0x810f5c769 <evalexpr_0_2+1609>
0x0000000810f5c72e <+1550>: test %rbx,%rbx
0x0000000810f5c731 <+1553>: movabs $0xc85a10,%rcx
0x0000000810f5c73b <+1563>: mov (%rcx),%rdx
0x0000000810f5c73e <+1566>: mov %rbx,%rcx
0x0000000810f5c741 <+1569>: mov %rdx,%r8
0x0000000810f5c744 <+1572>: cmove %rdx,%rcx
0x0000000810f5c748 <+1576>: mov %rax,%rbx
0x0000000810f5c74b <+1579>: mov %rdx,-0x50(%rbp)
0x0000000810f5c74f <+1583>: nop
0x0000000810f5c750 <+1584>: movzbl %sil,%edx
0x0000000810f5c754 <+1588>: testb $0x40,0x41(%rcx,%rdx,8)
0x0000000810f5c759 <+1593>: je 0x810f5c77d <evalexpr_0_2+1629>
0x0000000810f5c75b <+1595>: movzbl 0x1(%rbx),%esi
0x0000000810f5c75f <+1599>: add $0x1,%rbx
0x0000000810f5c763 <+1603>: cmp %esi,%edi
0x0000000810f5c765 <+1605>: jg 0x810f5c750 <evalexpr_0_2+1584>
0x0000000810f5c767 <+1607>: jmp 0x810f5c77d <evalexpr_0_2+1629>
0x0000000810f5c769 <+1609>: movabs $0xc85a10,%rcx
0x0000000810f5c773 <+1619>: mov (%rcx),%rcx
0x0000000810f5c776 <+1622>: mov %rcx,-0x50(%rbp)
0x0000000810f5c77a <+1626>: mov %rax,%rbx
0x0000000810f5c77d <+1629>: movabs $0x801769f90,%rax
0x0000000810f5c787 <+1639>: mov %rbx,%rdi
0x0000000810f5c78a <+1642>: callq *%rax
0x0000000810f5c78c <+1644>: mov -0x40(%rbp),%rcx
0x0000000810f5c790 <+1648>: test %rcx,%rcx
0x0000000810f5c793 <+1651>: mov -0x50(%rbp),%rdi
0x0000000810f5c797 <+1655>: cmovne %rcx,%rdi
0x0000000810f5c79b <+1659>: mov -0x48(%rbp),%edx
0x0000000810f5c79e <+1662>: xchg %ax,%ax
0x0000000810f5c7a0 <+1664>: mov %rax,%rsi
0x0000000810f5c7a3 <+1667>: test %rax,%rax
0x0000000810f5c7a6 <+1670>: je 0x810f5c7bc <evalexpr_0_2+1692>
0x0000000810f5c7a8 <+1672>: movzbl -0x1(%rbx,%rsi,1),%ecx
0x0000000810f5c7ad <+1677>: cmp %ecx,%edx
0x0000000810f5c7af <+1679>: jle 0x810f5c7bc <evalexpr_0_2+1692>
0x0000000810f5c7b1 <+1681>: lea -0x1(%rsi),%rax
0x0000000810f5c7b5 <+1685>: testb $0x40,0x41(%rdi,%rcx,8)
0x0000000810f5c7ba <+1690>: jne 0x810f5c7a0 <evalexpr_0_2+1664>
0x0000000810f5c7bc <+1692>: movabs $0xa3acc0,%rax
0x0000000810f5c7c6 <+1702>: lea -0x32(%rbp),%rdx
0x0000000810f5c7ca <+1706>: mov %rbx,%rdi
0x0000000810f5c7cd <+1709>: callq *%rax
0x0000000810f5c7cf <+1711>: test %al,%al
0x0000000810f5c7d1 <+1713>: je 0x810f5c9cc <evalexpr_0_2+2220>
0x0000000810f5c7d7 <+1719>: movzbl -0x32(%rbp),%eax
0x0000000810f5c7db <+1723>: mov %rax,0x3(%r13)
0x0000000810f5c7df <+1727>: mov -0x30(%rbp),%rdx
0x0000000810f5c7e3 <+1731>: mov 0x8(%r12),%rax
0x0000000810f5c7e8 <+1736>: mov 0x5(%r12),%cl
0x0000000810f5c7ed <+1741>: mov %cl,0xa(%rdx)
0x0000000810f5c7f0 <+1744>: mov -0x58(%rbp),%rcx
0x0000000810f5c7f4 <+1748>: mov %rax,0x50(%rcx)
0x0000000810f5c7f8 <+1752>: mov (%r14),%rax
0x0000000810f5c7fb <+1755>: mov (%r15),%cl
0x0000000810f5c7fe <+1758>: mov %rax,0x31e763(%r13)
0x0000000810f5c805 <+1765>: mov %cl,0x31e76b(%r13)
0x0000000810f5c80c <+1772>: movb $0x1,0x0(%r13)
0x0000000810f5c811 <+1777>: cmp $0x1,%cl
0x0000000810f5c814 <+1780>: je 0x810f5c995 <evalexpr_0_2+2165>
0x0000000810f5c81a <+1786>: cmpb $0x1,0x31e77b(%r13)
0x0000000810f5c822 <+1794>: je 0x810f5c995 <evalexpr_0_2+2165>
0x0000000810f5c828 <+1800>: movb $0x0,0x31e75f(%r13)
0x0000000810f5c830 <+1808>: lea 0x31e743(%r13),%rdi
0x0000000810f5c837 <+1815>: movabs $0xaa0b40,%rax
0x0000000810f5c841 <+1825>: callq *%rax
0x0000000810f5c843 <+1827>: mov -0x30(%rbp),%rdx
0x0000000810f5c847 <+1831>: mov 0x31e75f(%r13),%bl
0x0000000810f5c84e <+1838>: mov %rax,0x3(%r13)
0x0000000810f5c852 <+1842>: mov %bl,0x0(%r13)
0x0000000810f5c856 <+1846>: cmp $0x1,%bl
0x0000000810f5c859 <+1849>: je 0x810f5c995 <evalexpr_0_2+2165>
0x0000000810f5c85f <+1855>: mov 0x3(%r13),%rax
0x0000000810f5c863 <+1859>: mov %rax,0x31e803(%r13)
0x0000000810f5c86a <+1866>: movb $0x0,0x31e80b(%r13)
0x0000000810f5c872 <+1874>: lea 0x31e7e3(%r13),%rdi
0x0000000810f5c879 <+1881>: movabs $0xb7cfc0,%rax
0x0000000810f5c883 <+1891>: callq *%rax
0x0000000810f5c885 <+1893>: mov -0x30(%rbp),%rdx
0x0000000810f5c889 <+1897>: test %rax,%rax
0x0000000810f5c88c <+1900>: je 0x810f5c995 <evalexpr_0_2+2165>
0x0000000810f5c892 <+1906>: mov %rax,0x1733(%r13)
0x0000000810f5c899 <+1913>: mov %bl,0x173b(%r13)
0x0000000810f5c8a0 <+1920>: movb $0x0,0x172f(%r13)
0x0000000810f5c8a8 <+1928>: movzbl (%rax),%esi
0x0000000810f5c8ab <+1931>: movabs $0xc85a04,%rcx
0x0000000810f5c8b5 <+1941>: mov (%rcx),%r14d
0x0000000810f5c8b8 <+1944>: mov 0x1b178bc(%rip),%rcx # 0x812a7417b
0x0000000810f5c8bf <+1951>: mov %fs:0x0,%rdx
0x0000000810f5c8c8 <+1960>: mov (%rdx,%rcx,1),%rdi
0x0000000810f5c8cc <+1964>: cmp %esi,%r14d
0x0000000810f5c8cf <+1967>: mov %rax,-0x40(%rbp)
0x0000000810f5c8d3 <+1971>: mov %rdi,-0x48(%rbp)
0x0000000810f5c8d7 <+1975>: jle 0x810f5c91a <evalexpr_0_2+2042>
0x0000000810f5c8d9 <+1977>: test %rdi,%rdi
0x0000000810f5c8dc <+1980>: movabs $0xc85a10,%rcx
0x0000000810f5c8e6 <+1990>: mov (%rcx),%r15
0x0000000810f5c8e9 <+1993>: mov %rdi,%rcx
0x0000000810f5c8ec <+1996>: cmove %r15,%rcx
0x0000000810f5c8f0 <+2000>: mov %rax,%rbx
0x0000000810f5c8f3 <+2003>: nopw %cs:0x0(%rax,%rax,1)
0x0000000810f5c8fd <+2013>: nopl (%rax)
0x0000000810f5c900 <+2016>: movzbl %sil,%edx
0x0000000810f5c904 <+2020>: testb $0x40,0x41(%rcx,%rdx,8)
0x0000000810f5c909 <+2025>: je 0x810f5c92a <evalexpr_0_2+2058>
0x0000000810f5c90b <+2027>: movzbl 0x1(%rbx),%esi
0x0000000810f5c90f <+2031>: add $0x1,%rbx
0x0000000810f5c913 <+2035>: cmp %esi,%r14d
0x0000000810f5c916 <+2038>: jg 0x810f5c900 <evalexpr_0_2+2016>
0x0000000810f5c918 <+2040>: jmp 0x810f5c92a <evalexpr_0_2+2058>
0x0000000810f5c91a <+2042>: movabs $0xc85a10,%rcx
0x0000000810f5c924 <+2052>: mov (%rcx),%r15
0x0000000810f5c927 <+2055>: mov %rax,%rbx
0x0000000810f5c92a <+2058>: movabs $0x801769f90,%rax
0x0000000810f5c934 <+2068>: mov %rbx,%rdi
0x0000000810f5c937 <+2071>: callq *%rax
0x0000000810f5c939 <+2073>: mov -0x48(%rbp),%rcx
0x0000000810f5c93d <+2077>: test %rcx,%rcx
0x0000000810f5c940 <+2080>: cmovne %rcx,%r15
0x0000000810f5c944 <+2084>: nopw %cs:0x0(%rax,%rax,1)
0x0000000810f5c94e <+2094>: xchg %ax,%ax
0x0000000810f5c950 <+2096>: mov %rax,%rsi
0x0000000810f5c953 <+2099>: test %rax,%rax
0x0000000810f5c956 <+2102>: je 0x810f5c96e <evalexpr_0_2+2126>
0x0000000810f5c958 <+2104>: movzbl -0x1(%rbx,%rsi,1),%ecx
0x0000000810f5c95d <+2109>: cmp %ecx,%r14d
0x0000000810f5c960 <+2112>: jle 0x810f5c96e <evalexpr_0_2+2126>
0x0000000810f5c962 <+2114>: lea -0x1(%rsi),%rax
0x0000000810f5c966 <+2118>: testb $0x40,0x41(%r15,%rcx,8)
0x0000000810f5c96c <+2124>: jne 0x810f5c950 <evalexpr_0_2+2096>
0x0000000810f5c96e <+2126>: movabs $0xa3acc0,%rax
0x0000000810f5c978 <+2136>: lea -0x31(%rbp),%rdx
0x0000000810f5c97c <+2140>: mov %rbx,%rdi
0x0000000810f5c97f <+2143>: callq *%rax
0x0000000810f5c981 <+2145>: test %al,%al
0x0000000810f5c983 <+2147>: je 0x810f5ca14 <evalexpr_0_2+2292>
0x0000000810f5c989 <+2153>: movzbl -0x31(%rbp),%eax
0x0000000810f5c98d <+2157>: mov %rax,0x3(%r13)
0x0000000810f5c991 <+2161>: mov -0x30(%rbp),%rdx
0x0000000810f5c995 <+2165>: mov 0x8(%r12),%rax
0x0000000810f5c99a <+2170>: mov 0x5(%r12),%cl
0x0000000810f5c99f <+2175>: mov %cl,0xb(%rdx)
0x0000000810f5c9a2 <+2178>: mov -0x58(%rbp),%rcx
0x0000000810f5c9a6 <+2182>: mov %rax,0x58(%rcx)
0x0000000810f5c9aa <+2186>: mov 0x8(%r12),%rax
0x0000000810f5c9af <+2191>: mov 0x5(%r12),%cl
0x0000000810f5c9b4 <+2196>: and $0x1,%cl
0x0000000810f5c9b7 <+2199>: mov -0x68(%rbp),%rdx
0x0000000810f5c9bb <+2203>: mov %cl,(%rdx)
0x0000000810f5c9bd <+2205>: add $0x48,%rsp
0x0000000810f5c9c1 <+2209>: pop %rbx
0x0000000810f5c9c2 <+2210>: pop %r12
0x0000000810f5c9c4 <+2212>: pop %r13
0x0000000810f5c9c6 <+2214>: pop %r14
0x0000000810f5c9c8 <+2216>: pop %r15
0x0000000810f5c9ca <+2218>: pop %rbp
0x0000000810f5c9cb <+2219>: retq
0x0000000810f5c9cc <+2220>: movabs $0xbbf270,%rax
0x0000000810f5c9d6 <+2230>: mov $0x14,%edi
0x0000000810f5c9db <+2235>: xor %esi,%esi
0x0000000810f5c9dd <+2237>: callq *%rax
0x0000000810f5c9df <+2239>: movabs $0xbc0510,%rax
0x0000000810f5c9e9 <+2249>: mov $0x2020082,%edi
0x0000000810f5c9ee <+2254>: callq *%rax
0x0000000810f5c9f0 <+2256>: movabs $0x810f5d023,%rdi
0x0000000810f5c9fa <+2266>: movabs $0x810f5d04a,%rsi
0x0000000810f5ca04 <+2276>: movabs $0xbc0800,%rcx
0x0000000810f5ca0e <+2286>: mov -0x60(%rbp),%rdx
0x0000000810f5ca12 <+2290>: jmp 0x810f5ca5a <evalexpr_0_2+2362>
0x0000000810f5ca14 <+2292>: movabs $0xbbf270,%rax
0x0000000810f5ca1e <+2302>: mov $0x14,%edi
0x0000000810f5ca23 <+2307>: xor %esi,%esi
0x0000000810f5ca25 <+2309>: callq *%rax
0x0000000810f5ca27 <+2311>: movabs $0xbc0510,%rax
0x0000000810f5ca31 <+2321>: mov $0x2020082,%edi
0x0000000810f5ca36 <+2326>: callq *%rax
0x0000000810f5ca38 <+2328>: movabs $0x810f5d023,%rdi
0x0000000810f5ca42 <+2338>: movabs $0x810f5d04a,%rsi
0x0000000810f5ca4c <+2348>: movabs $0xbc0800,%rcx
0x0000000810f5ca56 <+2358>: mov -0x40(%rbp),%rdx
0x0000000810f5ca5a <+2362>: xor %eax,%eax
0x0000000810f5ca5c <+2364>: callq *%rcx
0x0000000810f5ca5e <+2366>: movabs $0x810f5d052,%rdi
0x0000000810f5ca68 <+2376>: movabs $0x810f5d059,%rdx
0x0000000810f5ca72 <+2386>: movabs $0xbbfc50,%rax
0x0000000810f5ca7c <+2396>: mov $0x9a,%esi
0x0000000810f5ca81 <+2401>: callq *%rax
End of assembler dump.

--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru ..: https://github.com/AMDmi3

* Andres Freund (andres@anarazel.de) wrote:

Environment details:
- FreeBSD 12.1 amd64
- PostgreSQL 13.0 (built from FreeBSD ports)
- llvm-10.0.1 (build from FreeBSD ports)

My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
the system, and PostgreSQL uses llvm9:

ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

I'll check once I'm home whether I can reproduce in my environment.

I did some digging. First of all, I've discovered that the problem
goes away if llvm bitcode optimization is disabled (by commenting out
llvm_optimize_module call).

I've dumped the opcode and tried compiling it back to match disassembly
of the failing function in gdb disassembly. It didn't match perfectly,
but this place looked similar:

# %bb.84: # %op.32.inputcall
movq %rax, 5267(%r13)
movb %bl, 5275(%r13)
movb $0, 5263(%r13)
movzbl (%rax), %esi
movl __mb_sb_limit(%rip), %edi
movq _ThreadRuneLocale@GOTTPOFF(%rip), %rcx
movq %fs:0, %rdx
movq (%rdx,%rcx), %rcx
cmpl %esi, %edi
movq %rax, -96(%rbp) # 8-byte Spill
movl %edi, -72(%rbp) # 4-byte Spill
movq %rcx, -64(%rbp) # 8-byte Spill
jle .LBB1_85

Here's my hypothesis:

The problem happens when boolin() function is inlined by LLVM.
The named function calls isspace() internally, which on FreeBSD is
locale-specific and involves caching some locale parameters in
thread-local variable defined as

extern _Thread_local const _RuneLocale *_ThreadRuneLocale;

The execution crashes on trying to access the named thread-local varible,
probably because something related to TLS is not set up properly in/for
LLVM.

I've confirmed this hypothesis by disabling isspace() calls in boolin()
which has also fixed the problem.

--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru ..: https://github.com/AMDmi3

* Dmitry Marakasov (amdmi3@amdmi3.ru) wrote:

Environment details:
- FreeBSD 12.1 amd64
- PostgreSQL 13.0 (built from FreeBSD ports)
- llvm-10.0.1 (build from FreeBSD ports)

My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
the system, and PostgreSQL uses llvm9:

ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

I'll check once I'm home whether I can reproduce in my environment.

I did some digging. First of all, I've discovered that the problem
goes away if llvm bitcode optimization is disabled (by commenting out
llvm_optimize_module call).

I've dumped the opcode and tried compiling it back to match disassembly
of the failing function in gdb disassembly. It didn't match perfectly,
but this place looked similar:

# %bb.84: # %op.32.inputcall
movq %rax, 5267(%r13)
movb %bl, 5275(%r13)
movb $0, 5263(%r13)
movzbl (%rax), %esi
movl __mb_sb_limit(%rip), %edi
movq _ThreadRuneLocale@GOTTPOFF(%rip), %rcx
movq %fs:0, %rdx
movq (%rdx,%rcx), %rcx
cmpl %esi, %edi
movq %rax, -96(%rbp) # 8-byte Spill
movl %edi, -72(%rbp) # 4-byte Spill
movq %rcx, -64(%rbp) # 8-byte Spill
jle .LBB1_85

Here's my hypothesis:

The problem happens when boolin() function is inlined by LLVM.
The named function calls isspace() internally, which on FreeBSD is
locale-specific and involves caching some locale parameters in
thread-local variable defined as

extern _Thread_local const _RuneLocale *_ThreadRuneLocale;

The execution crashes on trying to access the named thread-local varible,
probably because something related to TLS is not set up properly in/for
LLVM.

I've confirmed this hypothesis by disabling isspace() calls in boolin()
which has also fixed the problem.

Long story short, I was able to mitigate the crash with the following patch:

--- disable-inlining-tls-using-functions.patch begins here ---
commit f703544edc406293e39b7a59a245e798d18f458e
Author: Dmitry Marakasov <amdmi3@amdmi3.ru>
Date:   Thu Nov 5 02:56:00 2020 +0300

Do not inline functions accessing TLS in LLVM JIT

diff --git src/backend/jit/llvm/llvmjit_inline.cpp src/backend/jit/llvm/llvmjit_inline.cpp
index 2617a46..a063edb 100644
--- src/backend/jit/llvm/llvmjit_inline.cpp
+++ src/backend/jit/llvm/llvmjit_inline.cpp
@@ -608,6 +608,16 @@ function_inlinable(llvm::Function &F,
 		if (rv->materialize())
 			elog(FATAL, "failed to materialize metadata");

+		/*
+		 * Don't inline functions with thread-local variables until
+		 * related crashes are investigated (see BUG #16696)
+		 */
+		if (rv->isThreadLocal()) {
+			ilog(DEBUG1, "cannot inline %s due to thread-local variable %s",
+				F.getName().data(), rv->getName().data());
+			return false;
+		}
+
 		/*
 		 * Never want to inline externally visible vars, cheap enough to
 		 * reference.
--- disable-inlining-tls-using-functions.patch ends here ---

I have no knowledge of LLVM to investigate this further, but the guess
is that something TLS related is not initialized properly.

--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru ..: https://github.com/AMDmi3

Hi,

On 2020-11-05 00:20:15 +0300, Dmitry Marakasov wrote:

The problem happens when boolin() function is inlined by LLVM.
The named function calls isspace() internally, which on FreeBSD is
locale-specific and involves caching some locale parameters in
thread-local variable defined as

extern _Thread_local const _RuneLocale *_ThreadRuneLocale;

The execution crashes on trying to access the named thread-local varible,
probably because something related to TLS is not set up properly in/for
LLVM.

I've confirmed this hypothesis by disabling isspace() calls in boolin()
which has also fixed the problem.

Nice digging. I suspect your approach of disabling inlining for TLS
might be the most reasonable one for now.

Any chance you're interested in extending your patch with a testcase
that crashes reliably without it? Best on multiple platforms, including
linux? Easiest way would probably be to add a function referencing a
thread local variable in regress.c and set jit_above_cost = 0,
jit_inline_above_cost=0 for a query using that function.

Greetings,

Andres Freund

* Andres Freund (andres@anarazel.de) wrote:

The problem happens when boolin() function is inlined by LLVM.
The named function calls isspace() internally, which on FreeBSD is
locale-specific and involves caching some locale parameters in
thread-local variable defined as

extern _Thread_local const _RuneLocale *_ThreadRuneLocale;

The execution crashes on trying to access the named thread-local varible,
probably because something related to TLS is not set up properly in/for
LLVM.

I've confirmed this hypothesis by disabling isspace() calls in boolin()
which has also fixed the problem.

Nice digging. I suspect your approach of disabling inlining for TLS
might be the most reasonable one for now.

Any chance you're interested in extending your patch with a testcase
that crashes reliably without it? Best on multiple platforms, including
linux? Easiest way would probably be to add a function referencing a
thread local variable in regress.c and set jit_above_cost = 0,
jit_inline_above_cost=0 for a query using that function.

Can try. Here's what I could come with so far:

https://github.com/postgres/postgres/compare/master...AMDmi3:llvm-jit-regress.patch

Unfortunately it doesn't crash with the dedicated function, likely
because there's no llvm bitcode generation performed for regress.c.
It does however reliably crash on

SELECT (jsonb_array_elements('[true]'::jsonb)->>0)::boolean;
or
SELECT (case when random() >= 0.5 then 'true' else 'false' end)::boolean;

but it would likely not reproduce on Linux due to different isspace
implementation. Notably, it does not crash on simple 'true'::boolean,
I guess because it may be evaluated at compile time.

--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru ..: https://github.com/AMDmi3

Hi,

This seems to still not be fixed. Is it planned for 14.0 release?

Palle

--
Sent from: https://www.postgresql-archive.org/PostgreSQL-bugs-f2117394.html

On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:

This seems to still not be fixed. Is it planned for 14.0 release?

I don't know enough about LLVM and TLS to have an opinion on the
details, but I tested this and it works, and I think it should
probably be back-patched based on the discussion so far (and hopefully
in the future some way can be figured out to make it actually work?).

When I tested the original repro, I got SIGTRAP (not SIGSEGV as
reported) with LLVM 9 and 10, while 11, 12 and 13 failed gracefully
with:

2021-07-18 15:27:42.785 NZST [78884] FATAL: fatal llvm error:
Relocation type not implemented yet!

I was reminded of this thread because I happened to see a
relevant-looking commit whiz past in the FreeBSD commit log[1]https://cgit.freebsd.org/src/commit/?id=9c97062b620137a1f7cad4c6b3fb030a396b3266: libc
is changing to a different TLS model. I just tried that, and it makes
no difference.

In case it helps someone who wants to see this failure, I recently
XKCD1319'd myself into producing Vagrant files for PostgreSQL hacking
on various OSes[2]https://github.com/macdice/postgresql-dev-vagrant/ after a clash with illumos on the build farm (cd
freebsd12; vagrant up; vagrant ssh; you'll find a fresh build of the
PostgreSQL master branch built against LLVM 9 under ~/install).

[1]: https://cgit.freebsd.org/src/commit/?id=9c97062b620137a1f7cad4c6b3fb030a396b3266
[2]: https://github.com/macdice/postgresql-dev-vagrant/

Hi!

I’m on holiday and AFK another week or so. Please feel free to commit if it seems correct.

Palle

Hi,

On 2021-07-18 17:13:50 +1200, Thomas Munro wrote:

On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:

This seems to still not be fixed. Is it planned for 14.0 release?

I don't know enough about LLVM and TLS to have an opinion on the
details, but I tested this and it works, and I think it should
probably be back-patched based on the discussion so far (and hopefully
in the future some way can be figured out to make it actually work?).

When I tested the original repro, I got SIGTRAP (not SIGSEGV as
reported) with LLVM 9 and 10, while 11, 12 and 13 failed gracefully
with:

2021-07-18 15:27:42.785 NZST [78884] FATAL: fatal llvm error:
Relocation type not implemented yet!

I was reminded of this thread because I happened to see a
relevant-looking commit whiz past in the FreeBSD commit log[1]: libc
is changing to a different TLS model. I just tried that, and it makes
no difference.

FWIW, there's ongoing work in llvm to make the jit stuff handle TLS
correctly.

Greetings,

Andres Freund

On Thu, Jul 22, 2021 at 4:59 AM Andres Freund <andres@anarazel.de> wrote:

On 2021-07-18 17:13:50 +1200, Thomas Munro wrote:

On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:

This seems to still not be fixed. Is it planned for 14.0 release?

I don't know enough about LLVM and TLS to have an opinion on the
details, but I tested this and it works, and I think it should
probably be back-patched based on the discussion so far (and hopefully
in the future some way can be figured out to make it actually work?).

Done.

For the record, the original repro now causes PostgreSQL on FreeBSD
(with INLINE_DEBUG defined) to say:

2021-07-22 15:20:51.738 NZST [82982] DEBUG: checking inlinability of boolin
2021-07-22 15:20:51.738 NZST [82982] DEBUG: cannot inline boolin due
to thread-local variable _ThreadRuneLocale
2021-07-22 15:20:51.738 NZST [82982] DEBUG: had to skip inlining boolin

FWIW, there's ongoing work in llvm to make the jit stuff handle TLS
correctly.

Cool.

Dmitry: Thanks very much for the report, reproducer, analysis and patch!

* Thomas Munro (thomas.munro@gmail.com) wrote:

On Thu, Jul 22, 2021 at 4:59 AM Andres Freund <andres@anarazel.de> wrote:

On 2021-07-18 17:13:50 +1200, Thomas Munro wrote:

On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:

This seems to still not be fixed. Is it planned for 14.0 release?

I don't know enough about LLVM and TLS to have an opinion on the
details, but I tested this and it works, and I think it should
probably be back-patched based on the discussion so far (and hopefully
in the future some way can be figured out to make it actually work?).

Done.

For the record, the original repro now causes PostgreSQL on FreeBSD
(with INLINE_DEBUG defined) to say:

2021-07-22 15:20:51.738 NZST [82982] DEBUG: checking inlinability of boolin
2021-07-22 15:20:51.738 NZST [82982] DEBUG: cannot inline boolin due
to thread-local variable _ThreadRuneLocale
2021-07-22 15:20:51.738 NZST [82982] DEBUG: had to skip inlining boolin

FWIW, there's ongoing work in llvm to make the jit stuff handle TLS
correctly.

Cool.

Dmitry: Thanks very much for the report, reproducer, analysis and patch!

Superb, thanks a lot!

--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru ..: https://github.com/AMDmi3 https://amdmi3.ru