BUG #19078: Segfaults in tts_minimal_store_tuple() following pg_upgrade

On Thu, 2025-10-09 at 08:35 +0000, PG Bug reporting form wrote:

- Disabled enable_hashagg for some queries (just now)

Hi,

How sure are you that the crashes are happening without HashAgg? I made
some changes in that area in v18, and a lot of the evidence points in
that direction. Please let me know if you are able to confirm that the
simpler (non-hashagg) plan you showed is actually crashing.

Regards,
Jeff Davis

Hi. I was able to reproduce the crash with the simpler (non hash-agg) plan from the previous message.
Basically I launched it in multiple infinite loops that do BEGIN - UPDATE - ROLLBACK. Other clients could also modify the tables during this time.
We've seen this query crash on multiple physical hosts.

Original query:

update Tcv_scenes cs
set
state_id=2,
stitching_server_id=null,
stitching_server_pid=null
from
tcv_scene_datas cd -- Partition key: RANGE (cv_scene_id)
where
cd.cv_scene_id=cs.id and
(
(cs.state_id=7 and cs.date_cr<now()-interval '24 hours' and cs.date_state_mo>now()-interval '15 minutes' and cd.stitcher_result::text like '%download%') or
(cs.state_id=3 and cs.date_state_mo<now()-interval '5 minutes')
)
returning cs.id

GDB Stack Trace:

On Fri, 10 Oct 2025 at 14:34, Yuri Zamyatin <yuri@yrz.am> wrote:

Hi. I was able to reproduce the crash with the simpler (non hash-agg) plan from the previous message.
Basically I launched it in multiple infinite loops that do BEGIN - UPDATE - ROLLBACK. Other clients could also modify the tables during this time.
We've seen this query crash on multiple physical hosts.

#0 0x0000555fe8678300 in PartitionDirectoryLookup (pdir=0x0, rel=0x7f14d172b288) at ./build/../src/backend/partitioning/partdesc.c:462
pde = <optimized out>
relid = 21856
found = 27

Looks like that's crashing in a different place from the last
backtrace you showed.

Are you able to test this without any extensions loaded to see if you
still get a crash?

At a wild guess, perhaps an extension has gone rogue and spawned
another thread resulting in something like concurrent palloc requests
getting confused and causing something strange to happen when
accessing certain palloc'd chunks. Running without extensions may help
narrow things down.

postgresql_effective_cache_size = 560GB
postgresql_shared_buffers = 225GB

Which extension are these GUCs from?

David

Hi.

Hash aggregation.

enable_hashagg=off

Fixed all the crashes in a subset of queries that used hash aggregation (tts_minimal_store_tuple).
I'm sure about that since I did some a/b testing within a timeframe of a few days.

Configuration.

postgresql_effective_cache_size = 560GB
postgresql_shared_buffers = 225GB

Which extension are these GUCs from?

It was a typo in my message, sorry about that.
In postgresql.conf they're stated correctly as effective_cache_size and shared_buffers.

Partition lookup.

Are you able to test this without any extensions loaded to see if you still get a crash?

Yes, reproduced it on a machine with a clean Debian 13 & Postgresql 18 install
from the sources mentioned earlier. The only extension loaded is plpgsql. Changes to
postgresql.conf: max_connections=1000, work_mem=2000MB, shared_buffers=10GB, max_wal_size=10GB.

I pg_restore'd two tables (+partitions) from production into a clean cluster and created
indexes manually. The partitioned table is 2.2TB in size. I hope to narrow things down
and provide better reproduction steps tomorrow.

To cause the segfault, these queries were launched simultaneously.

-- in 2 parallel infinite loops
with ids as (select (118998526-random()*100000)::int id from generate_series(1,10000))
update tcv_scene_datas set id=id where cv_scene_id in(select id from ids);
with ids as (select (118998526-random()*100000)::int id from generate_series(1,10000))
update tcv_scenes set id=id where id in(select id from ids);

-- in 10 parallel infinite loops
set jit = off;
begin;
-- <EXPLAIN to save query plan>
update Tcv_scenes cs -- CRASHES
set
state_id=2,
stitching_server_id=null,
stitching_server_pid=null
from
tcv_scene_datas cd -- partitioned
where
cd.cv_scene_id=cs.id and
(
(state_id=7 and date_cr<now()-interval '24 hours' and date_state_mo>now()-interval '15 minutes' and cd.stitcher_result::text like '%download%') or
(state_id=3 and date_state_mo<now()-interval '5 minutes')
)
returning cs.id;
rollback;

Plan stayed the same. Stacktrace also looks the same (now without analyze part):

#0 0x000055987ffde300 in PartitionDirectoryLookup (pdir=0x0, rel=0x7f3069be8c98) at ./build/../src/backend/partitioning/partdesc.c:462
pde = <optimized out>
relid = 0
found = false
#1 0x000055987febeb51 in InitExecPartitionPruneContexts (prunestate=<optimized out>, parent_plan=0x55988a30b7d0, initially_valid_subplans=<optimized out>,
n_total_subplans=<optimized out>) at ./build/../src/backend/executor/execPartition.c:2413
partkey = 0x5598892dd5a0
partdesc = <optimized out>
pprune = <optimized out>
nparts = 239
k = <optimized out>
prunedata = 0x559889a8f7b8
j = <optimized out>
estate = <optimized out>
new_subplan_indexes = <optimized out>
new_other_subplans = <optimized out>
i = 0
newidx = <optimized out>
fix_subplan_map = <optimized out>
estate = <optimized out>
new_subplan_indexes = <optimized out>
new_other_subplans = <error reading variable new_other_subplans (Cannot access memory at address 0x0)>
i = <optimized out>
newidx = <optimized out>
fix_subplan_map = <optimized out>
prunedata = <error reading variable prunedata (Cannot access memory at address 0x0)>
j = <optimized out>
pprune = <optimized out>
nparts = <optimized out>
k = <optimized out>
partkey = <optimized out>
partdesc = <optimized out>
oldidx = <optimized out>
subidx = <optimized out>
subprune = <optimized out>
#2 ExecInitPartitionExecPruning (planstate=planstate@entry=0x55988a30b7d0, n_total_subplans=<optimized out>, part_prune_index=<optimized out>, relids=<optimized out>,
initially_valid_subplans=initially_valid_subplans@entry=0x7ffefb55a2e0) at ./build/../src/backend/executor/execPartition.c:1934
prunestate = <optimized out>
estate = <optimized out>
pruneinfo = <optimized out>
__func__ = "ExecInitPartitionExecPruning"
#3 0x000055987fed1030 in ExecInitAppend (node=node@entry=0x7f2dcff14420, estate=estate@entry=0x55988a309b08, eflags=eflags@entry=0)
at ./build/../src/backend/executor/nodeAppend.c:147
prunestate = <optimized out>
appendstate = 0x55988a30b7d0
appendplanstates = <optimized out>
appendops = <optimized out>
validsubplans = 0x55988a325470
asyncplans = <optimized out>
nplans = <optimized out>
nasyncplans = <optimized out>
firstvalid = <optimized out>
i = <optimized out>
j = <optimized out>
#4 0x000055987febfad5 in ExecInitNode (node=0x7f2dcff14420, estate=estate@entry=0x55988a309b08, eflags=0) at ./build/../src/backend/executor/execProcnode.c:182
result = <optimized out>
subps = <optimized out>
l = <optimized out>
__func__ = "ExecInitNode"
#5 0x000055987feea383 in ExecInitNestLoop (node=node@entry=0x7f2dd0006f28, estate=estate@entry=0x55988a309b08, eflags=<optimized out>, eflags@entry=0)
at ./build/../src/backend/executor/nodeNestloop.c:301
nlstate = 0x55988a30a7b0
__func__ = "ExecInitNestLoop"
#6 0x000055987febf8f1 in ExecInitNode (node=node@entry=0x7f2dd0006f28, estate=estate@entry=0x55988a309b08, eflags=eflags@entry=0)
at ./build/../src/backend/executor/execProcnode.c:298
result = <optimized out>
subps = <optimized out>
l = <optimized out>
__func__ = "ExecInitNode"
#7 0x000055987feba80f in EvalPlanQualStart (epqstate=0x559889383d08, planTree=0x7f2dd0006f28) at ./build/../src/backend/executor/execMain.c:3152
parentestate = <optimized out>
oldcontext = 0x559889a8ea20
rtsize = <optimized out>
rcestate = 0x55988a309b08
l = <optimized out>
parentestate = <optimized out>
rtsize = <optimized out>
rcestate = <optimized out>
oldcontext = <optimized out>
l = <optimized out>
i = <optimized out>
l__state = <optimized out>
subplan = <optimized out>
subplanstate = <optimized out>
l__state = <optimized out>
earm = <optimized out>
l__state = <optimized out>
rtindex = <optimized out>
#8 EvalPlanQualBegin (epqstate=epqstate@entry=0x559889383d08) at ./build/../src/backend/executor/execMain.c:2930
parentestate = <optimized out>
recheckestate = <optimized out>
#9 0x000055987feba9ab in EvalPlanQual (epqstate=0x559889383d08, relation=relation@entry=0x7f3069be4710, rti=1, inputslot=inputslot@entry=0x559889fc01f0)
at ./build/../src/backend/executor/execMain.c:2650
slot = <optimized out>
testslot = <optimized out>
#10 0x000055987fee601d in ExecUpdate (context=context@entry=0x7ffefb55a5b0, resultRelInfo=resultRelInfo@entry=0x559889383e28, tupleid=tupleid@entry=0x7ffefb55a58a,
oldtuple=oldtuple@entry=0x0, oldSlot=<optimized out>, oldSlot@entry=0x559889fb3178, slot=slot@entry=0x559889fb3580, canSetTag=true)
at ./build/../src/backend/executor/nodeModifyTable.c:2606
inputslot = 0x559889fc01f0
epqslot = <optimized out>
lockedtid = {ip_blkid = {bi_hi = 31, bi_lo = 16528}, ip_posid = 88}
estate = 0x559889a8eb18
resultRelationDesc = <optimized out>
updateCxt = {crossPartUpdate = false, updateIndexes = TU_None, lockmode = LockTupleNoKeyExclusive}
result = <optimized out>
__func__ = "ExecUpdate"
#11 0x000055987fee7fff in ExecModifyTable (pstate=0x559889383c20) at ./build/../src/backend/executor/nodeModifyTable.c:4510
node = 0x559889383c20
context = {mtstate = 0x559889383c20, epqstate = 0x559889383d08, estate = 0x559889a8eb18, planSlot = 0x559889fac818, tmfd = {ctid = {ip_blkid = {bi_hi = 31,
bi_lo = 16528}, ip_posid = 62}, xmax = 17203, cmax = 4294967295, traversed = true}, cpDeletedSlot = 0x0, cpUpdateReturningSlot = 0x7ffefb55a620}
estate = 0x559889a8eb18
operation = CMD_UPDATE
resultRelInfo = 0x559889383e28
subplanstate = <optimized out>
slot = 0x559889fb3580
oldSlot = 0x559889fb3178
tuple_ctid = {ip_blkid = {bi_hi = 31, bi_lo = 16528}, ip_posid = 62}
oldtupdata = {t_len = 240, t_self = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 50744}, t_tableOid = 402685741, t_data = 0x559889facf00}
oldtuple = 0x0
tupleid = <optimized out>
tuplock = false
__func__ = "ExecModifyTable"
#12 0x000055987feb791b in ExecProcNode (node=0x559889383c20) at ./build/../src/include/executor/executor.h:315
No locals.
#13 ExecutePlan (queryDesc=0x5598892beb88, operation=CMD_UPDATE, sendTuples=true, numberTuples=0, direction=<optimized out>, dest=0x5598892beb00)
at ./build/../src/backend/executor/execMain.c:1697
estate = 0x559889a8eb18
use_parallel_mode = <optimized out>
slot = <optimized out>
planstate = 0x559889383c20
current_tuple_count = 536
estate = <optimized out>
planstate = <optimized out>
use_parallel_mode = <optimized out>
slot = <optimized out>
current_tuple_count = <optimized out>
#14 standard_ExecutorRun (queryDesc=0x5598892beb88, direction=<optimized out>, count=0) at ./build/../src/backend/executor/execMain.c:366
estate = 0x559889a8eb18
operation = CMD_UPDATE
dest = 0x5598892beb00
sendTuples = <optimized out>
oldcontext = 0x5598892be8f0
#15 0x0000559880092774 in ProcessQuery (plan=0x7f2dd001cb00,
sourceText=0x559889170158 "update Tcv_scenes cs\nset\n\tstate_id=2,\n\tstitching_server_id=null,\n\tstitching_server_pid=null\nfrom\n\ttcv_scene_datas cd\nwhere\n\tcd.cv_scene_id=cs.id and\n\t(\n\t\t(state_id=7 and date_cr<now()-interval '24 hou"..., params=0x0, queryEnv=0x0, dest=0x5598892beb00, qc=0x7ffefb55a780)
at ./build/../src/backend/tcop/pquery.c:161
queryDesc = 0x5598892beb88
#16 0x0000559880093421 in PortalRunMulti (portal=portal@entry=0x559889267908, isTopLevel=isTopLevel@entry=true, setHoldSnapshot=setHoldSnapshot@entry=true,
dest=dest@entry=0x5598892beb00, altdest=0x559880538ec0 <donothingDR>, qc=qc@entry=0x7ffefb55a780) at ./build/../src/backend/tcop/pquery.c:1272
pstmt = 0x7f2dd001cb00
stmtlist_item__state = {l = 0x7f2dd00225c0, i = 0}
active_snapshot_set = true
stmtlist_item = 0x7f2dd00225d8
#17 0x000055988009358f in FillPortalStore (portal=portal@entry=0x559889267908, isTopLevel=isTopLevel@entry=true) at ./build/../src/backend/tcop/pquery.c:1021
treceiver = 0x5598892beb00
qc = {commandTag = CMDTAG_UNKNOWN, nprocessed = 0}
__func__ = "FillPortalStore"
#18 0x000055988009396f in PortalRun (portal=portal@entry=0x559889267908, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
dest=dest@entry=0x7f2dd000be20, altdest=altdest@entry=0x7f2dd000be20, qc=qc@entry=0x7ffefb55a990) at ./build/../src/backend/tcop/pquery.c:760
_save_exception_stack = 0x7ffefb55ac70
_save_context_stack = 0x0
_local_sigjmp_buf = {{__jmpbuf = {94113624389896, 9025151889858365403, 94113630543472, 140733115115920, 139834739965472, 94113630543512, 9025151889659135963,
3022814109072099291}, __mask_was_saved = 0, __saved_mask = {__val = {0, 140728898420737, 94114140538477, 94113624401432, 94113473390454, 140733115115744,
94113624389896, 94113473390454, 1, 139834740057536, 94113630543512, 140733115115808, 94113473059350, 140733115115808, 2, 140733115115808}}}}
_do_rethrow = <optimized out>
result = <optimized out>
nprocessed = <optimized out>
saveTopTransactionResourceOwner = 0x5598892157f8
saveTopTransactionContext = 0x55988927c5c0
saveActivePortal = 0x0
saveResourceOwner = 0x5598892157f8
savePortalContext = 0x0
saveMemoryContext = 0x55988927c5c0
__func__ = "PortalRun"
#19 0x000055988008f668 in exec_simple_query (
query_string=0x559889170158 "update Tcv_scenes cs\nset\n\tstate_id=2,\n\tstitching_server_id=null,\n\tstitching_server_pid=null\nfrom\n\ttcv_scene_datas cd\nwhere\n\tcd.cv_scene_id=cs.id and\n\t(\n\t\t(state_id=7 and date_cr<now()-interval '24 hou"...) at ./build/../src/backend/tcop/postgres.c:1273
cmdtaglen = 6
snapshot_set = <optimized out>
per_parsetree_context = 0x0
plantree_list = <optimized out>
parsetree = 0x559889845e70
commandTag = <optimized out>
qc = {commandTag = CMDTAG_UNKNOWN, nprocessed = 0}
querytree_list = <optimized out>
portal = 0x559889267908
receiver = 0x7f2dd000be20
format = 0
cmdtagname = <optimized out>
parsetree_item__state = {l = 0x559889845e98, i = 0}
dest = DestRemote
oldcontext = 0x55988927c5c0
parsetree_list = 0x559889845e98
parsetree_item = 0x559889845eb0
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str = "\340yW\200\230U\000\000Q\000\000\000\000\000\000\000ЫU\373\376\177\000\000\004\000\000\000\000\000\000"
__func__ = "exec_simple_query"
#20 0x000055988009156d in PostgresMain (dbname=<optimized out>, username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4766
query_string = 0x559889170158 "update Tcv_scenes cs\nset\n\tstate_id=2,\n\tstitching_server_id=null,\n\tstitching_server_pid=null\nfrom\n\ttcv_scene_datas cd\nwhere\n\tcd.cv_scene_id=cs.id and\n\t(\n\t\t(state_id=7 and date_cr<now()-interval '24 hou"...
firstchar = <optimized out>
input_message = {
data = 0x559889170158 "update Tcv_scenes cs\nset\n\tstate_id=2,\n\tstitching_server_id=null,\n\tstitching_server_pid=null\nfrom\n\ttcv_scene_datas cd\nwhere\n\tcd.cv_scene_id=cs.id and\n\t(\n\t\t(state_id=7 and date_cr<now()-interval '24 hou"..., len = 381, maxlen = 1024, cursor = 381}
local_sigjmp_buf = {{__jmpbuf = {140733115116464, 9025151890000971739, 293935616, 4, 0, 1, 9025151889751410651, 3022814102951954395}, __mask_was_saved = 1,
__saved_mask = {__val = {4194304, 143360, 12259252146692762112, 16, 132672, 18446744073709551312, 132656, 0, 8290, 139845915581216, 139845914247356,
94113473250832, 139845913448464, 2047, 18446744073709551312, 94113623012352}}}}
send_ready_for_query = false
idle_in_transaction_timeout_enabled = false
idle_session_timeout_enabled = false
__func__ = "PostgresMain"
#21 0x000055988008ba33 in BackendMain (startup_data=<optimized out>, startup_data_len=<optimized out>) at ./build/../src/backend/tcop/backend_startup.c:124
bsdata = <optimized out>
#22 0x000055987ffe9cfd in postmaster_child_launch (child_type=B_BACKEND, child_slot=6, startup_data=startup_data@entry=0x7ffefb55ae90,
startup_data_len=startup_data_len@entry=24, client_sock=client_sock@entry=0x7ffefb55aeb0) at ./build/../src/backend/postmaster/launch_backend.c:290
pid = <optimized out>
#23 0x000055987ffed802 in BackendStartup (client_sock=0x7ffefb55aeb0) at ./build/../src/backend/postmaster/postmaster.c:3587
bn = 0x5598891e6700
pid = <optimized out>
startup_data = {canAcceptConnections = CAC_OK, socket_created = 813767320273791, fork_started = 813767320273794}
cac = <optimized out>
bn = <optimized out>
pid = <optimized out>
startup_data = <optimized out>
cac = <optimized out>
__func__ = "BackendStartup"
__errno_location = <optimized out>
save_errno = <optimized out>
__errno_location = <optimized out>
__errno_location = <optimized out>
#24 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1702
s = {sock = 10, raddr = {addr = {ss_family = 1,
__ss_padding = "\2230\"\026\237\276\000\000\000\000\000\000\000\000\v\203\024\211\230U\000\000\000\000\000\000\000\000\000\000 \257U\373\376\177\000\000\360\256U\373\376\177\000\000\000\004\000\000\000\000\000\000\000\203\024\211\230U\000\000\213\331\036\200\230U", '\000' <repeats 18 times>, "@\257U\373\376\177\000\000\236(&\200\230U\000\000\000\000\000\000\000\000\000\000\255\226\tj0\177\000", __ss_align = 1}, salen = 2}}
i = 0
now = <optimized out>
last_lockfile_recheck_time = 1760452076
last_touch_time = 1760450375
events = {{pos = 3, events = 2, fd = 8, user_data = 0x0}, {pos = 0, events = 0, fd = 8, user_data = 0x0}, {pos = -1995336688, events = 21912, fd = 0,
user_data = 0x559880252a37}, {pos = 0, events = 0, fd = -1995336440, user_data = 0x40000000069}, {pos = 0, events = 21912, fd = -1995144437,
user_data = 0x0}, {pos = 126953984, events = 2854329568, fd = -1995328432, user_data = 0x55988058ef60 <errordata>}, {pos = -78270480, events = 32766,
fd = -2145478054, user_data = 0xf}, {pos = 0, events = 0, fd = -78270400, user_data = 0x0}, {pos = -78270400, events = 32766, fd = -1994499072,
user_data = 0x559880252a37}, {pos = -2141748608, events = 21912, fd = 0, user_data = 0x559880234239 <pg_freeaddrinfo_all+73>}, {pos = 8, events = 0,
fd = -78270160, user_data = 0x7ffefb55b970}, {pos = 2146491570, events = 21912, fd = 0, user_data = 0x153800000000}, {pos = -1994974088, events = 21912,
fd = -78270160, user_data = 0x7ffefb55b08c}, {pos = 1, events = 1, fd = -1994516715, user_data = 0x1891e171c}, {pos = -1994499072, events = 21912,
fd = -1994516682, user_data = 0x100000001}, {pos = 1, events = 0, fd = 0, user_data = 0x0}, {pos = 0, events = 0, fd = 0, user_data = 0x7f0032333435}, {
pos = -2145100672, events = 21912, fd = -1994516568, user_data = 0x5598891e17b2}, {pos = -1994516541, events = 21912, fd = -1994516531,
user_data = 0x5598891e17de}, {pos = -1994516506, events = 21912, fd = -1994516443, user_data = 0x5598891e182b}, {pos = -1994516425, events = 21760,
fd = 1782978392, user_data = 0x6e75722f7261762f}, {pos = 1936683055, events = 1701996404, fd = 795636083, user_data = 0x3334352e4c515347}, {
pos = -2145058766, events = 21912, fd = -2145015462, user_data = 0x7ffefb55b7a0}, {pos = -2145100583, events = 21912, fd = 0, user_data = 0x0}, {
pos = -78268464, events = 32766, fd = -2145100753, user_data = 0x7ffefb55b7e0}, {pos = 2382895, events = 0, fd = 0, user_data = 0x0}, {pos = 0, events = 0,
fd = 1782978392, user_data = 0x0}, {pos = 1779089596, events = 32560, fd = 0, user_data = 0x5598891e38b0}, {pos = 0, events = 0, fd = 1782978392,
user_data = 0x8}, {pos = 1780423360, events = 32560, fd = 255, user_data = 0xfffffffffffffed0}, {pos = 0, events = 0, fd = 399,
user_data = 0x5598891e5420}, {pos = 1779088954, events = 32560, fd = -1994499536, user_data = 0x570}, {pos = 0, events = 0, fd = 10, user_data = 0x0}, {
pos = 1780423360, events = 32560, fd = 255, user_data = 0xfffffffffffffed0}, {pos = 1780416464, events = 32560, fd = 8,
user_data = 0x7f306a1effd0 <_IO_file_jumps>}, {pos = 1779092818, events = 32560, fd = 2996, user_data = 0x5598891e5420}, {pos = 4096, events = 0,
fd = -78269312, user_data = 0x7f306a1effd0 <_IO_file_jumps>}, {pos = 1778941388, events = 32560, fd = 25, user_data = 0x21fa}, {pos = 1, events = 0,
fd = 33152, user_data = 0x68}, {pos = 0, events = 0, fd = 1, user_data = 0x100000000}, {pos = 2, events = 17, fd = 0, user_data = 0x3}, {pos = 0,
events = 1, fd = 0, user_data = 0x0}, {pos = 0, events = 0, fd = 0, user_data = 0x0}, {pos = 0, events = 0, fd = 0, user_data = 0x0}, {pos = 0, events = 0,
fd = 0, user_data = 0x0}, {pos = 0, events = 0, fd = -1994501088, user_data = 0x5598891e5420}, {pos = 8, events = 0, fd = 1780423360, user_data = 0x802}, {
pos = -304, events = 4294967295, fd = 5, user_data = 0x55988024bdd6}, {pos = -2145008860, events = 21912, fd = 1779092818, user_data = 0x7ffefb55b420}, {
pos = -2145095639, events = 21912, fd = 32768, user_data = 0x9}, {pos = -78269152, events = 32766, fd = 1779329413, user_data = 0x7f0000000000}, {pos = 9,
events = 0, fd = -78269184, user_data = 0x7f306a0e69ff}, {pos = 2346, events = 0, fd = 16318532, user_data = 0x5598891fed30}, {pos = 126953984,
events = 2854329568, fd = -1994363536, user_data = 0x9}, {pos = -1994396368, events = 21912, fd = -360, user_data = 0x9}, {pos = -1994396368,
events = 21912, fd = -360, user_data = 0x7f306a0ad3c0 <free+384>}, {pos = 926193527, events = 0, fd = 1760102962, user_data = 0x37349777}, {pos = 0,
events = 0, fd = 0, user_data = 0x9}, {pos = -78269184, events = 32766, fd = -78269152, user_data = 0x5598891fed40}, {pos = -2145075754, events = 21912,
fd = -2145008860, user_data = 0x7f306a0e678d <closedir+13>}, {pos = -1995282496, events = 21912, fd = -2147108231, user_data = 0x5598891fed40}, {
pos = -2145075649, events = 21912, fd = -78268032, user_data = 0x55988005f0c8 <RemovePgTempFiles+312>}, {pos = -2141655518, events = 21912, fd = -78268820,
user_data = 0x7367702f65736162}, {pos = 1952410737, events = 28781, fd = 771766842, user_data = 0x7f306a0abe3a}}
nevents = <optimized out>
__func__ = "ServerLoop"
#25 0x000055987ffef110 in PostmasterMain (argc=argc@entry=5, argv=argv@entry=0x5598891172e0) at ./build/../src/backend/postmaster/postmaster.c:1400
opt = <optimized out>
status = <optimized out>
userDoption = <optimized out>
listen_addr_saved = true
output_config_variable = <optimized out>
__func__ = "PostmasterMain"
#26 0x000055987fce5880 in main (argc=5, argv=0x5598891172e0) at ./build/../src/backend/main/main.c:227
do_check_root = <optimized out>
dispatch_option = <optimized out>

__
Yuri Zamyatin

On Wed, 15 Oct 2025 at 04:51, Yuri Zamyatin <yuri@yrz.am> wrote:

To cause the segfault, these queries were launched simultaneously.

-- in 2 parallel infinite loops
with ids as (select (118998526-random()*100000)::int id from generate_series(1,10000))
update tcv_scene_datas set id=id where cv_scene_id in(select id from ids);
with ids as (select (118998526-random()*100000)::int id from generate_series(1,10000))
update tcv_scenes set id=id where id in(select id from ids);

Are you able to mock this up using the schema and some test data then
share the script to populate the database?

David

I found a much easier way.

create.sql:

drop database if exists segtest;
create database segtest;
\c segtest;
create table tcv_scene_datas(cv_scene_id bigint primary key) partition by range (cv_scene_id);
do $$
declare
i int;
range_start bigint;
range_end bigint;
partition_name text;
begin
for i in 0..100 loop
range_start := 1 + (i * 10000);
range_end := range_start + 10000;
partition_name := 'tcv_scene_datas_' || LPAD(i::TEXT, 3, '0');
execute format(
'create table %I partition of tcv_scene_datas for values from (%s) to (%s)',
partition_name,
range_start,
range_end
);
end loop;
end $$;
insert into tcv_scene_datas(cv_scene_id) select id from generate_series(1,1_000_000) id;

crash.sql:

\c segtest
with ids as (select (random()*1_000_000)::int id from generate_series(1,1000))
update tcv_scene_datas set cv_scene_id=cv_scene_id where cv_scene_id in(select id from ids);

Launch crash.sql in 16 threads of infinite loops:

seq 16 | xargs -P 16 -I {} sh -c 'while true; do psql -f crash.sql; done'

In 1-2 minutes, 5 processes died with segfault.
Also I expected deadlocks with such query, strangely database did not report them.

Let me know if you need more data.

__
Best wishes, Yuri

On Thu, 16 Oct 2025 at 03:21, Yuri Zamyatin <yuri@yrz.am> wrote:

create.sql:

drop database if exists segtest;
create database segtest;
\c segtest;
create table tcv_scene_datas(cv_scene_id bigint primary key) partition by range (cv_scene_id);
do $$
declare
i int;
range_start bigint;
range_end bigint;
partition_name text;
begin
for i in 0..100 loop
range_start := 1 + (i * 10000);
range_end := range_start + 10000;
partition_name := 'tcv_scene_datas_' || LPAD(i::TEXT, 3, '0');
execute format(
'create table %I partition of tcv_scene_datas for values from (%s) to (%s)',
partition_name,
range_start,
range_end
);
end loop;
end $$;
insert into tcv_scene_datas(cv_scene_id) select id from generate_series(1,1_000_000) id;

crash.sql:

\c segtest
with ids as (select (random()*1_000_000)::int id from generate_series(1,1000))
update tcv_scene_datas set cv_scene_id=cv_scene_id where cv_scene_id in(select id from ids);

Launch crash.sql in 16 threads of infinite loops:

seq 16 | xargs -P 16 -I {} sh -c 'while true; do psql -f crash.sql; done'

In 1-2 minutes, 5 processes died with segfault.

Perfect. Thank you.

It seems to be some more forgotten EPQ stuff from d47cbf474. Amit got
some of these in 8741e48e5, but evidently the test case didn't do
pruning during execution, (only init plan pruning) so the partition
directory wasn't needed.

The attached seems to fix it for me.

David

On Thu, 16 Oct 2025 at 11:45, David Rowley <dgrowleyml@gmail.com> wrote:

On Thu, 16 Oct 2025 at 03:21, Yuri Zamyatin <yuri@yrz.am> wrote:

In 1-2 minutes, 5 processes died with segfault.

Perfect. Thank you.

It seems to be some more forgotten EPQ stuff from d47cbf474. Amit got
some of these in 8741e48e5, but evidently the test case didn't do
pruning during execution, (only init plan pruning) so the partition
directory wasn't needed.

I forgot to mention, this isn't the same thing as the
tts_minimal_store_tuple() issue you first reported, so if there is a
problem there, this one has nothing to do with it.

Any chance of a self-contained test case for the enable_hashagg=on crash?

David

Thank you very much.

I just tested pruning on the original case with the patch
you sent and confirm segfaults went away.

Regarding hash aggregation, I'll try to find a test case and
follow up within a day or so (cloning a huge db right now).

On Thu, Oct 16, 2025 at 7:51 AM David Rowley <dgrowleyml@gmail.com> wrote:

On Thu, 16 Oct 2025 at 11:45, David Rowley <dgrowleyml@gmail.com> wrote:

On Thu, 16 Oct 2025 at 03:21, Yuri Zamyatin <yuri@yrz.am> wrote:

In 1-2 minutes, 5 processes died with segfault.

Thanks Yuri for the report and the test case.

Perfect. Thank you.

It seems to be some more forgotten EPQ stuff from d47cbf474. Amit got
some of these in 8741e48e5, but evidently the test case didn't do
pruning during execution, (only init plan pruning) so the partition
directory wasn't needed.

I forgot to mention, this isn't the same thing as the
tts_minimal_store_tuple() issue you first reported, so if there is a
problem there, this one has nothing to do with it.

Thanks again, David.

I've attached an updated patch with a test case.

--
Thanks, Amit Langote

On Thu, 16 Oct 2025 at 16:29, Amit Langote <amitlangote09@gmail.com> wrote:

I've attached an updated patch with a test case.

Looks good to me. Nice simple test.

David

On Thu, Oct 16, 2025 at 1:04 PM David Rowley <dgrowleyml@gmail.com> wrote:

On Thu, 16 Oct 2025 at 16:29, Amit Langote <amitlangote09@gmail.com> wrote:

I've attached an updated patch with a test case.

Looks good to me. Nice simple test.

Thanks for checking. Pushed.

--
Thanks, Amit Langote

On Thu, 2025-10-16 at 11:51 +1300, David Rowley wrote:

I forgot to mention, this isn't the same thing as the
tts_minimal_store_tuple() issue you first reported, so if there is a
problem there, this one has nothing to do with it.

I investigated, but came up empty so far. Any additional info on the
hashagg crash would be appreciated.

I appended my raw notes below in case someone notices a mistake.

Regards,
Jeff Davis

Raw notes:

* Somehow entry->firstTuple==0x1b, which is obviously wrong.

* The entry structure lives in the bucket array, allocated in the
metacxt using MCXT_ALLOC_ZERO, so there's no uninitialized memory
floating around in the bucket array.

* The metacxt (aggstate->hash_metacxt) is an AllocSet, and it's never
reset. It contains the bucket array as well as some ExprStates and an
ExprContext for evaluating hash functions.

* Hash entries are never deleted, but between batches the entire hash
table is reset (which memsets the entire bucket array to zero).

* The entry->firstTuple is assigned only in one place, from
ExecCopySlotMinimalTupleExtra(). The 'extra' argument is a multiple of
16.

* ExecCopySlotMinimalTupleExtra() does some interesting pointer math,
but I didn't find any path that could plausibly return something like
0x1b. The memory is allocated with palloc/palloc0, which cannot return
zero, and 0x1b is not a multiple of 16 so seems unrelated to the extra
argument.

* JIT does not seem to be involved, because it's going through
ExecInterpExpr().

* When the hash table grows, it invalidates previously-returned entry
pointers. But, given the site of the crash, I don't see that as a
problem in this case.

Haven't managed to reproduce it consistently so far.
Perhaps we could use the existing dump to investigate.
I'm unfamiliar with Postgres internals. If it would be useful,
could you walk me through the structures I need to capture?

(gdb) frame 2
#2 0x0000555fe8566ec2 in agg_retrieve_hash_table_in_memory (aggstate=aggstate@entry=0x55601c7567d0) at ./build/../src/include/executor/executor.h:176
176 in ./build/../src/include/executor/executor.h
(gdb) print *perhash
$83 = {hashtable = 0x55601c182ac8, hashiter = {cur = 10, end = 43, done = false}, hashslot = 0x55601c765bb0, hashfunctions = 0x55601c765b28,
eqfuncoids = 0x55601c765b18, numCols = 2, numhashGrpCols = 2, largestGrpColIdx = 4, hashGrpColIdxInput = 0x55601c7659f0, hashGrpColIdxHash = 0x55601c765a00,
aggnode = 0x55601eba19f8}
(gdb) print *hashtable->hashtab
$84 = {size = 16, members = 4, sizemask = 15, grow_threshold = 14, data = 0x55601c182b98, ctx = 0x55601c1819b0, private_data = 0x55601c182ac8}
(gdb) print *entry
$86 = {firstTuple = 0x1b, status = 1, hash = 21856}

Does it look suspicious?
perhash->hashiter->end=43, hashtable->hashtab->size=16, 43-16=0x1b

Some more details from that step:

Thanks for doing all the extra debugging.

On Sat, 18 Oct 2025 at 09:09, Yuri Zamyatin <yuri@yrz.am> wrote:

$84 = {size = 16, members = 4, sizemask = 15, grow_threshold = 14, data = 0x55601c182b98, ctx = 0x55601c1819b0, private_data = 0x55601c182ac8}
(gdb) print *entry
$86 = {firstTuple = 0x1b, status = 1, hash = 21856}

Does it look suspicious?
perhash->hashiter->end=43, hashtable->hashtab->size=16, 43-16=0x1b

If that's the iterator for that hash table, then that's a big problem.
hashiter->end should never be >= hashtab->size. If that happens we'll
index over the end of the bucket array, and that might explain why the
firstTuple field is set to an invalid pointer.

Are you able to build with Asserts and try and get an Assert failure
with the attached patch?

If this fails then maybe we're using the wrong iterator somewhere in
nodeAgg.c. I can't see any other way for the iterator's 'end' field to
be bigger than the table's size.

David

On Sat, 18 Oct 2025 at 10:25, David Rowley <dgrowleyml@gmail.com> wrote:

If this fails then maybe we're using the wrong iterator somewhere in
nodeAgg.c. I can't see any other way for the iterator's 'end' field to
be bigger than the table's size.

I started looking for places that this could happen and quickly found
the following code:

/*
* Switch to next grouping set, reinitialize, and restart the
* loop.
*/
select_current_set(aggstate, nextset, true);

perhash = &aggstate->perhash[aggstate->current_set];

ResetTupleHashIterator(hashtable, &perhash->hashiter);

The hash table and the iterator for each set are meant to be in the
same AggStatePerHash, but the above code moves to the next set,
changes the "perhash" then resets the next iterator using the previous
hash table.

I think that line needs to be:

ResetTupleHashIterator(perhash->hashtable, &perhash->hashiter);

David

Thank you for the patch. Will do.
It will probably take me 1-2 days to test
due to the pattern of crashes.
__
Yuri

Nice, should I still try to reproduce the bug
with assertions and the patch you provided?

On Sat, 18 Oct 2025 at 11:18, Yuri Zamyatin <yuri@yrz.am> wrote:

Nice, should I still try to reproduce the bug
with assertions and the patch you provided?

I think I've got it. I can get the Asserts in the patch to fail with:

drop table if exists ab;
create table ab (a int, b int);
insert into ab select x%2,x%99 from generate_series(1,1001)x;

select a,b,count(*) from ab group by grouping sets(b,a,(a,b),a,b);

David