BUG #19796: io_uring crashes with specific kernel/liburing versions + io_max_concurrency due to rings overlap
The following bug has been logged on the website:
Bug reference: 19796
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 19rc1
Operating system: Ubuntu 24.04
Description:
The following script:
psql -c "CREATE TABLE t AS SELECT repeat('x', 100) FROM generate_series(1,
500000)"
echo "SELECT count(*) FROM t" > /tmp/q.sql
pgbench -n -f /tmp/q.sql -c 4 -T 60
with these parameters set:
io_method = io_uring
io_max_concurrency = 1024
shared_buffers = 16MB
max_connections = 30
crashes on Ubuntu 24.04 as below:
2026-10-25 06:03:22.401 UTC [39572] PANIC: io_uring submit submitted only 0
of 1
2026-10-25 06:03:22.401 UTC [39572] STATEMENT: SELECT count(*) FROM t
2026-10-25 06:03:22.401 UTC [39523] ERROR: io_uring submit submitted only 0
of 1
2026-10-25 06:03:22.401 UTC [39523] CONTEXT: parallel worker
or
TRAP: failed Assert("ioh->state == PGAIO_HS_SUBMITTED"), File: "aio.c",
Line: 530, PID: 39816
ExceptionalCondition at assert.c:51:13
pgaio_io_process_completion at aio.c:532:2
pgaio_uring_drain_locked at method_io_uring.c:565:20
pgaio_uring_wait_one at method_io_uring.c:643:4
pgaio_io_wait at aio.c:623:6
pgaio_wref_wait at aio.c:999:1
WaitReadBuffers at bufmgr.c:1831:17
read_stream_next_buffer at read_stream.c:1191:17
heap_fetch_next_buffer at heapam.c:743:16
heapgettup_pagemode at heapam.c:1110:8
heap_getnextslot at heapam.c:1484:3
table_scan_getnextslot at tableam.h:1105:1
SeqNext at nodeSeqscan.c:90:5
...
With this logging added:
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -357,6 +357,12 @@ pgaio_uring_shmem_init(void *arg)
struct io_uring_params p = {0};
ret = io_uring_queue_init_mem(io_max_concurrency,
&context->io_uring_ring, &p, ring_mem_next, ring_mem_remain);
+if (ret > 0)
+{
+ int overlap = (char *) (context->io_uring_ring.sq.array +
p.sq_entries) - (ring_mem_next + ret);
+ if (overlap > 0)
+ elog(LOG, "!!!pgaio_uring_shmem_init| io_uring ring %d
overlaps the next ring by %d bytes", contextno, overlap);
+}
ring_mem_remain -= ret;
ring_mem_next += ret;
I can see:
2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 0 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 1 overlaps the next ring by 64 bytes
...
2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 70 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 71 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.354 UTC [39804] LOG: starting PostgreSQL 20devel on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1)
13.3.0, 64-bit
I observed this on Ubuntu machines with kernels:
Linux a 7.0.0-1014-aws #14~24.04.1-Ubuntu SMP PREEMPT Fri Sep 18 21:50:22
UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
Linux b 6.14.1-061401-generic #202504071048 SMP PREEMPT_DYNAMIC Mon Apr 7
11:20:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
and liburing 2.5. The issue is not reproduced after
https://github.com/axboe/liburing/commit/9f4464d.
вс, 25 окт. 2026 г. в 18:34, PG Bug reporting form <noreply@postgresql.org>:
The following bug has been logged on the website:
Bug reference: 19796
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 19rc1
Operating system: Ubuntu 24.04
Description:The following script:
psql -c "CREATE TABLE t AS SELECT repeat('x', 100) FROM generate_series(1,
500000)"echo "SELECT count(*) FROM t" > /tmp/q.sql
pgbench -n -f /tmp/q.sql -c 4 -T 60with these parameters set:
io_method = io_uring
io_max_concurrency = 1024
shared_buffers = 16MB
max_connections = 30crashes on Ubuntu 24.04 as below:
2026-10-25 06:03:22.401 UTC [39572] PANIC: io_uring submit submitted only
0
of 1
2026-10-25 06:03:22.401 UTC [39572] STATEMENT: SELECT count(*) FROM t
2026-10-25 06:03:22.401 UTC [39523] ERROR: io_uring submit submitted only
0
of 1
2026-10-25 06:03:22.401 UTC [39523] CONTEXT: parallel workeror
TRAP: failed Assert("ioh->state == PGAIO_HS_SUBMITTED"), File: "aio.c",
Line: 530, PID: 39816
ExceptionalCondition at assert.c:51:13
pgaio_io_process_completion at aio.c:532:2
pgaio_uring_drain_locked at method_io_uring.c:565:20
pgaio_uring_wait_one at method_io_uring.c:643:4
pgaio_io_wait at aio.c:623:6
pgaio_wref_wait at aio.c:999:1
WaitReadBuffers at bufmgr.c:1831:17
read_stream_next_buffer at read_stream.c:1191:17
heap_fetch_next_buffer at heapam.c:743:16
heapgettup_pagemode at heapam.c:1110:8
heap_getnextslot at heapam.c:1484:3
table_scan_getnextslot at tableam.h:1105:1
SeqNext at nodeSeqscan.c:90:5
...With this logging added: --- a/src/backend/storage/aio/method_io_uring.c +++ b/src/backend/storage/aio/method_io_uring.c @@ -357,6 +357,12 @@ pgaio_uring_shmem_init(void *arg) struct io_uring_params p = {0};ret = io_uring_queue_init_mem(io_max_concurrency, &context->io_uring_ring, &p, ring_mem_next, ring_mem_remain); +if (ret > 0) +{ + int overlap = (char *) (context->io_uring_ring.sq.array + p.sq_entries) - (ring_mem_next + ret); + if (overlap > 0) + elog(LOG, "!!!pgaio_uring_shmem_init| io_uring ring %d overlaps the next ring by %d bytes", contextno, overlap); +}ring_mem_remain -= ret;
ring_mem_next += ret;I can see:
2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 0 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 1 overlaps the next ring by 64 bytes
...
2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 70 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
io_uring ring 71 overlaps the next ring by 64 bytes
2026-10-25 06:04:51.354 UTC [39804] LOG: starting PostgreSQL 20devel on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1)
13.3.0, 64-bitI observed this on Ubuntu machines with kernels:
Linux a 7.0.0-1014-aws #14~24.04.1-Ubuntu SMP PREEMPT Fri Sep 18 21:50:22
UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
Linux b 6.14.1-061401-generic #202504071048 SMP PREEMPT_DYNAMIC Mon Apr 7
11:20:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
and liburing 2.5. The issue is not reproduced after
https://github.com/axboe/liburing/commit/9f4464d.
Hi, Alexander!
Thanks for the report.
PostgreSQL stores one io_uring ring per backend in one shared-memory
buffer. The next ring starts at the byte count io_uring_queue_init_mem()
returns. Before liburing 2.7 that count leaves out the io_rings header. The
kernel's ring is longer, and the next ring overlaps its tail.
The real length is in the parameters the kernel fills in. cq_off.cqes is
where the CQE array starts, and that offset already includes the header.
Add the CQE array, align to the cache line, then add the SQ index array if
the kernel created one. On a 1024-entry ring the missing part is 64 bytes.
With liburing 2.5 it is the end of the SQ index array.
Liburing 2.6 has no index array, and those 64 bytes are the end of the CQE
array. Liburing 2.7 includes the header, so the returned count is already
enough.
A fixed extra 64 bytes would also move a correct liburing, and the next
ring would no longer start on a page. The kernel rejects such an address.
The patch keeps whichever is further, the computed end or liburing's count,
and rounds that up to a page. If liburing's count is already past that end,
the reservation stays as it is. Without io_uring_queue_init_mem() and
IORING_SETUP_NO_MMAP this code is not compiled. Each ring still gets its
own mapping.
Checked with io_method=io_uring and io_max_concurrency=1024. liburing 2.5
reported 102400 bytes per ring.
Unpatched, pgbench -c 4 hit both "io_uring submit submitted only 0 of 1"
and Assert(ioh->state == PGAIO_HS_SUBMITTED).
Patched, the server reserved 106496 bytes and the same run finished.
On liburing 2.7 the reservation stayed 102400 bytes and the run finished.
Alexander, please try the builds you still have. liburing 2.5, 2.6 and 2.7,
on both machines from the report.
I only ran 2.5 and 2.7 on x86_64.
Andres in CC.
--
Regards,
Rachitskiy Andrey
Hi,
On 2026-10-26 13:27:28 +0500, Andrey Rachitskiy wrote:
Thanks for the report.
PostgreSQL stores one io_uring ring per backend in one shared-memory
buffer. The next ring starts at the byte count io_uring_queue_init_mem()
returns. Before liburing 2.7 that count leaves out the io_rings header. The
kernel's ring is longer, and the next ring overlaps its tail.The real length is in the parameters the kernel fills in. cq_off.cqes is
where the CQE array starts, and that offset already includes the header.
Add the CQE array, align to the cache line, then add the SQ index array if
the kernel created one. On a 1024-entry ring the missing part is 64 bytes.
With liburing 2.5 it is the end of the SQ index array.
Liburing 2.6 has no index array, and those 64 bytes are the end of the CQE
array. Liburing 2.7 includes the header, so the returned count is already
enough.
A fixed extra 64 bytes would also move a correct liburing, and the next
ring would no longer start on a page. The kernel rejects such an address.
The patch keeps whichever is further, the computed end or liburing's count,
and rounds that up to a page. If liburing's count is already past that end,
the reservation stays as it is. Without io_uring_queue_init_mem() and
IORING_SETUP_NO_MMAP this code is not compiled. Each ring still gets its
own mapping.Checked with io_method=io_uring and io_max_concurrency=1024. liburing 2.5
reported 102400 bytes per ring.
Unpatched, pgbench -c 4 hit both "io_uring submit submitted only 0 of 1"
and Assert(ioh->state == PGAIO_HS_SUBMITTED).
Patched, the server reserved 106496 bytes and the same run finished.
On liburing 2.7 the reservation stayed 102400 bytes and the run finished.Alexander, please try the builds you still have. liburing 2.5, 2.6 and 2.7,
on both machines from the report.
I only ran 2.5 and 2.7 on x86_64.
I'd really appreciate if you stopped copy-pasting bug reports to an LLM an
then copying the answer back. We all have access to this wondrous copy-paste
magic and most of us have access to LLMs. All this does is to further increase
the noise level of the lists which is already very high.
It's useful to analyze bugs, but it does require actual analysis and the
messages need to be actually readable stuff, rather than LLM goop.
Greetings,
Andres Freund
Hi,
On 2026-10-25 07:00:02 +0000, PG Bug reporting form wrote:
I observed this on Ubuntu machines with kernels:
Linux a 7.0.0-1014-aws #14~24.04.1-Ubuntu SMP PREEMPT Fri Sep 18 21:50:22
UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
Linux b 6.14.1-061401-generic #202504071048 SMP PREEMPT_DYNAMIC Mon Apr 7
11:20:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
and liburing 2.5. The issue is not reproduced after
https://github.com/axboe/liburing/commit/9f4464d.
Seems we either could error out on buggy liburings or we could just add
64bytes to the ring sizes for those versions. I assume the latter fixes the
problem for you?
Greetings,
Andres Freund
пн, 26 окт. 2026 г. в 18:31, Andres Freund <andres@anarazel.de>:
we could just add 64bytes to the ring sizes for those versions.
v1 built the end of the ring itself from cq_off and the SQ index array, and
increased the reservation only when that end was longer than liburing's
answer.
In v2 on liburing older than 2.7, adds 64 bytes to the answer and rounds up
to a page - the code came out more compact.
Tested 2.5/2.6/2.7 - all good.
I've taken your remarks about the LLM for the future.