BUG #19725: PostgreSQL 18.6: pg_restore read failure with io_uring, not observed with worker
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:
Hello,
I can reproduce a read failure during pg_restore with io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.
Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64
Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2
Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &
One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);
Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000
Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.
I have not measured the failure rate or prepared a self-contained
minimal reproducer.
One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.
Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?
Thank you.
After setting backtrace_functions=md_readv_report, I received more log information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000: could not read blocks 143..158 in file "base/6406098/6407866": Operation canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION: md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
postgres: postgres test001 [local] CREATE INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
postgres: postgres test001 [local] CREATE INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
postgres: postgres test001 [local] CREATE INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88) [0x5878e8]
postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05) [0x5b7325]
postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c) [0x61fa2c]
postgres: postgres test001 [local] CREATE INDEX(index_create+0x1328) [0x621608]
postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb) [0x6d11cb]
postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
postgres: postgres test001 [local] CREATE INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
/usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291) [0x93d371]
postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
postgres: postgres test001 [local] CREATE INDEX(PostgresMain+0x1f1b) [0x937f6b]
postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
postgres: postgres test001 [local] CREATE INDEX(postmaster_child_launch+0x12d) [0x888ced]
postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
postgres: postgres test001 [local] CREATE INDEX(PostmasterMain+0xce9) [0x894309]
postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) [0x531bc0]
/lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
postgres: postgres test001 [local] CREATE INDEX(_start+0x25) [0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT: CREATE INDEX idx_sys_notification_channel_notification_id ON dhr.sys_notification_channel USING btree (c_notification_id);
Show quoted text
2026年9月28日 11:34，PG Bug reporting form <noreply@postgresql.org> 写道：
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:Hello,
I can reproduce a read failure during pg_restore with io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.I have not measured the failure rate or prepared a self-contained
minimal reproducer.One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?Thank you.
Hi!
I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think
that I have the explanation for how it happens + proposal for the fix as
well.
*What I think is happening*: postgres io_uring requests hit the kernel
codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this
limit is hit, no new workers can't be spawned, but also postgresmaster
can't fork() or system() anymore, leading to "could not read blocks X..Y in
file ...: Operation canceled"
*How it could be fixed*: postgres should either call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
*Detailed description of the chain of events follows:*
*1. Postgres asks io_uring to send buffered reads to worker threads*
In method_io_uring.c (line 441), once a backend already has more than 4
I/Os in flight, every further buffered read gets IOSQE_ASYNC:
if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
If `maintenance_io_concurrency` is higher that 4, then all of the
maintenance reads starting from 5th will take this path.
*2. The kernel places these reads into the pool whose only cap is
RLIMIT_NPROC*
Each task's io_uring worker pool has two halves:
- bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).
Requests are classified in io_prep_async_work (
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492):
if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
else if (!req->file || !S_ISBLK(...)) {
if (def->unbound_nonreg_file) // set for READ/READV
atomic_or(IO_WQ_WORK_UNBOUND, ...);
}
However, io_init_req sets req->file = NULL (at
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046),
and the file is only attached later, at issue time. An IOSQE_ASYNC request
goes straight though io_queue_iowq() to io_prep_async_work() while
`req->file` is still NULL.
So a read of a regular file ends up routed to the unbound pool, whose cap
is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work
as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead
of waiting for a running one.
*3. Workers get created far faster than the actual I/O needs*
In io_wq_dec_running, when a worker blocks and it was the last running
worker while work is still queued, the kernel queues creation of a
brand-new worker immediately.
It doesn't wake one of the idle workers. On a cold buffer cache almost
every read blocks on disk, so this code path fires constantly. As a result,
the new worker handles one item and then goes idle. Only the head of the
free list gets reused, so the rest of the workers sit idle until
WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
As a result, the steady-state thread count is therefore roughly "creation
rate × 5 s", not "I/Os in flight". The cap for the workers, as was
mentioned earlier, is RLIMIT_NPROC.
*Similar threads or bugs from the past*
-
Justin Pryzby on pgsql-hackers
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html>
(2025-10-17)
reported could not read blocks 122..126 ...: Operation canceled from
pg_dump under io_uring. He guessed it might be related to the io_uring
file descriptors leaking after crash-restarts, but I cannot find a
resolution.
-
Alexander Lakhin's AIO stress test
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html>
(pgsql-hackers,
April 2025, before 18.0) hit could not read blocks 0..0 in file
"global/1213": Operation canceled with parallel createdb/dropdb jobs, on
kernels 6.8 and 6.11. The bugs fixed later in that thread (1
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html>
, 2
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html>)
were different ones, a pg_aios crash and a PANIC from interrupt
handling. As far as I can see, the ECANCELED was never explained or fixed.
io_uring bugs that look relevant to me
-
liburing #420 <https://github.com/axboe/liburing/issues/420> (2021): a
user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+
threads". There is a commen there that bounded work is limited but
unbounded work "really [is] not". That discussion led to creation of
IORING_REGISTER_IOWQ_MAX_WORKERS.
-
liburing #349 <https://github.com/axboe/liburing/issues/349> (2021)
reported "unexpected high count of io_worker threads by using the
IOSQE_ASYNC flag".
-
Cloudflare, "Missing Manuals – io_uring worker pool"
<https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/> explains
that unbound workers default to RLIMIT_NPROC for each thread, and that
raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can
backfire. The io_uring_register_iowq_max_workers(3)
<https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html>
man
page says the same.
So, in conclusion, it looks to me that the issue is that req->file is still
NULL when an IOSQE_ASYNC request is classified, and the reads end up in the
unbound pool, and the fix is either to call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
Best regards, Dmytro
On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com> wrote:
Show quoted text
After setting backtrace_functions=md_readv_report, I received more log
information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000:
could not read blocks 143..158 in file "base/6406098/6407866": Operation
canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION:
md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
postgres: postgres test001 [local] CREATE
INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
postgres: postgres test001 [local] CREATE
INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
postgres: postgres test001 [local] CREATE
INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88)
[0x5878e8]
postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05)
[0x5b7325]
postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c)
[0x61fa2c]
postgres: postgres test001 [local] CREATE
INDEX(index_create+0x1328) [0x621608]
postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb)
[0x6d11cb]
postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
postgres: postgres test001 [local] CREATE
INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
/usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291)
[0x93d371]
postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
postgres: postgres test001 [local] CREATE
INDEX(PostgresMain+0x1f1b) [0x937f6b]
postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
postgres: postgres test001 [local] CREATE
INDEX(postmaster_child_launch+0x12d) [0x888ced]
postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
postgres: postgres test001 [local] CREATE
INDEX(PostmasterMain+0xce9) [0x894309]
postgres: postgres test001 [local] CREATE INDEX(main+0x1d0)
[0x531bc0]
/lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
postgres: postgres test001 [local] CREATE INDEX(_start+0x25)
[0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT:
CREATE INDEX idx_sys_notification_channel_notification_id ON
dhr.sys_notification_channel USING btree (c_notification_id);2026年9月28日 11:34，PG Bug reporting form <noreply@postgresql.org> 写道：
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:Hello,
I can reproduce a read failure during pg_restore with io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.I have not measured the failure rate or prepared a self-contained
minimal reproducer.One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?Thank you.
Hi Dmytro,
Thanks for reproducing the issue and for the detailed analysis and proposed fixes. This is very helpful.
Over the past few days, normal SQL queries in our test environment have also triggered this error. To work around the issue in the current version, I’ve switched all our production environments to the default io_method = worker setting.
Show quoted text
2026年10月22日 01:44，Dmytro Astapov <dastapov@gmail.com> 写道：
Hi!
I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think that I have the explanation for how it happens + proposal for the fix as well.
What I think is happening: postgres io_uring requests hit the kernel codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this limit is hit, no new workers can't be spawned, but also postgresmaster can't fork() or system() anymore, leading to "could not read blocks X..Y in file ...: Operation canceled"
How it could be fixed: postgres should either call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
Detailed description of the chain of events follows:
1. Postgres asks io_uring to send buffered reads to worker threads
In method_io_uring.c (line 441), once a backend already has more than 4 I/Os in flight, every further buffered read gets IOSQE_ASYNC:
if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);If `maintenance_io_concurrency` is higher that 4, then all of the maintenance reads starting from 5th will take this path.
2. The kernel places these reads into the pool whose only cap is RLIMIT_NPROC
Each task's io_uring worker pool has two halves:
- bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).Requests are classified in io_prep_async_work (https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492):
if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
else if (!req->file || !S_ISBLK(...)) {
if (def->unbound_nonreg_file) // set for READ/READV
atomic_or(IO_WQ_WORK_UNBOUND, ...);
}However, io_init_req sets req->file = NULL (at https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046), and the file is only attached later, at issue time. An IOSQE_ASYNC request goes straight though io_queue_iowq() to io_prep_async_work() while `req->file` is still NULL.
So a read of a regular file ends up routed to the unbound pool, whose cap is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead of waiting for a running one.
3. Workers get created far faster than the actual I/O needs
In io_wq_dec_running, when a worker blocks and it was the last running worker while work is still queued, the kernel queues creation of a brand-new worker immediately.
It doesn't wake one of the idle workers. On a cold buffer cache almost every read blocks on disk, so this code path fires constantly. As a result, the new worker handles one item and then goes idle. Only the head of the free list gets reused, so the rest of the workers sit idle until WORKER_IDLE_TIMEOUT = 5s expires and then they exit.
As a result, the steady-state thread count is therefore roughly "creation rate × 5 s", not "I/Os in flight". The cap for the workers, as was mentioned earlier, is RLIMIT_NPROC.
Similar threads or bugs from the past
Justin Pryzby on pgsql-hackers <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html> (2025-10-17) reported could not read blocks 122..126 ...: Operation canceled from pg_dump under io_uring. He guessed it might be related to the io_uring file descriptors leaking after crash-restarts, but I cannot find a resolution.
Alexander Lakhin's AIO stress test <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html> (pgsql-hackers, April 2025, before 18.0) hit could not read blocks 0..0 in file "global/1213": Operation canceled with parallel createdb/dropdb jobs, on kernels 6.8 and 6.11. The bugs fixed later in that thread (1 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html>, 2 <https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html>) were different ones, a pg_aios crash and a PANIC from interrupt handling. As far as I can see, the ECANCELED was never explained or fixed.io_uring bugs that look relevant to me
liburing #420 <https://github.com/axboe/liburing/issues/420> (2021): a user found IOSQE_ASYNC "spawns a thread for each request, meaning 10000+ threads". There is a commen there that bounded work is limited but unbounded work "really [is] not". That discussion led to creation of IORING_REGISTER_IOWQ_MAX_WORKERS.
liburing #349 <https://github.com/axboe/liburing/issues/349> (2021) reported "unexpected high count of io_worker threads by using the IOSQE_ASYNC flag".
Cloudflare, "Missing Manuals – io_uring worker pool" <https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/> explains that unbound workers default to RLIMIT_NPROC for each thread, and that raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can backfire. The io_uring_register_iowq_max_workers(3) <https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html> man page says the same.So, in conclusion, it looks to me that the issue is that req->file is still NULL when an IOSQE_ASYNC request is classified, and the reads end up in the unbound pool, and the fix is either to call io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.
Best regards, Dmytro
On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>> wrote:
After setting backtrace_functions=md_readv_report, I received more log information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000: could not read blocks 143..158 in file "base/6406098/6407866": Operation canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION: md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
postgres: postgres test001 [local] CREATE INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
postgres: postgres test001 [local] CREATE INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
postgres: postgres test001 [local] CREATE INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
postgres: postgres test001 [local] CREATE INDEX(heap_getnext+0x88) [0x5878e8]
postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05) [0x5b7325]
postgres: postgres test001 [local] CREATE INDEX(index_build+0x19c) [0x61fa2c]
postgres: postgres test001 [local] CREATE INDEX(index_create+0x1328) [0x621608]
postgres: postgres test001 [local] CREATE INDEX(DefineIndex+0x9bb) [0x6d11cb]
postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
postgres: postgres test001 [local] CREATE INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
/usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291) [0x93d371]
postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
postgres: postgres test001 [local] CREATE INDEX(PostgresMain+0x1f1b) [0x937f6b]
postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
postgres: postgres test001 [local] CREATE INDEX(postmaster_child_launch+0x12d) [0x888ced]
postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
postgres: postgres test001 [local] CREATE INDEX(PostmasterMain+0xce9) [0x894309]
postgres: postgres test001 [local] CREATE INDEX(main+0x1d0) [0x531bc0]
/lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
postgres: postgres test001 [local] CREATE INDEX(_start+0x25) [0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT: CREATE INDEX idx_sys_notification_channel_notification_id ON dhr.sys_notification_channel USING btree (c_notification_id);2026年9月28日 11:34，PG Bug reporting form <noreply@postgresql.org <mailto:noreply@postgresql.org>> 写道：
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com <mailto:weijie1006jl@gmail.com>
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:Hello,
I can reproduce a read failure during pg_restore with io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.I have not measured the failure rate or prepared a self-contained
minimal reproducer.One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?Thank you.
Hi,
A follow-up to my earlier message. On further examination, *this is a
kernel bug, not a Postgres bug*. The changes I suggested for Postgres
(calling io_uring_register_iowq_max_workers(), or not setting IOSQE_ASYNC)
would only be workarounds, and my suggestion was therefore wrong.
As a quick recap, Postgres assumes that a buffered read of a regular file,
submitted with IOSQE_ASYNC, goes to io_uring's bounded worker pool. This
assumption is valid. That pool is limited to min(SQ ring size, 4 × CPUs),
which for Postgres is about io_max_concurrency workers per backend. This is
how io_uring is meant to work: reads and writes of regular files and block
devices are bounded work.
The kernel instead sends these requests to the unbound pool, whose only
limit is RLIMIT_NPROC. It decides the pool in io_prep_async_work(), but for
an IOSQE_ASYNC request the file has not been looked up yet at that point (
req->file is NULL), so the read is treated as "not a regular file". With
enough backends and a cold cache, the worker threads use up RLIMIT_NPROC.
After that, fork() fails and queued reads complete with -ECANCELED.
This bug was introduced in Linux 5.18 by commit 6bf9c47a3989 ("io_uring:
defer file assignment"), which was also marked for 5.15+ stable kernels:
https://github.com/torvalds/linux/commit/6bf9c47a398911e0ab920e362115153596c80432
A fix was made on 2025-10-15, and is queued for 7.4 in Jens Axboe's tree,
as commit 98bc044d5a53 ("io_uring: look up the file before queueing to
io-wq"):
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?id=98bc044d5a5374f8d3605fe55a8e2df6d6fc3f82
It is not (yet?) tagged for stable releases, so older kernels might only
get it if someone backports it. Until then, io_method = worker is the safe
choice, as you have already done.
I have a small C program that reproduces the issue (no Postgres needed),
which could be used to check the if host is susceptible to this problem.
Let me know if anyone wants it.
Best regards, Dmytro
On Fri, Oct 23, 2026 at 8:16 AM weijie JL <weijie1006jl@gmail.com> wrote:
Show quoted text
Hi Dmytro,
Thanks for reproducing the issue and for the detailed analysis and
proposed fixes. This is very helpful.Over the past few days, normal SQL queries in our test environment have
also triggered this error. To work around the issue in the current version,
I’ve switched all our production environments to the default io_method =
worker setting.2026年10月22日 01:44，Dmytro Astapov <dastapov@gmail.com> 写道：
Hi!
I can reproduce this on version 18.6 + kernel 6.12.109.el8, and I think
that I have the explanation for how it happens + proposal for the fix as
well.*What I think is happening*: postgres io_uring requests hit the kernel
codepath where new io workers are spawned up to RLIMIT_NPRPOC. Once this
limit is hit, no new workers can't be spawned, but also postgresmaster
can't fork() or system() anymore, leading to "could not read blocks X..Y
in file ...: Operation canceled"*How it could be fixed*: postgres should either call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.*Detailed description of the chain of events follows:*
*1. Postgres asks io_uring to send buffered reads to worker threads*
In method_io_uring.c (line 441), once a backend already has more than 4
I/Os in flight, every further buffered read gets IOSQE_ASYNC:if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);If `maintenance_io_concurrency` is higher that 4, then all of the
maintenance reads starting from 5th will take this path.*2. The kernel places these reads into the pool whose only cap is
RLIMIT_NPROC*
Each task's io_uring worker pool has two halves:
- bounded (meant for regular files), capped at min(sq_entries, 4*ncpu);
- unbound, capped only at the task's RLIMIT_NPROC (io-wq.c).Requests are classified in io_prep_async_work (
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L480-L492
):if (req->file && (req->flags & REQ_F_ISREG)) { ... bound ... }
else if (!req->file || !S_ISBLK(...)) {
if (def->unbound_nonreg_file) // set for READ/READV
atomic_or(IO_WQ_WORK_UNBOUND, ...);
}However, io_init_req sets req->file = NULL (at
https://github.com/torvalds/linux/blob/v6.12/io_uring/io_uring.c#L2046),
and the file is only attached later, at issue time. An IOSQE_ASYNC request
goes straight though io_queue_iowq() to io_prep_async_work() while
`req->file` is still NULL.So a read of a regular file ends up routed to the unbound pool, whose cap
is RLIMIT_NPROC (65536), not a capped one. IOSQE_ASYNC also marks the work
as IO_WQ_WORK_CONCURRENT, which lets each enqueue create a worker instead
of waiting for a running one.*3. Workers get created far faster than the actual I/O needs*
In io_wq_dec_running, when a worker blocks and it was the last running
worker while work is still queued, the kernel queues creation of a
brand-new worker immediately.It doesn't wake one of the idle workers. On a cold buffer cache almost
every read blocks on disk, so this code path fires constantly. As a result,
the new worker handles one item and then goes idle. Only the head of the
free list gets reused, so the rest of the workers sit idle until
WORKER_IDLE_TIMEOUT = 5s expires and then they exit.As a result, the steady-state thread count is therefore roughly "creation
rate × 5 s", not "I/Os in flight". The cap for the workers, as was
mentioned earlier, is RLIMIT_NPROC.*Similar threads or bugs from the past*
- Justin Pryzby on pgsql-hackers
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg208462.html> (2025-10-17)
reported could not read blocks 122..126 ...: Operation canceled from
pg_dump under io_uring. He guessed it might be related to the io_uring
file descriptors leaking after crash-restarts, but I cannot find a
resolution.
- Alexander Lakhin's AIO stress test
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196158.html> (pgsql-hackers,
April 2025, before 18.0) hit could not read blocks 0..0 in file
"global/1213": Operation canceled with parallel createdb/dropdb jobs,
on kernels 6.8 and 6.11. The bugs fixed later in that thread (1
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg196804.html>
, 2
<https://www.mail-archive.com/pgsql-hackers@lists.postgresql.org/msg197025.html>)
were different ones, a pg_aios crash and a PANIC from interrupt
handling. As far as I can see, the ECANCELED was never explained or fixed.io_uring bugs that look relevant to me
- liburing #420 <https://github.com/axboe/liburing/issues/420> (2021):
a user found IOSQE_ASYNC "spawns a thread for each request, meaning
10000+ threads". There is a commen there that bounded work is limited but
unbounded work "really [is] not". That discussion led to creation of
IORING_REGISTER_IOWQ_MAX_WORKERS.
- liburing #349 <https://github.com/axboe/liburing/issues/349> (2021)
reported "unexpected high count of io_worker threads by using the
IOSQE_ASYNC flag".
- Cloudflare, "Missing Manuals – io_uring worker pool"
<https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/> explains
that unbound workers default to RLIMIT_NPROC for each thread, and that
raising RLIMIT_NPROC without IORING_REGISTER_IOWQ_MAX_WORKERS can
backfire. The io_uring_register_iowq_max_workers(3)
<https://man7.org/linux/man-pages/man3/io_uring_register_iowq_max_workers.3.html> man
page says the same.So, in conclusion, it looks to me that the issue is that req->file is
still NULL when an IOSQE_ASYNC request is classified, and the reads end
up in the unbound pool, and the fix is either to call
io_uring_register_iowq_max_workers, or don't set IOSQE_ASYNC for reads.Best regards, Dmytro
On Sun, Oct 4, 2026 at 10:51 AM weijie JL <weijie1006jl@gmail.com> wrote:
After setting backtrace_functions=md_readv_report, I received more log
information:
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOG: XX000:
could not read blocks 143..158 in file "base/6406098/6407866": Operation
canceled
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]LOCATION:
md_readv_report, md.c:2059
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]BACKTRACE:
postgres: postgres test001 [local] CREATE INDEX() [0x92eafc]
postgres: postgres test001 [local] CREATE INDEX() [0x92f7ec]
postgres: postgres test001 [local] CREATE
INDEX(pgaio_io_call_complete_shared+0xfa) [0x8ef3ea]
postgres: postgres test001 [local] CREATE INDEX() [0x8f209c]
postgres: postgres test001 [local] CREATE INDEX() [0x8f2b63]
postgres: postgres test001 [local] CREATE INDEX() [0x8f0127]
postgres: postgres test001 [local] CREATE
INDEX(WaitReadBuffers+0x1f7) [0x8f5137]
postgres: postgres test001 [local] CREATE
INDEX(read_stream_next_buffer+0x211) [0x8f6ac1]
postgres: postgres test001 [local] CREATE INDEX() [0x5874d6]
postgres: postgres test001 [local] CREATE
INDEX(heap_getnext+0x88) [0x5878e8]
postgres: postgres test001 [local] CREATE INDEX() [0x58dcbd]
postgres: postgres test001 [local] CREATE INDEX() [0x5b11ae]
postgres: postgres test001 [local] CREATE INDEX(btbuild+0xb05)
[0x5b7325]
postgres: postgres test001 [local] CREATE
INDEX(index_build+0x19c) [0x61fa2c]
postgres: postgres test001 [local] CREATE
INDEX(index_create+0x1328) [0x621608]
postgres: postgres test001 [local] CREATE
INDEX(DefineIndex+0x9bb) [0x6d11cb]
postgres: postgres test001 [local] CREATE INDEX() [0xb18111]
postgres: postgres test001 [local] CREATE
INDEX(standard_ProcessUtility+0x3f8) [0x93c1d8]
/usr/pgsql-18/lib/pg_stat_statements.so(+0x7375) [0x7f209b366375]
postgres: postgres test001 [local] CREATE INDEX() [0x93cc3a]
postgres: postgres test001 [local] CREATE INDEX() [0x93cdaf]
postgres: postgres test001 [local] CREATE INDEX(PortalRun+0x291)
[0x93d371]
postgres: postgres test001 [local] CREATE INDEX() [0x935ab0]
postgres: postgres test001 [local] CREATE
INDEX(PostgresMain+0x1f1b) [0x937f6b]
postgres: postgres test001 [local] CREATE INDEX() [0x9389d5]
postgres: postgres test001 [local] CREATE
INDEX(postmaster_child_launch+0x12d) [0x888ced]
postgres: postgres test001 [local] CREATE INDEX() [0x891cb2]
postgres: postgres test001 [local] CREATE
INDEX(PostmasterMain+0xce9) [0x894309]
postgres: postgres test001 [local] CREATE INDEX(main+0x1d0)
[0x531bc0]
/lib64/libc.so.6(+0x2a610) [0x7f209b82a610]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f209b82a6c0]
postgres: postgres test001 [local] CREATE INDEX(_start+0x25)
[0x5321b5]
2026-10-04 17:46:08.322 CST [local] test001 postgres [154450]STATEMENT:
CREATE INDEX idx_sys_notification_channel_notification_id ON
dhr.sys_notification_channel USING btree (c_notification_id);2026年9月28日 11:34，PG Bug reporting form <noreply@postgresql.org> 写道：
The following bug has been logged on the website:
Bug reference: 19725
Logged by: weijie JL
Email address: weijie1006jl@gmail.com
PostgreSQL version: 18.6
Operating system: RockyLinux9
Description:Hello,
I can reproduce a read failure during pg_restore with
io_method=io_uring.
The error disappears with worker and recurs after switching back to
io_uring.Environment:
- PostgreSQL 18.6, x86_64
- VMware VM: 8 vCPUs, 32 GB RAM
- Kernel: 5.14.0-611.55.1.el9_7.x86_64Settings:
effective_io_concurrency = 256
maintenance_io_concurrency = 16
max_parallel_maintenance_workers = 2Restore command:
nohup pg_restore -d test001 dhr.dump/ -j8 -v > test.log 2>&1 &One failing statement:
ALTER TABLE ONLY dhr.obj_permission_object
ADD CONSTRAINT obj_permission_object_pkey PRIMARY KEY (id);Server error:
ERROR: could not read blocks 143..158 in file "base/1684908/1686127":
Operation canceled
LOCATION: md_readv_report, md.c:2059
SQLSTATE: XX000Observed test sequence:
1. io_method=io_uring: the read error occurred during index creation.
2. Changed io_method to worker: retried without errors.
3. Changed io_method back to io_uring: the read error was reproduced.I have not measured the failure rate or prepared a self-contained
minimal reproducer.One earlier run also reported "could not map dynamic shared memory
segment" from parallel workers immediately after the read error.
Those DSM errors have not recurred in subsequent failing tests.Does this match a known PostgreSQL bug or kernel-side io_uring issue?
What additional logs or tracing would help identify the cause?Thank you.