aio tests failing on newer Linux kernels
Hello hackers,
Multiple failures produced recently by debian animals, e.g. [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kestrel&dt=2026-07-25%2010%3A13%3A20, [2]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=melonworm&dt=2026-07-25%2017%3A18%3A24, [3]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-07-25%2021%3A15%3A22:
pgsql.build/testrun/test_aio/001_aio/log/regress_log_001_aio
...
[23:22:28.951](0.000s) ok 280 - worker: failure to open: recovers: expected stderr
### Stopping node "worker" using mode fast
# Running: pg_ctl --pgdata
/home/bf/bf-build/tamandua/HEAD/pgsql.build/testrun/test_aio/001_aio/data/t_001_aio_worker_data/pgdata --mode fast stop
waiting for server to shut down.... done
server stopped
# No postmaster PID for node "worker"
### Starting node "io_uring"
# Running: pg_ctl --wait --pgdata
/home/bf/bf-build/tamandua/HEAD/pgsql.build/testrun/test_aio/001_aio/data/t_001_aio_io_uring_data/pgdata --log
/home/bf/bf-build/tamandua/HEAD/pgsql.build/testrun/test_aio/001_aio/log/001_aio_io_uring.log --options
--cluster-name=io_uring start
waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
# pg_ctl start failed; see logfile for details:
/home/bf/bf-build/tamandua/HEAD/pgsql.build/testrun/test_aio/001_aio/log/001_aio_io_uring.log
# No postmaster PID for node "io_uring"
[23:22:29.992](1.040s) Bail out! pg_ctl start failed
pgsql.build/testrun/test_aio/001_aio/log/001_aio_io_uring.log
...
2026-07-25 23:22:29.949 CEST [2396766][postmaster][:0] FATAL: could not setup io_uring queue: Cannot allocate memory
...
show a new anomaly, probably caused by a kernel update. I can't see in
the test log, which kernels the animals use, but I reproduced the same
failure locally, on debian sid, having multiplied test/modules/test_aio
x30 and executed:
$ meson test test_aio_* -j 30
...
12/120 test_aio_2 - postgresql:test_aio_2/004_read_stream OK 10.26s 31 subtests passed
13/120 test_aio_5 - postgresql:test_aio_5/004_read_stream OK 10.41s 31 subtests passed
14/120 test_aio_7 - postgresql:test_aio_7/003_initdb ERROR 11.75s (exit status 255 or 0xff)
...
119/120 test_aio_28 - postgresql:test_aio_28/001_aio OK 31.15s 826 subtests passed
120/120 test_aio_30 - postgresql:test_aio_30/003_initdb OK 26.77s 6 subtests passed
Ok: 93
Fail: 27
$ grep -r 'could not setup io_uring queue: Cannot allocate memory' testrun/ | wc -l
27
(The tests pass for me with -j 10.)
This is reproduced on
Linux debian-13 6.19.6+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.19.6-1 (2026-03-06) x86_64 GNU/Linux
but not reproduced on:
Linux debian-13 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64
AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kestrel&dt=2026-07-25%2010%3A13%3A20
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=melonworm&dt=2026-07-25%2017%3A18%3A24
[3]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-07-25%2021%3A15%3A22
Best regards,
Alexander
Hi,
Multiple failures produced recently by debian animals, e.g. [1], [2], [3]:
pgsql.build/testrun/test_aio/001_aio/log/regress_log_001_aio
...This is reproduced on
Linux debian-13 6.19.6+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.19.6-1 (2026-03-06) x86_64 GNU/Linux
but not reproduced on:
Linux debian-13 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.
Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
liburing 2.5 and the tests pass fine for me.
--
Best regards,
Aleksander Alekseev
On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.
No idea about the AI argument, but..
Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
liburing 2.5 and the tests pass fine for me.
You are not alone. Various buildfarm members are also turning red
with random failures across multiple tests because of this issue.
--
Michael
Hello Aleksander and Michael,
28.07.2026 05:24, Michael Paquier пишет:
On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.No idea about the AI argument, but..
Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
liburing 2.5 and the tests pass fine for me.You are not alone. Various buildfarm members are also turning red
with random failures across multiple tests because of this issue.
I think, I've found the difference -- on Debian I'm seeing:
user@debian:~$ ulimit -l
8192
vs
vagrant@ubuntu:~$ ulimit -l
2044616
Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64
GNU/Linux
$ meson test test_aio_* -j 30
...
Ok: 101
Expected Fail: 0
Fail: 19
Unexpected Pass: 0
Also reproduced on Fedora, with:
Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
$ ulimit -l
8192
(default)
Best regards,
Alexander
Hi,
On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
Hello Aleksander and Michael,
28.07.2026 05:24, Michael Paquier пишет:
On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.No idea about the AI argument, but..
Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
liburing 2.5 and the tests pass fine for me.You are not alone. Various buildfarm members are also turning red
with random failures across multiple tests because of this issue.I think, I've found the difference -- on Debian I'm seeing:
user@debian:~$ ulimit -l
8192
vs
vagrant@ubuntu:~$ ulimit -l
2044616Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux$ meson test test_aio_* -j 30
...
Ok: 101
Expected Fail: 0
Fail: 19
Unexpected Pass: 0Also reproduced on Fedora, with:
Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
$ ulimit -l
8192
(default)
Out of interest, I let Sol do some investigations for this issue. The
conclusion it reached seems to align with that of the AI tool used by
Alexander. However, I lacked the expertise to fully review/confirm it.
Not sure whether it is acceptable to post a 'could-be-sloppy' report
here.
--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
Attachments:
io_uring_enomem_investigation_rev3.mdtext/markdown; charset=UTF-8; name=io_uring_enomem_investigation_rev3.mdDownload
On Tue, Jul 28, 2026 at 7:15 AM Xuneng Zhou <xunengzhou@gmail.com> wrote:
Hi,
On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
Hello Aleksander and Michael,
28.07.2026 05:24, Michael Paquier пишет:
On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
AI suggests the new behavior is caused by the "io_uring: use region api
for SQ/CQ" change landed in the kernel version 6.14.No idea about the AI argument, but..
Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
liburing 2.5 and the tests pass fine for me.You are not alone. Various buildfarm members are also turning red
with random failures across multiple tests because of this issue.I think, I've found the difference -- on Debian I'm seeing:
user@debian:~$ ulimit -l
8192
vs
vagrant@ubuntu:~$ ulimit -l
2044616Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux$ meson test test_aio_* -j 30
...
Ok: 101
Expected Fail: 0
Fail: 19
Unexpected Pass: 0Also reproduced on Fedora, with:
Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
$ ulimit -l
8192
(default)Out of interest, I let Sol do some investigations for this issue. The
conclusion it reached seems to align with that of the AI tool used by
Alexander. However, I lacked the expertise to fully review/confirm it.
Not sure whether it is acceptable to post a 'could-be-sloppy' report
here.
Interesting. To me that output makes a lot of sense. We are having plenty of
io_urings allocated (one for each connection up to max_connections), they are
small, but the tests allocate many separate clusters too and all of this hits
the ulimit of max locked memory. I think the 8MB is the norm (e.g.
systemd-system.conf(5) says "DefaultLimitMEMLOCK= defaults to 8MB and there
can be per-service LimitMEMLOCK too), but Ubuntu seems to be way less
conservative and more open as it's mostly for users with X11/audio daemons
(pipewire) and stuff like that.
Sadly soft:hard limit is the same, so we won't be able to bump it like in [1]/messages/by-id/DJS6KELUH3T0.2DGCBB9I190X@jeltef.nl
Dunno what to do, it's somewhat silent(?) kernel change: reduce parallelism of
the tests maybe with even lower max_connections, bump DefaultLimitMEMLOCK ordo
it in per-service (but dunno how's buildfarm env looks like), bring back older
kernels (but what about security fixes?)
If that AI report is to believed we are going to need to update docs as it
will be hit in the wild too:
- as max_connections=1k + io_uring won't possible on newest kernels (but I
haven't verified the math behind it)
- on non-standard > 4kB page systems (ARM?) same thing might be happening
(2 pages won't 8192 there, but much more hitting earlier the ENOMEM
-J.