huge_pages=on cause could not map anonymous shared memory: Cannot allocate memory
Hi,
I have problem to run postgresql on ubuntu 24.04 server with huge_pages =
on. My instance have 8GB ram and 2 vcpus (t3a.large). My configuration is:
max_connections = 1000
superuser_reserved_connections = 3
shared_buffers = 1960MB
effective_cache_size = 5881MB
huge_pages = on
temp_buffers = 32MB
max_prepared_transactions = 100
work_mem = 1MB
maintenance_work_mem = 392MB
max_stack_depth = 4MB
dynamic_shared_memory_type = posix
archive_mode = on
max_wal_senders = 10
wal_sender_timeout = 0
min_wal_size = 4GB
max_wal_size = 16GB
wal_level = logical
checkpoint_timeout = 15min
checkpoint_completion_target = 0.9
In my sysct.conf I have:
vm.overcommit_memory=2
vm.overcommit_ratio=50
vm.vfs_cache_pressure=50
vm.dirty_background_ratio=10
vm.dirty_ratio=40
fs.nr_open=10000000
fs.file-max=10000000
vm.nr_hugepages=980
vm.hugetlb_shm_group=1010
my huge meminfo looks:
cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 980
HugePages_Free: 980
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 2007040 kB
When I start postgres instance I get error:
May 07 10:56:32 pg1 pg_ctl[16753]: server starting
May 07 10:56:32 pg1 systemd[1]: Started postgres-17.4-local.service -
PostgreSQL 17.4 database local server.
May 07 10:56:35 pg1 pg_ctl[16756]: 2025-05-07 10:56:35.826 CEST [16756] - [
- - ] [] : XX000FATAL: could not map anonymous shared memory: Cannot
allocate memory
May 07 10:56:35 pg1 pg_ctl[16756]: 2025-05-07 10:56:35.826 CEST [16756] - [
- - ] [] : XX000HINT: This error usually means that PostgreSQL's request
for a shared memory segment exceeded available memory, swap space, or huge
pages. To reduce the request size (currently 2204106752 bytes), reduce
PostgreSQL's shared memory usage, perhaps by reducing "shared_buffers"
or "max_connections".
May 07 10:56:35 pg1 pg_ctl[16756]: 2025-05-07 10:56:35.826 CEST [16756] - [
- - ] [] : 00000LOG: database system is shut down
May 07 10:56:35 pg1 systemd[1]: postgres-17.4-sdx.service: Main process
exited, code=exited, status=1/FAILURE
May 07 10:56:35 pg1 systemd[1]: postgres-17.4-sdx.service: Failed with
result 'exit-code'.
May 07 10:56:35 pg1 systemd[1]: postgres-17.4-sdx.service: Consumed 3.493s
CPU time, 2.8M memory peak, 0B memory swap peak.
When I comment #huge_pages=on in configuration postgresql started without
problems.
Can someone tell me where is the problem?
Regards
Bogdan
Hi Bogdan,
The root cause here is that the number of huge pages you've configured
(vm.nr_hugepages = 980) is not sufficient.
Each huge page on your system is 2 MB in size, so 980 pages give you
roughly 1.96 GB of memory (980 × 2 MB). However, PostgreSQL is clearly
requesting about 2.2 GB of shared memory (specifically, 2204106752 bytes as
shown in the error message you provided), which exceeds what's available
through huge pages.
That’s why PostgreSQL fails to start when huge_pages = on - it requires the
entire shared memory segment to come from huge pages and refuses to fall
back to regular ones.
Earlier, you had the huge_pages setting commented out, which means
PostgreSQL used the default value: huge_pages = try. In that mode, it first
attempts to use huge pages, but if that fails (like in your case due to
insufficient allocation), it falls back to standard memory pages — which is
why the instance started without issues then.
To fix the issue, you should increase vm.nr_hugepages to at least 1100 to
fully cover the shared memory request (you can go a bit higher to be safe
and then reduce it as described in the article I'm pasting the link to).
Also, a side note: max_connections = 1000 is quite high for an instance
with 8 GB of RAM and only 2 vCPUs. Even if huge pages are properly
allocated, such a high number of connections can lead to performance
issues. You might want to consider lowering it or using a connection pooler
like PgBouncer.
If you’d like to understand how huge pages work in PostgreSQL, including
how to calculate memory needs and configure the OS properly, I wrote a
detailed article some time ago (still valid). It’s in Polish, which I
assume is fine for you:
https://linuxpolska.com/pl/baza-wiedzy/blog/postgres-pamieci-ram-tipstricks/
best regards,
Alicja Kucharczyk
Show quoted text
Hi Alicja,
Thanks for your advice, now postgresql works fine with 'huge_pages=on'.
Regards
Bogdan
śr., 7 maj 2025 o 14:17 Alicja Kucharczyk <zaledwie10minut@gmail.com>
napisał(a):
Show quoted text
Hi Bogdan,
The root cause here is that the number of huge pages you've configured
(vm.nr_hugepages = 980) is not sufficient.
Each huge page on your system is 2 MB in size, so 980 pages give you
roughly 1.96 GB of memory (980 × 2 MB). However, PostgreSQL is clearly
requesting about 2.2 GB of shared memory (specifically, 2204106752 bytes as
shown in the error message you provided), which exceeds what's available
through huge pages.That’s why PostgreSQL fails to start when huge_pages = on - it requires
the entire shared memory segment to come from huge pages and refuses to
fall back to regular ones.Earlier, you had the huge_pages setting commented out, which means
PostgreSQL used the default value: huge_pages = try. In that mode, it first
attempts to use huge pages, but if that fails (like in your case due to
insufficient allocation), it falls back to standard memory pages — which is
why the instance started without issues then.To fix the issue, you should increase vm.nr_hugepages to at least 1100 to
fully cover the shared memory request (you can go a bit higher to be safe
and then reduce it as described in the article I'm pasting the link to).Also, a side note: max_connections = 1000 is quite high for an instance
with 8 GB of RAM and only 2 vCPUs. Even if huge pages are properly
allocated, such a high number of connections can lead to performance
issues. You might want to consider lowering it or using a connection pooler
like PgBouncer.If you’d like to understand how huge pages work in PostgreSQL, including
how to calculate memory needs and configure the OS properly, I wrote a
detailed article some time ago (still valid). It’s in Polish, which I
assume is fine for you:
https://linuxpolska.com/pl/baza-wiedzy/blog/postgres-pamieci-ram-tipstricks/best regards,
Alicja Kucharczyk
Kudos to Alicja for that excellent answer.
It would be nice if the Postgres log message was a little more descriptive.
(throws it onto my huge pile of "future maybe enhancement ideas")
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support