Running docker in postgres, SHM size of the docker container in postgres 16
Hello all,
Assume a machine with 16 CPU cores and 64GB of RAM. 300 max connections.
Postgres is running here, inside a docker container. Docker containers
receive a small amount of shared memory from /dev/shm
Experience teaches that the default 64MB of /dev/shm for the postgres
container, is not enough for databases of even 50GB in size. Previously
I've increased the /dev/shm setting to 256MB for the postgres container.
Issues that made this necessary all came in the form of too many queries
running at once, asking too much memory space, I assume.
I recently upgraded a setup from postgres 11 to postgres 16 via logical
replication, and found that running "vacuum(analyze, verbose)" fails unless
I give the container 1GB of /dev/shm. Where previously 256MB sufficed on
the old postgres 11 version.
Why would that be? It's the exact same data. The install is about 50GB in
size. Is there something wrong with postgres 16, or did some settings
significantly change, that I need to know about? I went over all the
changelogs, nothing stood out as relevant to this, but that might be a lack
of experience on my end.
Regards,
Koen De Groote
On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote:
Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end.
Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic
shared memory" to share state between workers, and assuming
dynamic_shared_memory_type=posix, that means shm_open(), which opens
files under /dev/shm on Linux.
That explains a lot.
I have the default of 2 max_parallel_maintenance_workers set, should I set
this to 0?
I realize this is of course an improvement, but working with docker
containers, I'd like to avoid taking /dev/shm away from regular queries.
I assume setting max_parallel_maintenance_workers to 0 is the fix here, is
there perhaps something else I should know about, if I want to have control
over this?
Regards,
Koen De Groote
On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <thomas.munro@gmail.com>
wrote:
Show quoted text
On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote:
Why would that be? It's the exact same data. The install is about 50GB
in size. Is there something wrong with postgres 16, or did some settings
significantly change, that I need to know about? I went over all the
changelogs, nothing stood out as relevant to this, but that might be a lack
of experience on my end.Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic
shared memory" to share state between workers, and assuming
dynamic_shared_memory_type=posix, that means shm_open(), which opens
files under /dev/shm on Linux.
Docker databases do not perform well under heavy load.
Everything can be configured to improve performance but going through the Docker disk access layer will significantly decrease performance.
On Wednesday, November 20, 2024 at 06:06:36 AM GMT-5, Koen De Groote <kdg.dev@gmail.com> wrote:
That explains a lot.
I have the default of 2 max_parallel_maintenance_workers set, should I set this to 0?
I realize this is of course an improvement, but working with docker containers, I'd like to avoid taking /dev/shm away from regular queries.
I assume setting max_parallel_maintenance_workers to 0 is the fix here, is there perhaps something else I should know about, if I want to have control over this?
Regards,Koen De Groote
On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <thomas.munro@gmail.com> wrote:
On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote:
Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end.
Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic
shared memory" to share state between workers, and assuming
dynamic_shared_memory_type=posix, that means shm_open(), which opens
files under /dev/shm on Linux.