Shared memory and processes

On Thu, Apr 28, 2016 at 12:56 PM, <david@andl.org> wrote:

Does Postgres provide a convenient way for one process to pass data to
another using shared memory?

1. Look at ShmemInitStruct, ShmemInitHash (as in hash table),
ShmemInitQueue in storage/shmem.h. These use memory that is mapped at
the same address in every backend (process) which is convenient for
sharing data structures with internal pointers. The amount of memory
available is fixed at cluster startup however.

2. Look at dsm_XXX in storage/dsm.h. This subsystem provides
segments of memory that is "dynamic" in the sense that it is limited
only by your system's available virtual memory, but you have to
explicitly attach these segment in any backend that wants to access
them by passing a handle around and the memory may be mapped at any
address in each backend, so you need to work harder to build data
structures that reference each other (using offsets rather than
pointers, that kind of thing). DSM segments won't work well if you
try to create large numbers of them, so you'll need to provide a way
to manage the space inside a smallish number of large chunks of DSM
memory yourself (this is a problem I'm working to fix by providing a
general purpose allocator backed by a bunch of DSM segments -- watch
this space). LWLocks (our usual lock primitive for cases where
spinlocks are inappropriate) currently don't work correctly inside DSM
segments (this too will be fixed).

--
Thomas Munro
http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Apr 28, 2016 at 9:56 AM, <david@andl.org> wrote:

Does Postgres provide a convenient way for one process to pass data to
another using shared memory?

If you are talking about enabling the use of shared memory by 3rd-part
plugins or modules, there is shmem_startup_hook for this purpose.
Another thing that you may want to look at is DSM (dynamic shared
memory), with its interface in dsm.h. An example of use is in
src/test/modules/test_shm_mq.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Thomas Munro [mailto:thomas.munro@enterprisedb.com]

Does Postgres provide a convenient way for one process to pass data to
another using shared memory?

1. Look at ShmemInitStruct, ShmemInitHash (as in hash table), ShmemInitQueue
in storage/shmem.h. These use memory that is mapped at the same address in
every backend (process) which is convenient for sharing data structures with
internal pointers. The amount of memory available is fixed at cluster
startup however.

Thanks. That limit could be an issue. The notes in shmem.c are helpful.

2. Look at dsm_XXX in storage/dsm.h. This subsystem provides segments of
memory that is "dynamic" in the sense that it is limited only by your
system's available virtual memory, but you have to explicitly attach these
segment in any backend that wants to access them by passing a handle around
and the memory may be mapped at any address in each backend, so you need to
work harder to build data structures that reference each other (using offsets
rather than pointers, that kind of thing). DSM segments won't work well if
you try to create large numbers of them, so you'll need to provide a way to
manage the space inside a smallish number of large chunks of DSM memory
yourself (this is a problem I'm working to fix by providing a general purpose
allocator backed by a bunch of DSM segments -- watch this space). LWLocks
(our usual lock primitive for cases where spinlocks are inappropriate)
currently don't work correctly inside DSM segments (this too will be fixed).

I've now found this through the test_shm_mq sample. Looks like an answer, if quite a bit of machinery needed.

Thanks for the pointers.

Regards
David M Bennett FACS

Andl - A New Database Language - andl.org

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Michael Paquier [mailto:michael.paquier@gmail.com]

Does Postgres provide a convenient way for one process to pass data to
another using shared memory?

If you are talking about enabling the use of shared memory by 3rd-part
plugins or modules, there is shmem_startup_hook for this purpose.

Thanks -- looks interesting.

Another thing that you may want to look at is DSM (dynamic shared memory),
with its interface in dsm.h. An example of use is in src/test/modules/test_shm_mq.

Yes, that is useful. I'm curious why the documentation (F.41) was removed between 9.4 and 9.5 -- is there a problem?

Regards
David M Bennett FACS

Andl - A New Database Language - andl.org

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Apr 28, 2016 at 12:24 PM, <david@andl.org> wrote:

From: Michael Paquier [mailto:michael.paquier@gmail.com]
Another thing that you may want to look at is DSM (dynamic shared memory),
with its interface in dsm.h. An example of use is in src/test/modules/test_shm_mq.

Yes, that is useful. I'm curious why the documentation (F.41) was removed between 9.4 and 9.5 -- is there a problem?

No, some of the contrib/ modules have been moved in the code tree and
are now considered as only test modules, which do not get installed by
default. As a result, their documentation has been removed.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers