Re: New shared memory hooks proposal (was Re: pre_load_libraries)

On Tue, 2006-07-18 at 16:36 -0700, Marc Munro wrote:

The attached patch provides add-ins with the means to register for
shared memory and LWLocks. This greatly improves the ease with which
shared memory may be used from add-ins, while blah blah blah....

I have tried to be patient but this is my first patch and the suspense
is killing me.

I'd like some idea of what is likely to happen to this patch. If it's a
non-starter I'd like to know. Ditto if it needs work. I realise
everyone is busy right now, so I'm also prepared to be told to be
patient; I'd just like to know it hasn't been overlooked and forgotten
in the sudden rush of more interesting patches.

Just to re-iterate, this patch enables Veil
http://pgfoundry.org/projects/veil/ to be a good postgres citizen,
removing its motivation to steal shared memory.

__
Marc

Marc Munro <marc@bloodnok.com> writes:

The attached patch provides add-ins with the means to register for
shared memory and LWLocks. This greatly improves the ease with which
shared memory may be used from add-ins, while blah blah blah....

I have tried to be patient but this is my first patch and the suspense
is killing me.

Sorry about that, but we're at the end of the development cycle and
there's a long list of patches awaiting review :-(. Yours seems
unlikely to affect any other work, so it's not high on the priority
list to get pushed in.

I'd like some idea of what is likely to happen to this patch.

We'll look at it, and you will have an opportunity to fix anything
anyone doesn't like. Most likely this will happen in early August.
If you have schedule constraints (like you're on vacation in August)
you should let us know.

regards, tom lane

I updated the style of your patch, and added a little to your comment
block about how to use this capability. I don't think any additional
documentation is necessary.

Thanks.

---------------------------------------------------------------------------

Marc Munro wrote:
-- Start of PGP signed section.

The attached patch provides add-ins with the means to register for
shared memory and LWLocks. This greatly improves the ease with which
shared memory may be used from add-ins, while keeping the accounting and
management for that shared memory separate.

Specifically it adds named add-in shared memory contexts. From these,
memory can be allocated without affecting the memory available in other
contexts.

Usage is as follows:
from add-in functions called from preload_libraries, you may call
RegisterAddinContext(const * name, size_t size)
to register a new (logical) shared memory segment.

and
RegisterAddinLWLock(LWLockid *lock_ptr);
to request that a LWLock be allocated, placed into *lock_ptr.

The actual creation of the shared memory segment and lwlocks is
performed later as part of shared memory initialisation.

To allocate shared memory from a named context you would use
ShmemAllocFromContext(size_t size, const char *name);

To reset a shared memory context back to its original unused state (from
which new allocations may be performed), you may use
ShmemResetContext(const char *name);

This works for me (for Veil) and make check runs fine.

I have not included any documentation updates in the patch as I'm not
sure where such API changes should be documented.

All comments, questions and suggestions are welcomed.

__
Marc

[ Attachment, skipping... ]
-- End of PGP section, PGP failed!

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Marc Munro <marc@bloodnok.com> writes:

The attached patch provides add-ins with the means to register for
shared memory and LWLocks.

I finally got around to reviewing this patch, and realized that it's got
a pretty fundamental design flaw: it isn't useful under Windows (or any
other EXEC_BACKEND platform), because there isn't any provision for
backends to locate structs allocated by other backends by means of
searching in shared memory. AFAICS the code can only do something
useful in a platform where allocations made in the postmaster process
can be found by backends via fork inheritance of pointers.

The right way to handle shmem allocations is to use ShmemInitStruct
to either allocate a shared struct for the first time or attach to a
previously made instance of the struct. (This "struct" could be a
memory allocation arena itself, but that's not the core code's problem.)
Now we could extend the patch so that each "addin" has its own
ShmemIndex within its private workspace, but I think that's probably
overkill. My inclination is to rip out ShmemAllocFromContext and expect
addins to use ShmemInitStruct the same as everyone else. The hook
callable at shared_preload_libraries time should just serve to add
the necessary amount to the computed size of the shared memory segment.

RegisterAddinLWLock is broken in the same way: it could only be used
safely if the registered lock ID were remembered in shared memory,
but since shared memory doesn't exist at the time it's supposed to be
called, there's no way to do that. Again, it'd seem to work as long as
the lock ID value were inherited via fork, but that's gonna fail on
EXEC_BACKEND builds. I think we should probably take this out in favor
of something that just increments a counter that replaces
NUM_USER_DEFINED_LWLOCKS, and expect people to use LWLockAssign() at an
appropriate time while initializing their shared memory areas.

It strikes me that there's a race condition here, which we've not seen
in previous use because backends expect all standard shared memory
structs to have already been initialized by the postmaster. An add-on
will instead have to operate under the regime of "first backend wanting
to use the struct must init it". Although ShmemInitStruct returns a
"found" bool telling you you've got to init it, there's no interlock
ensuring that you can do so before someone else comes along and tries to
use the struct (which he'll assume is done because he sees found = true).
And, per above discussion, an add-on can't solve this for itself using
an add-on LWLock, because it really has to acquire its add-on locks
while initializing that same shmem struct, which is where it's going to
keep the locks' identity :-(

So I think we need to provide a standard LWLock reserved for the purpose
of synchronizing first-time use of a shmem struct. The coding rules for
an add-on would then look like:

* in the shared_preload_libraries hook:

RequestAddinShmemSpace(size);
RequestAddinLWLocks(n);

* in a backend, to access a shared memory struct:

static mystruct *ptr = NULL;

if (!ptr)
{
bool found;

LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
ptr = ShmemInitStruct("my struct name", size, &found);
if (!ptr)
elog(ERROR, "out of shared memory");
if (!found)
{
initialize contents of shmem area;
acquire any requested LWLocks using:
ptr->mylockid = LWLockAssign();
}
LWLockRelease(AddinShmemInitLock);
}

Thoughts?

regards, tom lane

On Sat, 2006-10-14 at 14:55 -0400, Tom Lane wrote:

Marc Munro <marc@bloodnok.com> writes:

The attached patch provides add-ins with the means to register for
shared memory and LWLocks.

I finally got around to reviewing this patch, and realized that it's got
a pretty fundamental design flaw: it isn't useful under Windows (or any
other EXEC_BACKEND platform), because there isn't any provision for
backends to locate structs allocated by other backends by means of
searching in shared memory. AFAICS the code can only do something
useful in a platform where allocations made in the postmaster process
can be found by backends via fork inheritance of pointers.

Rats! You are right. I had quite overlooked the windows case.

The right way to handle shmem allocations is to use ShmemInitStruct
to either allocate a shared struct for the first time or attach to a
previously made instance of the struct. (This "struct" could be a
memory allocation arena itself, but that's not the core code's problem.)
Now we could extend the patch so that each "addin" has its own
ShmemIndex within its private workspace, but I think that's probably
overkill. My inclination is to rip out ShmemAllocFromContext and expect
addins to use ShmemInitStruct the same as everyone else. The hook
callable at shared_preload_libraries time should just serve to add
the necessary amount to the computed size of the shared memory segment.

I think that works for me.

RegisterAddinLWLock is broken in the same way: it could only be used
safely if the registered lock ID were remembered in shared memory,
but since shared memory doesn't exist at the time it's supposed to be
called, there's no way to do that. Again, it'd seem to work as long as
the lock ID value were inherited via fork, but that's gonna fail on
EXEC_BACKEND builds. I think we should probably take this out in favor
of something that just increments a counter that replaces
NUM_USER_DEFINED_LWLOCKS, and expect people to use LWLockAssign() at an
appropriate time while initializing their shared memory areas.

Agreed.

It strikes me that there's a race condition here, which we've not seen
in previous use because backends expect all standard shared memory
structs to have already been initialized by the postmaster. An add-on
will instead have to operate under the regime of "first backend wanting
to use the struct must init it". Although ShmemInitStruct returns a
"found" bool telling you you've got to init it, there's no interlock
ensuring that you can do so before someone else comes along and tries to
use the struct (which he'll assume is done because he sees found = true).
And, per above discussion, an add-on can't solve this for itself using
an add-on LWLock, because it really has to acquire its add-on locks
while initializing that same shmem struct, which is where it's going to
keep the locks' identity :-(

So I think we need to provide a standard LWLock reserved for the purpose
of synchronizing first-time use of a shmem struct. The coding rules for
an add-on would then look like:

* in the shared_preload_libraries hook:

RequestAddinShmemSpace(size);
RequestAddinLWLocks(n);

* in a backend, to access a shared memory struct:

static mystruct *ptr = NULL;

if (!ptr)
{
bool found;

LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
ptr = ShmemInitStruct("my struct name", size, &found);
if (!ptr)
elog(ERROR, "out of shared memory");
if (!found)
{
initialize contents of shmem area;
acquire any requested LWLocks using:
ptr->mylockid = LWLockAssign();
}
LWLockRelease(AddinShmemInitLock);
}

Thoughts?

I am content that what you suggest is the right way to go. I will work
on a new patch immediately, unless you would prefer to do this yourself.

It seems to me that these coding rules should be documented in the main
documentation, I guess in the section that describes Dynamic Loading.
Would you like me to take a stab at that?

__
Marc

Marc Munro <marc@bloodnok.com> writes:

I am content that what you suggest is the right way to go. I will work
on a new patch immediately, unless you would prefer to do this yourself.

I've already got some of the relevant changes made in my local copy,
so I might as well just go ahead and finish the coding work. But if
you'd like to do the documentation part, be my guest.

It seems to me that these coding rules should be documented in the main
documentation, I guess in the section that describes Dynamic Loading.

I'm not entirely sure where to put it, either. Subsection 32.9.1 (Dynamic
Loading) is pretty basic stuff for first-time C-function authors ---
this seems off-topic for that section. Maybe a new subsection down near
the end of 32.9? Another possibility is somewhere in the Internals
chapters, although I don't offhand see an obvious place there either.

regards, tom lane

On Sun, 2006-10-15 at 16:36 -0400, Tom Lane wrote:

I'm not entirely sure where to put it, either. Subsection 32.9.1 (Dynamic
Loading) is pretty basic stuff for first-time C-function authors ---
this seems off-topic for that section. Maybe a new subsection down near
the end of 32.9? Another possibility is somewhere in the Internals
chapters, although I don't offhand see an obvious place there either.

This patch, against xfunc.sgml, adds a new subsection 33.9.12, Shared
Memory and LWLocks in C-Language Functions, describing how shared memory
and lwlocks may be requested by C add-in functions.

I have, btw, verified that this works in the forthcoming release of
Veil.

__
Marc

Patch applied. Thanks. Your documentation changes can be viewed in
five minutes using links on the developer's page,
http://www.postgresql.org/developer/testing.

---------------------------------------------------------------------------

Marc Munro wrote:
-- Start of PGP signed section.

On Sun, 2006-10-15 at 16:36 -0400, Tom Lane wrote:

I'm not entirely sure where to put it, either. Subsection 32.9.1 (Dynamic
Loading) is pretty basic stuff for first-time C-function authors ---
this seems off-topic for that section. Maybe a new subsection down near
the end of 32.9? Another possibility is somewhere in the Internals
chapters, although I don't offhand see an obvious place there either.

This patch, against xfunc.sgml, adds a new subsection 33.9.12, Shared
Memory and LWLocks in C-Language Functions, describing how shared memory
and lwlocks may be requested by C add-in functions.

I have, btw, verified that this works in the forthcoming release of
Veil.

__
Marc

[ Attachment, skipping... ]
-- End of PGP section, PGP failed!

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +