LISTEN/NOTIFY enhancement: Portable signal handling?

Basically I want to introduce formal support for turning
PostgreSQL into a message bus. To start with, it won't be a
scalable message bus that can scale to thousands of
concurrent connections (that's something I'd like to do long
term(tm), but I digress). The problem is with the BLOCK'ing
semantics or implementation. There are two ways that I can
do this, IMHO, but I'm out to solicit alternatives as I'm
still getting a handle on what the best interfaces/APIs are
internally to get something done.

Sounds great :-)

Option 1) Use sleep(3) for the given timeout and wake up on
some interruptible a signal (USR2?). This is the simplest
solution, but likely the least portable to win32. Given the
new world order of 8.0 and it's portability headaches, it's
something I'm aware of.

There should be no problem doing this on win32. The portability layer
(in backend/port/win32) implements a sleep (pg_usleep, that is) that can
be interrupted by any signal (our win32 signal implementation actually
supports 256 signals, but you'd have to stick to a signal that's
supported on all platforms and also not already in use. And that's going
to be a lot harder unless you can piggyback on something that's already
there).

The async interface is nice, but not really useful to me as
it requires polling, instead of unblocking when an event
comes through, which would create a vastly more real time
interface that should be easier on the database.

As long as you're working only from the client side, you don't need to
do polling. See the bottom of the page at
http://developer.postgresql.org/docs/postgres/libpq-notify.html. You
just need to pass the socket down to select(). I'm doing this from C and
perl, don't know about other languages.

That said, there are certainly a lot of cases where a blocking call at
the SQL level would also be useful. Just don't kill what we have now :-)

//Magnus

Sean Chittenden <sean@chittenden.org> writes:

* Allow LISTEN to block until a value changes. LISTEN [BLOCK|WAIT]
'relname'

* Allow LISTEN to have a timeout

LISTEN name [BLOCK] [WITH] [TIMEOUT [=] ''::INTERVAL];

* Allow blocking LISTEN queries to update the status of the proc title
while blocking.

I don't believe in any of these things, at least not on the server side.
You can get the same effect on the client side without cluttering the
LISTEN semantics and implementation.

regards, tom lane

Sean Chittenden <sean@chittenden.org> writes:

The async interface is nice, but not really useful to me as it requires
polling, instead of unblocking when an event comes through, which would
create a vastly more real time interface that should be easier on the
database.

BTW, this is nonsense; the backend sends a message exactly when the
NOTIFY occurs. It may well be that libpq needs some work to make it
easier to wait for a NOTIFY without polling, but again there isn't a
reason to clutter the server-side code with it.

regards, tom lane

Sean Chittenden wrote:

Option 1) Use sleep(3) for the given timeout and wake up on some
interruptible a signal (USR2?). This is the simplest solution, but
likely the least portable to win32. Given the new world order of 8.0
and it's portability headaches, it's something I'm aware of.

Option 2) block on an exclusive lock. Check to see if relname has

been

registered. If it has, block on the existing exclusive lock (can I
block on a lock for a given number of sec/ms?). If it hasn't, create
an exclusive lock, but give the lock away (to the parent postmaster, a
lockmgr proc, etc) so that a notify can remove the remove and unlock
the exclusive lock so that all of the blocking children wake up.

The async interface is nice, but not really useful to me as it

requires

polling, instead of unblocking when an event comes through, which

would

create a vastly more real time interface that should be easier on the
database. Does anyone think there would be any conflicts with the use
of the existing alarm code from storage/lmgr/proc.c for the
LISTEN/NOTIFY interface? It looks like SIGALRM has a reserved

purpose.

I haven't found any global alarm handling interface that can be used
to assign different meanings when an SIGALRM is received. Any other
thoughts on the best way to proceed?

You can make cooperative blocking locks using some slick client side
code and the swiss army knife userlock module. Since user locks pierce
transaction encapsulation they can be used for these types of things.

select user_write_lock(some number);
if return = 1
notify some message
else
wait a and try again, etc.
end if

// release lock, etc.

NOTIFY 'relname' a_expr;

This would be great to have...at least I think this is what you are
driving at: (adding a noiseword for readability)

LISTEN system_messages;
NOTIFY system_messages MESSAGE logoff;
NOTIFY request_unlock MESSAGE 12345; <-- for use with user locks!

Etc.

Another cute tidbit about the listen/notify interface is that it can be
abused to create server side storage that is cleaned up by the server
when a backend drops. I was using this to crate a crude pessimistic
locking system before I discovered the userlock module. Note that this
is not a scalable or robust approach however.

listen 12345; <--acquired a lock on '12345'
select listenerpid from pg_listener where relname = '12345'

Merlin

NOTIFY 'relname' a_expr;

This would be great to have...at least I think this is what you are
driving at: (adding a noiseword for readability)

LISTEN system_messages;
NOTIFY system_messages MESSAGE logoff;
NOTIFY request_unlock MESSAGE 12345; <-- for use with user locks!

Hrm... the userlock module may work for what I'm after, but the module
is GPL'ed, which is something I don't want to rely on. I wanted to
have a backend process essentially act as a long running client that
would act on notify messages to dispatch work. I was hoping to
accomplish something like, "LISTEN REGISTER event_name; LISTEN * BLOCK
WITH TIMEOUT = 30;" which would only listen to events that you've
registered to listen to, but would only deliver messages when you
explicitly were going to block waiting for message delivery (ie, LISTEN
BLOCK). I'll settle for getting just message passing via NOTIFY/LISTEN
to work and will revisit extending things further once I have this bit
done. -sc

--
Sean Chittenden

The async interface is nice, but not really useful to me as it
requires
polling, instead of unblocking when an event comes through, which
would
create a vastly more real time interface that should be easier on the
database.

BTW, this is nonsense; the backend sends a message exactly when the
NOTIFY occurs. It may well be that libpq needs some work to make it
easier to wait for a NOTIFY without polling, but again there isn't a
reason to clutter the server-side code with it.

It's asynchronous with regards to client delivery of the message.
Sending of the NOTIFY message is synchronous, but delivery to the
client is not. I don't see how it could be any other way in
PostgreSQL. libpq(3) actually has a reasonable interface that relies
on the developer to block on the fd as described here:

http://developer.postgresql.org/docs/postgres/libpq-notify.html

The problem is that this doesn't work in pl/*, which is the problem I
was trying to address. *shrug* -sc

--
Sean Chittenden

In case anyone's wondering why I'm replying to an article that's nearly a
year old, it came up in a search while I was looking for the more recent
LISTEN/NOTIFY thread...

In article <7373.1104169267@sss.pgh.pa.us>,
Tom Lane <tgl@sss.pgh.pa.us> wrote:

Sean Chittenden <sean@chittenden.org> writes:

The async interface is nice, but not really useful to me as it requires
polling, instead of unblocking when an event comes through, which would
create a vastly more real time interface that should be easier on the
database.

BTW, this is nonsense; the backend sends a message exactly when the
NOTIFY occurs. It may well be that libpq needs some work to make it
easier to wait for a NOTIFY without polling, but again there isn't a
reason to clutter the server-side code with it.

One thing that persistently annoys me about using LISTEN in clients
is that much of the time I'm not using libpq directly, but some layer
above libpq that implements a generic SQL interface. These interfaces
generally assume that SQL calls are synchronous and blocking unless
you use some kind of escape to the lower-level driver, and in a few
environments there is no such escape at all.

I'd really like to have something that looks as much like an SQL statement
as possible, which blocks until a NOTIFY event or a timeout occurs. If not
a stand-alone SQL command then at least a function.

In one case where I really needed this, I implemented a really *ugly*
PL/PerlU function that did the following:

open a new client connection (with caching) to the server from
the server's own backend

set up appropriate LISTEN commands,

go to sleep on a poll() from the file descriptor,

then return immediately after poll() does.

Thankfully this particular application was not required to scale beyond
a half-dozen clients!
--
Zygo Blaxell (Laptop) <zblaxell@feedme.hungrycats.org>
GPG = B203 9402 B0E7 0F20 3B74 B13C DFBC C916 4395 FD03