Inter-app communication via DB
Hi list.
One pattern I've used is for apps to communicate events to each other
through the database.
ie:
- App 1 sents a boolean value to True
- App 2 queries the field every 10s, sets the value to False, and does
something.
Is this reasonable, or should apps avoid this pattern?
I have seen the NOTIFY and LISTEN SQL statements. However:
1) App 2 might not be running at the time (eg: it's launched from
cron, or it was temporarily stopped), and the expectation is that App
2 will run the special logic when it is started.
2) App 2 is usually single-threaded, and needs to do other things in
it's main thread besides wait for a DB notification.
I also know of RPC, but haven't used it before, and don't see a need
if you can use the above pattern.
I would use RPC (or unix signals if on the same host) if App 2 needed
to respond quickly, and I didn't want to hammer the DB & network with
constant polling.
Any comments?
David.
On Thu, Jun 19, 2008 at 11:09:12AM +0200, David wrote:
One pattern I've used is for apps to communicate events to each other
through the database.
Works nicely with LISTEN/NOTIFY. We use it a lot in GNUmed.
- App 1 sents a boolean value to True
- App 2 queries the field every 10s, sets the value to False, and does
something.Is this reasonable, or should apps avoid this pattern?
Add more app instances and you'll have a lot of polling.
I have seen the NOTIFY and LISTEN SQL statements. However:
1) App 2 might not be running at the time (eg: it's launched from
cron, or it was temporarily stopped), and the expectation is that App
2 will run the special logic when it is started.
That will happen anyway, no matter what the message
transport is like. Apps will have to read state at startup
anyway, no ?
2) App 2 is usually single-threaded, and needs to do other things in
it's main thread besides wait for a DB notification.
Well, threads in Python aren't *that* hard to get right. But
you could also write a small listener demon which gets
started on behalf of the local app instance which writes to
a local watch file which is being polled by the local app
instance. Takes the poll pressure off the database and
avoids having to thread the app, too.
There's a bunch of Python code here
that might be of interest.
Karsten
--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346
hello
look to orafce package
http://www.pgsql.cz/index.php/Oracle_functionality_%28en%29
regards
Pavel
2008/6/19 David <wizzardx@gmail.com>:
Show quoted text
Hi list.
One pattern I've used is for apps to communicate events to each other
through the database.ie:
- App 1 sents a boolean value to True
- App 2 queries the field every 10s, sets the value to False, and does
something.Is this reasonable, or should apps avoid this pattern?
I have seen the NOTIFY and LISTEN SQL statements. However:
1) App 2 might not be running at the time (eg: it's launched from
cron, or it was temporarily stopped), and the expectation is that App
2 will run the special logic when it is started.2) App 2 is usually single-threaded, and needs to do other things in
it's main thread besides wait for a DB notification.I also know of RPC, but haven't used it before, and don't see a need
if you can use the above pattern.I would use RPC (or unix signals if on the same host) if App 2 needed
to respond quickly, and I didn't want to hammer the DB & network with
constant polling.Any comments?
David.
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On Thu, Jun 19, 2008 at 11:25 AM, Karsten Hilbert
<Karsten.Hilbert@gmx.net> wrote:
On Thu, Jun 19, 2008 at 11:09:12AM +0200, David wrote:
[...]
One pattern I've used is for apps to communicate events to each other
through the database.Works nicely with LISTEN/NOTIFY. We use it a lot in GNUmed.
- App 1 sents a boolean value to True
- App 2 queries the field every 10s, sets the value to False, and does
something.Is this reasonable, or should apps avoid this pattern?
Add more app instances and you'll have a lot of polling.
Good point.
I have seen the NOTIFY and LISTEN SQL statements. However:
1) App 2 might not be running at the time (eg: it's launched from
cron, or it was temporarily stopped), and the expectation is that App
2 will run the special logic when it is started.That will happen anyway, no matter what the message
transport is like. Apps will have to read state at startup
anyway, no ?
Another good point.
I have a small problem with this. If app1 wants to tell app2 to
perform an expensive operation (which you don't want app2 to do each
time it starts up), in the original pattern it could just set a
boolean variable. Now it needs to both set a boolean variable (in case
app2 isn't running at the moment) and use NOTIFY (for when it is),
which seems a bit redundant.
2) App 2 is usually single-threaded, and needs to do other things in
it's main thread besides wait for a DB notification.Well, threads in Python aren't *that* hard to get right. But
you could also write a small listener demon which gets
started on behalf of the local app instance which writes to
a local watch file which is being polled by the local app
instance. Takes the poll pressure off the database and
avoids having to thread the app, too.
I don't mind threads in Python. It's mostly for C and C++ apps where I
don't want to add threading and an extra db connection to the code
just to be able to receive notifications from other apps.
For cases like that, your idea of a separate listener daemon will be
useful if there are a lot of instances that want to poll at the same
time :-)
David.
On Thu, Jun 19, 2008 at 11:46:42AM +0200, David wrote:
That will happen anyway, no matter what the message
transport is like. Apps will have to read state at startup
anyway, no ?I have a small problem with this. If app1 wants to tell app2 to
perform an expensive operation (which you don't want app2 to do each
time it starts up), in the original pattern it could just set a
boolean variable. Now it needs to both set a boolean variable (in case
app2 isn't running at the moment) and use NOTIFY (for when it is),
which seems a bit redundant.
You could add a trigger to the table with the boolean flag.
The trigger sends the NOTIFY all by itself. So when the app
is listening it gets the signal. When it isn't it will read
the flag at startup. All the other app does is set the flag.
Karsten
--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346
Le jeudi 19 juin 2008, David a écrit :
One pattern I've used is for apps to communicate events to each other
through the database.ie:
- App 1 sents a boolean value to True
- App 2 queries the field every 10s, sets the value to False, and does
something.Is this reasonable, or should apps avoid this pattern?
The PGQ package from skytools allows you to produce events into a queue from
the provider side and retrieve them at the consumer(s) side. The consumer(s)
needs not be running while the provider produces them, this will simply cause
lag.
The consuming of events has the options to set any event processing as failed
or retry.
PGQ comes with several APIs, the common one is its SQL one, atop of it reside
both a python one and (recently) a PHP one.
You might want to read those URLs to get more ideas about it, but I'd
definitely consider using PGQ in your case:
http://skytools.projects.postgresql.org/doc/pgq-sql.html
http://www.pgcon.org/2008/schedule/events/79.en.html
http://kaiv.wordpress.com/2007/10/19/skytools-database-scripting-framework-pgq/
Regards,
--
dim