Listen/Notify feedback
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis,
etc.I came across Postgresql Listen/Notify and was easily able to write
code to listen to messages. For the people who have been using this for a
while: what are its downsides, things to consider when writing good code
that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?
Thanks
--
--- Get your facts first, then you can distort them as you please.--
One aspect is if there is no one listening when a notify happens, the
message is lost (e.g. no durability). If this is important to you, it can
be addressed by writing the messages to a table as well when you NOTIFY,
and the listener deletes messages after they are processed. On connection
the listener can query the table to catch up on any missed messages, or
messages that were mid-process during a crash. This is trickier with more
than one listener. This isn't a whole lot more efficient than just using
the table alone, but it saves you from having to poll so better response
times.
On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
Show quoted text
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis,
etc.I came across Postgresql Listen/Notify and was easily able to write
code to listen to messages. For the people who have been using this for a
while: what are its downsides, things to consider when writing good code
that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?Thanks -- --- Get your facts first, then you can distort them as you please.--
Thats good to know. Are there some standard patterns or best practices I
should follow when using messaging and with listen/notify?
On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com> wrote:
One aspect is if there is no one listening when a notify happens, the
message is lost (e.g. no durability). If this is important to you, it can
be addressed by writing the messages to a table as well when you NOTIFY,
and the listener deletes messages after they are processed. On connection
the listener can query the table to catch up on any missed messages, or
messages that were mid-process during a crash. This is trickier with more
than one listener. This isn't a whole lot more efficient than just using
the table alone, but it saves you from having to poll so better response
times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis,
etc.I came across Postgresql Listen/Notify and was easily able to write
code to listen to messages. For the people who have been using this for a
while: what are its downsides, things to consider when writing good code
that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?Thanks -- --- Get your facts first, then you can distort them as you please.--
--
--- Get your facts first, then you can distort them as you please.--
On Sun, 12 Jul 2020 at 21:39, Rita <rmorgan466@gmail.com> wrote:
Thats good to know. Are there some standard patterns or best practices I
should follow when using messaging and with listen/notify?On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com> wrote:
One aspect is if there is no one listening when a notify happens, the
message is lost (e.g. no durability). If this is important to you, it can
be addressed by writing the messages to a table as well when you NOTIFY,
and the listener deletes messages after they are processed. On connection
the listener can query the table to catch up on any missed messages, or
messages that were mid-process during a crash. This is trickier with more
than one listener. This isn't a whole lot more efficient than just using
the table alone, but it saves you from having to poll so better response
times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
I am investigating various pub/sub tools such as ActiveMQ, Rabbit,
Redis, etc.I came across Postgresql Listen/Notify and was easily able to
write code to listen to messages. For the people who have been using this
for a while: what are its downsides, things to consider when writing good
code that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?Thanks -- --- Get your facts first, then you can distort them as you please.--
A couple of years ago I started looking into listen/notify in PG10 and
found that the throughput decreased quite a bit as I added more and more
listeners. Given the number of apps I needed to have listening and the
number of messages that I expected to be consuming, I ended up writing a
single listener app which then republished the messages via MQTT. Not sure
if the performance has improved in subsequent versions (or whether this
will affect you at all) but it's something to keep in mind.
Andrew Smith <laconical@gmail.com> writes:
A couple of years ago I started looking into listen/notify in PG10 and
found that the throughput decreased quite a bit as I added more and more
listeners. Given the number of apps I needed to have listening and the
number of messages that I expected to be consuming, I ended up writing a
single listener app which then republished the messages via MQTT. Not sure
if the performance has improved in subsequent versions (or whether this
will affect you at all) but it's something to keep in mind.
FWIW, we made some efficiency improvements in v13 for the case of NOTIFY
with a lot of listeners [1]https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=51004c717. Can't say of course whether that would
have fixed your problem, and v13 is still in beta anyway.
regards, tom lane
[1]: https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=51004c717
Good to know about potential performance problems. I don't plan to have
more than 5 hosts. Also, good to know about MQTT.
On Sun, Jul 12, 2020 at 8:52 AM Andrew Smith <laconical@gmail.com> wrote:
On Sun, 12 Jul 2020 at 21:39, Rita <rmorgan466@gmail.com> wrote:
Thats good to know. Are there some standard patterns or best practices I
should follow when using messaging and with listen/notify?On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com>
wrote:One aspect is if there is no one listening when a notify happens, the
message is lost (e.g. no durability). If this is important to you, it can
be addressed by writing the messages to a table as well when you NOTIFY,
and the listener deletes messages after they are processed. On connection
the listener can query the table to catch up on any missed messages, or
messages that were mid-process during a crash. This is trickier with more
than one listener. This isn't a whole lot more efficient than just using
the table alone, but it saves you from having to poll so better response
times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
I am investigating various pub/sub tools such as ActiveMQ, Rabbit,
Redis, etc.I came across Postgresql Listen/Notify and was easily able to
write code to listen to messages. For the people who have been using this
for a while: what are its downsides, things to consider when writing good
code that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?Thanks -- --- Get your facts first, then you can distort them as you please.--A couple of years ago I started looking into listen/notify in PG10 and
found that the throughput decreased quite a bit as I added more and more
listeners. Given the number of apps I needed to have listening and the
number of messages that I expected to be consuming, I ended up writing a
single listener app which then republished the messages via MQTT. Not sure
if the performance has improved in subsequent versions (or whether this
will affect you at all) but it's something to keep in mind.
--
--- Get your facts first, then you can distort them as you please.--
On Sat, Jul 11, 2020 at 10:44 AM Brian Dunavant <dunavant@gmail.com> wrote:
One aspect is if there is no one listening when a notify happens, the
message is lost (e.g. no durability). If this is important to you, it can
be addressed by writing the messages to a table as well when you NOTIFY,
and the listener deletes messages after they are processed. On connection
the listener can query the table to catch up on any missed messages, or
messages that were mid-process during a crash. This is trickier with more
than one listener. This isn't a whole lot more efficient than just using
the table alone, but it saves you from having to poll so better response
times.
Good advice from Brian here that mirrors my own experience, I'd like to
point out that if you do go the multiple listener route working a
persistent table, it's important to avoid races with SELECT FOR UPDATE SKIP
LOCKED.
https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE
I know this is old news to most of the people on this list, but I've run
into enough folks who don't know about this little gem that I figured I'd
mention it, it's saved my bacon more than once.
-Michel
Show quoted text
On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis,
etc.I came across Postgresql Listen/Notify and was easily able to write
code to listen to messages. For the people who have been using this for a
while: what are its downsides, things to consider when writing good code
that use pub/sub, how do you deal with large messages, can I have
subscribers listen to replica nodes?Thanks -- --- Get your facts first, then you can distort them as you please.--