Local visibility with logical decoding

Marko Tiikkaja <marko@joh.to> wrote:

It appears that when logical decoding sends out the data from the output
plugin, it is not guaranteed that the decoded transaction's effects are
visible on the source server. Is this the way it's supposed to work?

Can you please share the test that indicates this behavior? As far as I
understand, the transaction must have been committed before the output plugin
starts to receive the changes.

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Mon, Jul 20, 2020 at 7:36 PM Antonin Houska <ah@cybertec.at> wrote:

Marko Tiikkaja <marko@joh.to> wrote:

It appears that when logical decoding sends out the data from the output
plugin, it is not guaranteed that the decoded transaction's effects are
visible on the source server. Is this the way it's supposed to work?

Can you please share the test that indicates this behavior? As far as I
understand, the transaction must have been committed before the output
plugin
starts to receive the changes.

I don't have a reliable test program, but you can reproduce quite easily
with test_decoding if you put a breakpoint before the SyncRepWaitForLSN()
call in src/backend/access/transam/xact.c. pg_logicalrecv will see the
changes while the session is sitting on the breakpoint, and not finishing
its commit.

-marko

Hi,

On 2020-07-20 17:27:30 +0300, Marko Tiikkaja wrote:

It appears that when logical decoding sends out the data from the output
plugin, it is not guaranteed that the decoded transaction's effects are
visible on the source server. Is this the way it's supposed to work?

At the moment the visibility behaviour is basically the same as crash
recovery / standbys. And they just look at the WAL...

If so, would doing something like this in the output plugin be reasonable?

TransactionId xid = transaction->xid;
if (transaction->is_known_as_subxact)
xid = transaction->toplevel_xid;

if (TransactionIdIsInProgress(xid))
XactLockTableWait(xid, NULL, NULL, XLTW_None);

I'd not be surprised if this had a potential to cause deadlocks.

Greetings,

Andres Freund