pgoutput: schema cache cleanup after streamed 2PC
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253820psql -h localhost -U postgresBuilt from patchset v5 (message #5), September 17, 2026 at 01:55 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253820_5 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253820_5 && git checkout t253820_5Patchset v5 (message #5) is on t253820_5
Hi,
I think we're missing some cleanup in pgoutput for streamed 2PC.
We call cleanup_rel_sync_cache() on stream commit and abort, but not
on COMMIT PREPARED or ROLLBACK PREPARED.
For streamed transactions, maybe_send_schema() records the top-level XID
in each relation's streamed_txns list when it sends the schema. It looks
like those entries are left behind after COMMIT PREPARED or ROLLBACK
PREPARED, until the cache entry is rebuilt or the walsender exits.
The attached patch adds the same cleanup to those two callbacks, guarded
by rbtxn_is_streamed(txn). Is there a reason to keep these entries once
the prepared transaction has finished?
I tested both commit and rollback with a publisher/subscriber setup.
The data matched, and I no longer saw the cache growth with the patch.
Thoughts?
Regards,
Ayush
Hi Ayush,
The attached patch adds the same cleanup to those two callbacks, guarded
by rbtxn_is_streamed(txn). Is there a reason to keep these entries once
the prepared transaction has finished?
Sorry if it's a stupid question. Why can't we put in pgoutput_stream_prepare_txn()?
It may be able to avoid checking by rbtxn_is_streamed(txn).
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Hi,
On Thu, 17 Sept 2026 at 17:05, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Hi Ayush,
The attached patch adds the same cleanup to those two callbacks, guarded
by rbtxn_is_streamed(txn). Is there a reason to keep these entries once
the prepared transaction has finished?Sorry if it's a stupid question. Why can't we put in pgoutput_stream_prepare_txn()?
It may be able to avoid checking by rbtxn_is_streamed(txn).
Thanks for checking this, Kuroda-san!
You are right, it makes sense to put it in pgoutput_stream_prepare_txn().
I've moved cleanup_rel_sync_cache(txn->xid, true) to
pgoutput_stream_prepare_txn() in v2 and updated the comments. That also
removes the extra checks, as you suggested.
Regards,
Ayush
Dear Ayush,
You are right, it makes sense to put it in pgoutput_stream_prepare_txn().
Thanks for the quick update. Let me dump my thought in [1]IIUC pgoutput_commit_prepared_txn() and pgoutput_rollback_prepared_txn() are used for both streamed and non-streamed cases. So putting the cleanup for streamed transactions should be in pgoutput_stream_prepare_txn() as much as possible. just in case.
Some tests may be needed (no need to include in core though).
I've moved cleanup_rel_sync_cache(txn->xid, true) to
pgoutput_stream_prepare_txn() in v2 and updated the comments.
I think the second argument should be renamed. Do you have anything in your mind?
My idea: mark_schema_sent.
[1]: IIUC pgoutput_commit_prepared_txn() and pgoutput_rollback_prepared_txn() are used for both streamed and non-streamed cases. So putting the cleanup for streamed transactions should be in pgoutput_stream_prepare_txn() as much as possible.
IIUC pgoutput_commit_prepared_txn() and pgoutput_rollback_prepared_txn() are used
for both streamed and non-streamed cases. So putting the cleanup for streamed
transactions should be in pgoutput_stream_prepare_txn() as much as possible.
The main question here is whether we pass true or false for is_commit. I think
true can be used, because no need to re-send RELATION messages once it's handled
by the subscriber side.
For streaming = on case, an apply worker firstly serialize streamed changes, then
it applies them when STREAM COMMIT or STREAM PREPARE are received. It means
RELATION messages have already handled in PREPARE phase.
For streaming = parallel case, both leader and parallel apply worker handle
RELATION messages immediately.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Hi,
On Thu, 17 Sept 2026 at 18:22, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Thanks for the quick update. Let me dump my thought in [1] just in case.
Some tests may be needed (no need to include in core though).
I used a separate publisher/subscriber test covering both commit and
rollback. It confirmed that the cache usage stays flat with the cleanup,
I too dont think this explicitly warrants a core test.
I've moved cleanup_rel_sync_cache(txn->xid, true) to
pgoutput_stream_prepare_txn() in v2 and updated the comments.I think the second argument should be renamed. Do you have anything in your mind?
My idea: mark_schema_sent.
Thanks for the suggestion. I initially thought of set_schema_sent, but
mark_schema_sent sounds better. I've used that in v3 and updated the nearby
comments.
[1]:
IIUC pgoutput_commit_prepared_txn() and pgoutput_rollback_prepared_txn() are used
for both streamed and non-streamed cases. So putting the cleanup for streamed
transactions should be in pgoutput_stream_prepare_txn() as much as possible.The main question here is whether we pass true or false for is_commit. I think
true can be used, because no need to re-send RELATION messages once it's handled
by the subscriber side.For streaming = on case, an apply worker firstly serialize streamed changes, then
it applies them when STREAM COMMIT or STREAM PREPARE are received. It means
RELATION messages have already handled in PREPARE phase.For streaming = parallel case, both leader and parallel apply worker handle
RELATION messages immediately.
Thanks for the analysis.
Attached v3 with changes.
Regards,
Ayush