Assertion failure in update_local_synced_slot(), slotsync.c:364
Hi all,
While reviewing slotsync.c, I hit the assertion failure:
TRAP: failed Assert("MyReplicationSlot->data.persistency ==
RS_TEMPORARY"), slotsync.c:364
IIUC this is a regression of commit 788ec96d591 (CC'ed Hou and Amit).
In update_local_synced_slot(), we try to advance the local slot only
when the remote slot is ahead of it, and if the standby cannot reach a
consistent snapshot we set the skip reason to
SS_SKIP_NO_CONSISTENT_SNAPSHOT so that
update_and_persist_local_synced_slot() doesn't persist the slot.
However, LogicalSlotAdvanceAndCheckSnapState() has already advanced
confirmed_flush_lsn to the remote's value in that case (restart_lsn
and catalog_xmin were already equal to the remote's when the slot was
created). So in the next cycle the remote slot is no longer ahead, we
skip the advance entirely, and update_slotsync_skip_stats() overwrites
the skip reason with SS_SKIP_NONE.
update_and_persist_local_synced_slot() then persists the slot and
reports it as sync-ready even though nothing has changed since the
previous cycle. Once the remote slot advances again and the standby
still cannot find a consistent snapshot, the assertion above fails.
On non-assert builds, IIUC the problem is worse: the persisted slot
can be used after promotion, and decoding from it skips the
transactions between its restart_lsn and the first consistent point,
causing data loss.
Before that commit, update_and_persist_local_synced_slot() also
checked found_consistent_snapshot returned by
update_local_synced_slot() for the current cycle, so a cycle that
didn't attempt to advance the slot never persisted it.
I've attached a TAP test to reproduce this issue, generated by AI.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Hi,
On Tue, Nov10, 2026 at 04:04 AM Masahiko Sawada wrote:
Hi all,
While reviewing slotsync.c, I hit the assertion failure:
TRAP: failed Assert("MyReplicationSlot->data.persistency ==
RS_TEMPORARY"), slotsync.c:364IIUC this is a regression of commit 788ec96d591 (CC'ed Hou and Amit).
In update_local_synced_slot(), we try to advance the local slot only
when the remote slot is ahead of it, and if the standby cannot reach a
consistent snapshot we set the skip reason to
SS_SKIP_NO_CONSISTENT_SNAPSHOT so that
update_and_persist_local_synced_slot() doesn't persist the slot.
However, LogicalSlotAdvanceAndCheckSnapState() has already advanced
confirmed_flush_lsn to the remote's value in that case (restart_lsn
and catalog_xmin were already equal to the remote's when the slot was
created). So in the next cycle the remote slot is no longer ahead, we
skip the advance entirely, and update_slotsync_skip_stats() overwrites
the skip reason with SS_SKIP_NONE.
update_and_persist_local_synced_slot() then persists the slot and
reports it as sync-ready even though nothing has changed since the
previous cycle. Once the remote slot advances again and the standby
still cannot find a consistent snapshot, the assertion above fails.On non-assert builds, IIUC the problem is worse: the persisted slot
can be used after promotion, and decoding from it skips the
transactions between its restart_lsn and the first consistent point,
causing data loss.Before that commit, update_and_persist_local_synced_slot() also
checked found_consistent_snapshot returned by
update_local_synced_slot() for the current cycle, so a cycle that
didn't attempt to advance the slot never persisted it.I've attached a TAP test to reproduce this issue, generated by AI.
Thanks for reporting the issue.
I agree with your analysis, and I think we can fix it by preserving
SS_SKIP_NO_CONSISTENT_SNAPSHOT in update_local_synced_slot() when the
remote slot has not advanced since a previous cycle failed to build a
consistent snapshot.
Attached is a patch for the fix. I also tried to simplify and stabilize
the test and merged it into 040_standby_failover_slots_sync.pl in case
we want to include a test.
--
Best Regards,
Zhijie Hou
Hi,
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
Hi,
On Tue, Nov10, 2026 at 04:04 AM Masahiko Sawada wrote:
Thanks for reporting the issue.
+1
I agree with your analysis, and I think we can fix it by preserving
SS_SKIP_NO_CONSISTENT_SNAPSHOT in update_local_synced_slot() when the remote
slot has not advanced since a previous cycle failed to build a consistent
snapshot.Attached is a patch for the fix. I also tried to simplify and stabilize the
test and merged it into 040_standby_failover_slots_sync.pl in case we want
to include a test.
Thanks for the patch!
+ else if (slot->slotsync_skip_reason == SS_SKIP_NO_CONSISTENT_SNAPSHOT)
Could slotsync_skip_reason be SS_SKIP_WAL_OR_ROWS_REMOVED here even if the slot
is still inconsistent? This could happen if the local catalog_xmin gets ahead
and the remote one later catches up?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Wed, Nov11, 2026 at 01:04 AM Bertrand Drouvot wrote:
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
I agree with your analysis, and I think we can fix it by preserving
SS_SKIP_NO_CONSISTENT_SNAPSHOT in update_local_synced_slot() when
the remote slot has not advanced since a previous cycle failed to
build a consistent snapshot.Attached is a patch for the fix. I also tried to simplify and
stabilize the test and merged it into
040_standby_failover_slots_sync.pl in case we want to include a
test.Thanks for the patch!
+ else if (slot->slotsync_skip_reason == SS_SKIP_NO_CONSISTENT_SNAPSHOT)
Could slotsync_skip_reason be SS_SKIP_WAL_OR_ROWS_REMOVED here even if
the slot is still inconsistent? This could happen if the local
catalog_xmin gets ahead and the remote one later catches up?
Thanks for catching this. I confirmed that this can happen.
Since any temporary slot that reaches a consistent snapshot is persisted
immediately in the same cycle, we can check "else if
(slot->data.persistency == RS_TEMPORARY)" here instead. Attached is the
v2 patch with this change.
--
Best Regards,
Zhijie Hou
Hi,
On Wed, Nov 11, 2026 at 11:13:13PM +0800, Zhijie Hou wrote:
Hi,
On Wed, Nov11, 2026 at 01:04 AM Bertrand Drouvot wrote:
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
I agree with your analysis, and I think we can fix it by preserving
SS_SKIP_NO_CONSISTENT_SNAPSHOT in update_local_synced_slot() when
the remote slot has not advanced since a previous cycle failed to
build a consistent snapshot.Attached is a patch for the fix. I also tried to simplify and
stabilize the test and merged it into
040_standby_failover_slots_sync.pl in case we want to include a
test.Thanks for the patch!
+ else if (slot->slotsync_skip_reason == SS_SKIP_NO_CONSISTENT_SNAPSHOT)
Could slotsync_skip_reason be SS_SKIP_WAL_OR_ROWS_REMOVED here even if
the slot is still inconsistent? This could happen if the local
catalog_xmin gets ahead and the remote one later catches up?Thanks for catching this. I confirmed that this can happen.
Thanks for confirming!
Since any temporary slot that reaches a consistent snapshot is persisted
immediately in the same cycle, we can check "else if (slot->data.persistency
== RS_TEMPORARY)" here instead. Attached is the v2 patch with this change.
Thanks for the patch!
Makes sense and LGTM. FWIW, v2 does not test the rare state transition we just
talked about but I don't think it's worth the extra complexity.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
Thank you for the patch. I tried v2 and it looks good to me.
What I did:
- Applied v2 to current master (ef1b740441c). It applies cleanly (git apply
--check) and builds without warnings (--enable-cassert --enable-tap-tests).
- t/
https://www.google.com/url?q=http://040_standby_failover_slots_sync.pl&source=gmail&ust=1794597551024000&sa=E
passes with the patch (39 tests), and I ran it 5 more times with the same
result.
- make check passes (243 tests). The whole src/test/recovery TAP suite
passes with the patch (59 files, 691 tests; 7 files that need injection
points were skipped, as my build has none).
- I applied only the t/040 hunk to unpatched master. The new test fails
there: the poll for "slotsync_skip_count >= 2 AND temporary AND
slotsync_skip_reason = 'no_consistent_snapshot'" times out and returns
false (test 39 of 39). So the new test does catch the problem. With the
full patch it passes. As expected, the failure comes as a timeout and not
as a fast failure.
The new "else if (slot->data.persistency == RS_TEMPORARY)" branch looks
right to me.
I have not tested the back branches, a meson build,
pg_sync_replication_slots() with a stopped primary slot, or the original
assertion failure scenario by hand, so I can not say anything about those.
Thanks again,
Joao Rito
On Wed, Nov 11, 2026 12:13 PM, Zhijie Hou <houzhijie22@gmail.com> wrote:
Show quoted text
Hi,
On Wed, Nov11, 2026 at 01:04 AM Bertrand Drouvot wrote:
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
I agree with your analysis, and I think we can fix it by preserving
SS_SKIP_NO_CONSISTENT_SNAPSHOT in update_local_synced_slot() when
the remote slot has not advanced since a previous cycle failed to
build a consistent snapshot.Attached is a patch for the fix. I also tried to simplify and
stabilize the test and merged it into
040_standby_failover_slots_sync.pl in case we want to include a
test.Thanks for the patch!
+ else if (slot->slotsync_skip_reason ==
SS_SKIP_NO_CONSISTENT_SNAPSHOT)
Could slotsync_skip_reason be SS_SKIP_WAL_OR_ROWS_REMOVED here even if
the slot is still inconsistent? This could happen if the local
catalog_xmin gets ahead and the remote one later catches up?Thanks for catching this. I confirmed that this can happen.
Since any temporary slot that reaches a consistent snapshot is persisted
immediately in the same cycle, we can check "else if
(slot->data.persistency == RS_TEMPORARY)" here instead. Attached is the
v2 patch with this change.--
Best Regards,
Zhijie Hou
On Wed, Nov 11, 2026 at 8:43 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
On Wed, Nov11, 2026 at 01:04 AM Bertrand Drouvot wrote:
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
Thanks for catching this. I confirmed that this can happen.
Since any temporary slot that reaches a consistent snapshot is persisted
immediately in the same cycle, we can check "else if
(slot->data.persistency == RS_TEMPORARY)" here instead. Attached is the
v2 patch with this change.
+ else if (slot->data.persistency == RS_TEMPORARY)
+ {
+ /*
+ * We reach this point when a previous cycle updated confirmed_flush
+ * but failed to build a consistent snapshot, and the remote slot has
+ * not advanced beyond the local slot since then.
+ */
+ skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT;
+ }
Is it better to directly use the slot's skip_reason rather than
assuming SS_SKIP_NO_CONSISTENT_SNAPSHOT? I understand today it is
correct but tomorrow if there are more reasons, it would be better to
rely on the slot's value.
--
With Regards,
Amit Kapila.
Hi,
On Fri, Nov 13, 2026 at 04:45:00PM +0530, Amit Kapila wrote:
On Wed, Nov 11, 2026 at 8:43 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
On Wed, Nov11, 2026 at 01:04 AM Bertrand Drouvot wrote:
On Tue, Nov 10, 2026 at 05:17:57PM +0800, Zhijie Hou wrote:
Thanks for catching this. I confirmed that this can happen.
Since any temporary slot that reaches a consistent snapshot is persisted
immediately in the same cycle, we can check "else if
(slot->data.persistency == RS_TEMPORARY)" here instead. Attached is the
v2 patch with this change.+ else if (slot->data.persistency == RS_TEMPORARY) + { + /* + * We reach this point when a previous cycle updated confirmed_flush + * but failed to build a consistent snapshot, and the remote slot has + * not advanced beyond the local slot since then. + */ + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT; + }Is it better to directly use the slot's skip_reason rather than
assuming SS_SKIP_NO_CONSISTENT_SNAPSHOT?
That could work but what about the case I mentioned upthread where the local
catalog_xmin gets ahead and the remote later catches up? I think that that would
keep slotsync_skip_reason as SS_SKIP_WAL_OR_ROWS_REMOVED, although the slot is
still inconsistent due to the missing snapshot.
I understand today it is correct but tomorrow if there are more reasons
Maybe we could keep v2 and assert the two possible previous reasons, so any new
reason would need explicit handling?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Fri, Nov 13, 2026 at 7:29 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Fri, Nov 13, 2026 at 04:45:00PM +0530, Amit Kapila wrote:
+ else if (slot->data.persistency == RS_TEMPORARY) + { + /* + * We reach this point when a previous cycle updated confirmed_flush + * but failed to build a consistent snapshot, and the remote slot has + * not advanced beyond the local slot since then. + */ + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT; + }Is it better to directly use the slot's skip_reason rather than
assuming SS_SKIP_NO_CONSISTENT_SNAPSHOT?That could work but what about the case I mentioned upthread where the local
catalog_xmin gets ahead and the remote later catches up? I think that that would
keep slotsync_skip_reason as SS_SKIP_WAL_OR_ROWS_REMOVED, although the slot is
still inconsistent due to the missing snapshot.
Right the reason can remain as SS_SKIP_WAL_OR_ROWS_REMOVED. Even in
the first cycle, the reason could be either SS_SKIP_WAL_NOT_FLUSHED or
SS_SKIP_WAL_OR_ROWS_REMOVED and in that case also the caller
update_and_persist_local_synced_slot() won't persist the slot. So, I
don't see a problem with retaining the previous reason when we didn't
perform anything in a particular sync cycle.
I understand today it is correct but tomorrow if there are more reasons
Maybe we could keep v2 and assert the two possible previous reasons, so any new
reason would need explicit handling?
Hmm, yeah that kind of forceful handling can also work but when we
don't see any harm in retaining the previous skip reason then why not
do that instead?
--
With Regards,
Amit Kapila.
On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Nov 13, 2026 at 7:29 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Fri, Nov 13, 2026 at 04:45:00PM +0530, Amit Kapila wrote:
+ else if (slot->data.persistency == RS_TEMPORARY) + { + /* + * We reach this point when a previous cycle updated confirmed_flush + * but failed to build a consistent snapshot, and the remote slot has + * not advanced beyond the local slot since then. + */ + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT; + }Is it better to directly use the slot's skip_reason rather than
assuming SS_SKIP_NO_CONSISTENT_SNAPSHOT?That could work but what about the case I mentioned upthread where the local
catalog_xmin gets ahead and the remote later catches up? I think that that would
keep slotsync_skip_reason as SS_SKIP_WAL_OR_ROWS_REMOVED, although the slot is
still inconsistent due to the missing snapshot.Right the reason can remain as SS_SKIP_WAL_OR_ROWS_REMOVED. Even in
the first cycle, the reason could be either SS_SKIP_WAL_NOT_FLUSHED or
SS_SKIP_WAL_OR_ROWS_REMOVED and in that case also the caller
update_and_persist_local_synced_slot() won't persist the slot. So, I
don't see a problem with retaining the previous reason when we didn't
perform anything in a particular sync cycle.
While I agree that retaining the previous reason doesn't make us
persist the slot, I think it's safe only because the previous reason
is never SS_SKIP_NONE for a temporary slot at that point. That's true
today, but it depends on several things and it would be hard to modify
these code correctly in the future.
I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of slotsync_skip_reason, but
they're not equivalent: found_consistent_snapshot is false unless the
current cycle synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle detected
a reason to skip. So an alternative idea would be to bring back
found_consistent_snapshot part from the commit. That way, the decision
of persisting the slot doesn't depend on the skip_reasons at all.
As for the reported reason, a temporary slot is the one we have not
confirmed to reach a consistent point yet. So I think we can simply
report no_consistent_snapshot for a temporary slot if the current
cycle didn't find a consistent snapshot, without relying on what
happened in the previous cycles.
I've attached a PoC patch.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
t258370_10found_consistent_snapshot.patchtext/x-patch; charset=US-ASCII; name=found_consistent_snapshot.patchDownload+91-10
Hi,
On Sat, Nov 14, 2026 at 09:04:57AM +0530, Amit Kapila wrote:
On Fri, Nov 13, 2026 at 7:29 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Fri, Nov 13, 2026 at 04:45:00PM +0530, Amit Kapila wrote:
+ else if (slot->data.persistency == RS_TEMPORARY) + { + /* + * We reach this point when a previous cycle updated confirmed_flush + * but failed to build a consistent snapshot, and the remote slot has + * not advanced beyond the local slot since then. + */ + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT; + }Is it better to directly use the slot's skip_reason rather than
assuming SS_SKIP_NO_CONSISTENT_SNAPSHOT?That could work but what about the case I mentioned upthread where the local
catalog_xmin gets ahead and the remote later catches up? I think that that would
keep slotsync_skip_reason as SS_SKIP_WAL_OR_ROWS_REMOVED, although the slot is
still inconsistent due to the missing snapshot.Right the reason can remain as SS_SKIP_WAL_OR_ROWS_REMOVED. Even in
the first cycle, the reason could be either SS_SKIP_WAL_NOT_FLUSHED or
SS_SKIP_WAL_OR_ROWS_REMOVED and in that case also the caller
Oh right, I missed SS_SKIP_WAL_NOT_FLUSHED.
update_and_persist_local_synced_slot() won't persist the slot. So, I
don't see a problem with retaining the previous reason when we didn't
perform anything in a particular sync cycle.
Right, but once the WAL has been flushed or the remote slot has caught up,
retaining SS_SKIP_WAL_NOT_FLUSHED or SS_SKIP_WAL_OR_ROWS_REMOVED would make
slotsync_skip_reason report a misleading reason: the slot remains temporary
because it still has no consistent snapshot.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Fri, Nov 13, 2026 at 10:18:34PM -0800, Masahiko Sawada wrote:
I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of slotsync_skip_reason, but
they're not equivalent: found_consistent_snapshot is false unless the
current cycle synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle detected
a reason to skip. So an alternative idea would be to bring back
found_consistent_snapshot part from the commit. That way, the decision
of persisting the slot doesn't depend on the skip_reasons at all.
Yeah, that makes sense to me.
I've attached a PoC patch.
Thanks! Will look at it early next week.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Sat, Nov14, 2026 at 02:18 PM Masahiko Sawada wrote:
On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Right the reason can remain as SS_SKIP_WAL_OR_ROWS_REMOVED. Even in
the first cycle, the reason could be either SS_SKIP_WAL_NOT_FLUSHED
or SS_SKIP_WAL_OR_ROWS_REMOVED and in that case also the caller
update_and_persist_local_synced_slot() won't persist the slot. So, I
don't see a problem with retaining the previous reason when we
didn't perform anything in a particular sync cycle.While I agree that retaining the previous reason doesn't make us
persist the slot, I think it's safe only because the previous reason
is never SS_SKIP_NONE for a temporary slot at that point. That's true
today, but it depends on several things and it would be hard to modify
these code correctly in the future.I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of slotsync_skip_reason, but
they're not equivalent: found_consistent_snapshot is false unless the
current cycle synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle detected
a reason to skip. So an alternative idea would be to bring back
found_consistent_snapshot part from the commit. That way, the decision
of persisting the slot doesn't depend on the skip_reasons at all.As for the reported reason, a temporary slot is the one we have not
confirmed to reach a consistent point yet. So I think we can simply
report no_consistent_snapshot for a temporary slot if the current
cycle didn't find a consistent snapshot, without relying on what
happened in the previous cycles.
I think it would be unnecessary to reintroduce the
found_consistent_snapshot out-parameter. Because by definition,
slotsync_skip_reason is SS_SKIP_NONE (NULL in pg_replication_slots) only
when slot synchronization succeeds. Even before commit 788ec96d591
(since commit e68b6adad9), resetting slotsync_skip_reason to
SS_SKIP_NONE on a no-op cycle for a temporary slot was already a bug,
because synchronization of that temporary slot had not succeeded yet.
Once update_local_synced_slot() sets SS_SKIP_NO_CONSISTENT_SNAPSHOT
whenever "slot->data.persistency == RS_TEMPORARY && !found_consistent"
(as in your patch and v2), slotsync_skip_reason is guaranteed not to be
SS_SKIP_NONE whenever a temporary slot cannot be persisted. So I think
fixing the update_local_synced_slot to set
SS_SKIP_NO_CONSISTENT_SNAPSHOT when necessary is sufficient and avoids
maintaining two separate states for the same condition.
--
Best Regards,
Zhijie Hou
On Mon, Nov 16, 2026 at 9:50 AM Zhijie Hou <houzhijie22@gmail.com> wrote:
On Sat, Nov14, 2026 at 02:18 PM Masahiko Sawada wrote:
On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
While I agree that retaining the previous reason doesn't make us
persist the slot, I think it's safe only because the previous reason
is never SS_SKIP_NONE for a temporary slot at that point. That's true
today, but it depends on several things and it would be hard to modify
these code correctly in the future.I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of slotsync_skip_reason, but
they're not equivalent: found_consistent_snapshot is false unless the
current cycle synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle detected
a reason to skip. So an alternative idea would be to bring back
found_consistent_snapshot part from the commit. That way, the decision
of persisting the slot doesn't depend on the skip_reasons at all.As for the reported reason, a temporary slot is the one we have not
confirmed to reach a consistent point yet. So I think we can simply
report no_consistent_snapshot for a temporary slot if the current
cycle didn't find a consistent snapshot, without relying on what
happened in the previous cycles.I think it would be unnecessary to reintroduce the
found_consistent_snapshot out-parameter. Because by definition,
slotsync_skip_reason is SS_SKIP_NONE (NULL in pg_replication_slots) only
when slot synchronization succeeds. Even before commit 788ec96d591
(since commit e68b6adad9), resetting slotsync_skip_reason to
SS_SKIP_NONE on a no-op cycle for a temporary slot was already a bug,
because synchronization of that temporary slot had not succeeded yet.
I agree that the bug is because we didn't maintain/update
slotsync_skip_reason correctly and we anyway need to maintain that
state to correctly display it in pg_replication_slots. Also, the slot
already carries the information about consistent_snapshot.
SS_SKIP_NO_CONSISTENT_SNAPSHOT says the slot hasn't reached a
consistent point, and it is visible in pg_replication_slots. A flag
passed back to the caller to say the same thing would be a duplicate
of that state.
--
With Regards,
Amit Kapila.
On Sat, Nov 14, 2026 at 12:11 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Sat, Nov 14, 2026 at 09:04:57AM +0530, Amit Kapila wrote:
On Fri, Nov 13, 2026 at 7:29 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:Right the reason can remain as SS_SKIP_WAL_OR_ROWS_REMOVED. Even in
the first cycle, the reason could be either SS_SKIP_WAL_NOT_FLUSHED or
SS_SKIP_WAL_OR_ROWS_REMOVED and in that case also the callerOh right, I missed SS_SKIP_WAL_NOT_FLUSHED.
update_and_persist_local_synced_slot() won't persist the slot. So, I
don't see a problem with retaining the previous reason when we didn't
perform anything in a particular sync cycle.Right, but once the WAL has been flushed or the remote slot has caught up,
retaining SS_SKIP_WAL_NOT_FLUSHED or SS_SKIP_WAL_OR_ROWS_REMOVED would make
slotsync_skip_reason report a misleading reason: the slot remains temporary
because it still has no consistent snapshot.
Fair enough, so we can update the skip_reason as the latest patch from
Hou-San does but we should add a DEBUG1 or LOG level message like we
do in other cases in the same function. How about something on the
following line:
ereport(DEBUG1,
errmsg("could not synchronize
replication slot \"%s\"",
remote_slot->name),
errdetail("The standby has not yet
reached a consistent "
"point when decoding
WAL from LSN %X/%08X, and "
"the remote slot has
not advanced since the "
"last attempt.",
LSN_FORMAT_ARGS(slot->data.restart_lsn)));
--
With Regards,
Amit Kapila.
On Sun, Nov 15, 2026 at 8:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Nov 16, 2026 at 9:50 AM Zhijie Hou <houzhijie22@gmail.com> wrote:
On Sat, Nov14, 2026 at 02:18 PM Masahiko Sawada wrote:
On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
While I agree that retaining the previous reason doesn't make us
persist the slot, I think it's safe only because the previous reason
is never SS_SKIP_NONE for a temporary slot at that point. That's true
today, but it depends on several things and it would be hard to modify
these code correctly in the future.I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of slotsync_skip_reason, but
they're not equivalent: found_consistent_snapshot is false unless the
current cycle synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle detected
a reason to skip. So an alternative idea would be to bring back
found_consistent_snapshot part from the commit. That way, the decision
of persisting the slot doesn't depend on the skip_reasons at all.As for the reported reason, a temporary slot is the one we have not
confirmed to reach a consistent point yet. So I think we can simply
report no_consistent_snapshot for a temporary slot if the current
cycle didn't find a consistent snapshot, without relying on what
happened in the previous cycles.I think it would be unnecessary to reintroduce the
found_consistent_snapshot out-parameter. Because by definition,
slotsync_skip_reason is SS_SKIP_NONE (NULL in pg_replication_slots) only
when slot synchronization succeeds. Even before commit 788ec96d591
(since commit e68b6adad9), resetting slotsync_skip_reason to
SS_SKIP_NONE on a no-op cycle for a temporary slot was already a bug,
because synchronization of that temporary slot had not succeeded yet.I agree that the bug is because we didn't maintain/update
slotsync_skip_reason correctly and we anyway need to maintain that
state to correctly display it in pg_replication_slots. Also, the slot
already carries the information about consistent_snapshot.
SS_SKIP_NO_CONSISTENT_SNAPSHOT says the slot hasn't reached a
consistent point, and it is visible in pg_replication_slots. A flag
passed back to the caller to say the same thing would be a duplicate
of that state.
I agree that the v2 patch fixes the problem correctly. My concern is
rather about readability and maintainability, which is why I proposed
the PoC patch.
+ else if (slot->data.persistency == RS_TEMPORARY)
+ {
+ /*
+ * We reach this point when a previous cycle updated confirmed_flush
+ * but failed to build a consistent snapshot, and the remote slot has
+ * not advanced beyond the local slot since then.
+ */
+ skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT;
+ }
Some my concerns are:
- When a temporary slot finds a consistent snapshot, it should be
persisted before update_local_synced_slot() is called for the slot
next time. If update_local_synced_slot() is called twice for a
temporary slot without the remote slot advancing, the two calls give
different results. The first call syncs the slot, finds a consistent
snapshot and leaves SS_SKIP_NONE, but the second call enters the new
branch and sets SS_SKIP_NO_CONSISTENT_SNAPSHOT, although nothing has
changed since the first call. This doesn't happen today only because
the caller persists the slot right after the first call.
- The comment explains the branch by what happened in a previous
cycle. So one needs to go through the possible history of the slot to
see why it's correct, whereas what the code checks is just that the
temporary slot hasn't found a consistent snapshot in this call.
- We set SS_SKIP_NO_CONSISTENT_SNAPSHOT in two places for what is
actually the same condition, which makes the code harder to follow.
I'm concerned that not only changes to this function but also changes
to its callers could break these assumptions, and nothing near those
places would tell us about them.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Hi,
On Sun, Nov 15, 2026 at 10:35:57PM -0800, Masahiko Sawada wrote:
I'm concerned that not only changes to this function but also changes
to its callers could break these assumptions, and nothing near those
places would tell us about them.
Yeah, I agree with this concern. Looking at 788ec96d591, I think remote_slot_precedes
was redundant with slotsync_skip_reason, but found_consistent_snapshot was not:
slotsync_skip_reason reports why synchronization was skipped, while found_consistent_snapshot
tells the caller whether the slot is safe to persist.
v2 is correct with the current temporary slot caller, which persists the slot
immediately. But relying on this call sequence seems fragile. So I'd vote for
restoring found_consistent_snapshot.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Mon, Nov16, 2026 at 02:35 PM Masahiko Sawada wrote:
On Sun, Nov 15, 2026 at 8:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Nov 16, 2026 at 9:50 AM Zhijie Hou <houzhijie22@gmail.com> wrote:
On Sat, Nov14, 2026 at 02:18 PM Masahiko Sawada wrote:
On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
While I agree that retaining the previous reason doesn't make us
persist the slot, I think it's safe only because the previous
reason is never SS_SKIP_NONE for a temporary slot at that point.
That's true today, but it depends on several things and it would
be hard to modify these code correctly in the future.I think the root cause is that commit 788ec96d591 replaced
found_consistent_snapshot with the check of
slotsync_skip_reason, but they're not equivalent:
found_consistent_snapshot is false unless the current cycle
synced the slot and found a consistent snapshot, whereas
slotsync_skip_reason is SS_SKIP_NONE unless the current cycle
detected a reason to skip. So an alternative idea would be to
bring back found_consistent_snapshot part from the commit. That
way, the decision of persisting the slot doesn't depend on the
skip_reasons at all.As for the reported reason, a temporary slot is the one we have
not confirmed to reach a consistent point yet. So I think we can
simply report no_consistent_snapshot for a temporary slot if the
current cycle didn't find a consistent snapshot, without relying
on what happened in the previous cycles.I think it would be unnecessary to reintroduce the
found_consistent_snapshot out-parameter. Because by definition,
slotsync_skip_reason is SS_SKIP_NONE (NULL in
pg_replication_slots) only when slot synchronization succeeds.
Even before commit 788ec96d591 (since commit e68b6adad9),
resetting slotsync_skip_reason to SS_SKIP_NONE on a no-op cycle
for a temporary slot was already a bug, because synchronization of
that temporary slot had not succeeded yet.I agree that the bug is because we didn't maintain/update
slotsync_skip_reason correctly and we anyway need to maintain that
state to correctly display it in pg_replication_slots. Also, the
slot already carries the information about consistent_snapshot.
SS_SKIP_NO_CONSISTENT_SNAPSHOT says the slot hasn't reached a
consistent point, and it is visible in pg_replication_slots. A flag
passed back to the caller to say the same thing would be a duplicate
of that state.I agree that the v2 patch fixes the problem correctly. My concern is
rather about readability and maintainability, which is why I proposed
the PoC patch.+ else if (slot->data.persistency == RS_TEMPORARY) + { + /* + * We reach this point when a previous cycle updated confirmed_flush + * but failed to build a consistent snapshot, and the remote slot has + * not advanced beyond the local slot since then. + */ + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT; + }Some my concerns are:
- When a temporary slot finds a consistent snapshot, it should be
persisted before update_local_synced_slot() is called for the slot
next time. If update_local_synced_slot() is called twice for a
temporary slot without the remote slot advancing, the two calls give
different results. The first call syncs the slot, finds a consistent
snapshot and leaves SS_SKIP_NONE, but the second call enters the new
branch and sets SS_SKIP_NO_CONSISTENT_SNAPSHOT, although nothing has
changed since the first call. This doesn't happen today only because
the caller persists the slot right after the first call.
To make this invariant explicit and catch any future caller changes, I
think we can add an assertion at the entry of update_local_synced_slot()
verifying that a temporary slot with a valid confirmed_flush always has
slotsync_skip_reason != SS_SKIP_NONE. (See the updated patch for details.)
I still prefer this over reintroducing an out-function parameter, since
update_local_synced_slot() already has to maintain slotsync_skip_reason
accurately on every path for pg_replication_slots. Relying on
slot->slotsync_skip_reason != SS_SKIP_NONE in
update_and_persist_local_synced_slot() avoids maintaining and passing a
duplicate state for the same condition which I think that's worth keeping.
- The comment explains the branch by what happened in a previous
cycle. So one needs to go through the possible history of the slot to
see why it's correct, whereas what the code checks is just that the
temporary slot hasn't found a consistent snapshot in this call.- We set SS_SKIP_NO_CONSISTENT_SNAPSHOT in two places for what is
actually the same condition, which makes the code harder to follow.
To address these points, I moved found_consistent_snapshot to the outer
scope of update_local_synced_slot() and checking "if
(slot->data.persistency == RS_TEMPORARY && !found_consistent_snapshot)"
in one central place, with the comment updated to explain the check in
terms of the current call. This also emits the LOG message in both
cases, addressing Amit's point upthread.
I'm concerned that not only changes to this function but also changes
to its callers could break these assumptions, and nothing near those
places would tell us about them.
Attached is the v3 patch set (with the TAP test split into 0002). Please
let me know what you think.
--
Best Regards,
Zhijie Hou
Attachments:
t258370_18v3-0001-Fix-premature-persistence-of-synced-slots-without.patchtext/plain; charset=UTF-8; name=v3-0001-Fix-premature-persistence-of-synced-slots-without.patchDownload+34-18
v3-0002-Add-test-for-keeping-synced-slots-temporary-witho.patchtext/plain; charset=UTF-8; name=v3-0002-Add-test-for-keeping-synced-slots-temporary-witho.patchDownload+53-1
On Mon, Nov 16, 2026 at 1:21 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Sun, Nov 15, 2026 at 10:35:57PM -0800, Masahiko Sawada wrote:
I'm concerned that not only changes to this function but also changes
to its callers could break these assumptions, and nothing near those
places would tell us about them.Yeah, I agree with this concern. Looking at 788ec96d591, I think remote_slot_precedes
was redundant with slotsync_skip_reason, but found_consistent_snapshot was not:
slotsync_skip_reason reports why synchronization was skipped, while found_consistent_snapshot
tells the caller whether the slot is safe to persist.v2 is correct with the current temporary slot caller, which persists the slot
immediately. But relying on this call sequence seems fragile. So I'd vote for
restoring found_consistent_snapshot.
I would have equally worried if this is an exposed function but this
is a local function used in a narrow scope. I feel the wrong usage, if
any, we expect could be covered by Asserts. And, it could be deduced
by slot's property whether it is safe to persist the slot or not but
if we still want to derive it by another flag, it is better to use a
more direct flag like 'can_persist_slot' to decide whether to persist
the slot. Having said that, I feel we are over worried about the wrong
usage of update_local_synced_slot().
--
With Regards,
Amit Kapila.
Hi,
On Mon, Nov 16, 2026 at 03:00:26PM +0530, Amit Kapila wrote:
On Mon, Nov 16, 2026 at 1:21 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Sun, Nov 15, 2026 at 10:35:57PM -0800, Masahiko Sawada wrote:
I'm concerned that not only changes to this function but also changes
to its callers could break these assumptions, and nothing near those
places would tell us about them.Yeah, I agree with this concern. Looking at 788ec96d591, I think remote_slot_precedes
was redundant with slotsync_skip_reason, but found_consistent_snapshot was not:
slotsync_skip_reason reports why synchronization was skipped, while found_consistent_snapshot
tells the caller whether the slot is safe to persist.v2 is correct with the current temporary slot caller, which persists the slot
immediately. But relying on this call sequence seems fragile. So I'd vote for
restoring found_consistent_snapshot.I would have equally worried if this is an exposed function but this
is a local function used in a narrow scope. I feel the wrong usage, if
any, we expect could be covered by Asserts. And, it could be deduced
by slot's property whether it is safe to persist the slot or not but
if we still want to derive it by another flag, it is better to use a
more direct flag like 'can_persist_slot' to decide whether to persist
the slot. Having said that, I feel we are over worried about the wrong
usage of update_local_synced_slot().
After sleeping on this, I think that was overcautious. Given that this is a local
function and v3 makes the assumption explicit with an Assert, that looks sufficient
to me.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com