Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

---------- Forwarded message ---------
From: Srinath Reddy Sadipiralla <srinath2133@gmail.com>
Date: Fri, Jul 10, 2026 at 9:48 AM
Subject: Fix "unexpected logical decoding status change" error; from
concurrent logical decoding activation
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>

Hi,

While stress-testing REPACK CONCURRENTLY, I was in that
area looking at BUG #19519 (REPACK failing with "missingchunk
number N for toast value" [0]/messages/by-id/19519-fe02d8ff679d834d@postgresql.org; note that one reproduces with plain
REPACK too, since it's the shared copy_table_data() path, so
it's independent of what follows), I hit a different, reproducible
failure under high concurrency:
ERROR: unexpected logical decoding status change 1

I believe this is a race bug in the on-demand logical decoding
activation machinery (logicalctl.c), exposed by
REPACK CONCURRENTLY but not actually specific to it, as
this can be reproduced using pg_create_logical_replication_slot
and CREATE_REPLICATION_SLOT.

At wal_level = 'replica', logical decoding is toggled on the first time a
logical slot is created. EnableLogicalDecoding() sets the shared
logical_decoding_enabled flag and writes an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record so standbys
learn about the change. On the decoding side, xlog_decode() treats
that record as unreachable:
elog(ERROR, "unexpected logical decoding status change %d", ...);

The reasoning (per the comment there) is that no running decoder can ever
have such a record within the LSN range it scans: there is exactly one
enable record per disabled->enabled transition, and it precedes the
decoding start point reserved by any slot.

That case does not hold under concurrency. EnableLogicalDecoding()
does:

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
if (logical_decoding_enabled) /* already on? -> done */
{ ... return; }
LWLockRelease(...);

WaitForProcSignalBarrier(...); /* lock released here */

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
logical_decoding_enabled = true;
write_logical_decoding_status_update_record(true); /* the record */
LWLockRelease(...);

The "already enabled?" check and the record write happen under two
separate lock acquisitions, with the barrier wait in between. The lock
must be dropped across the barrier; backends absorbing the barrier
take LogicalDecodingControlLock in shared mode (via
IsXLogLogicalInfoEnabled()), so holding it across WaitForProcSignalBarrier()
would deadlock.

So if two backends create the first logical slot(s) at the same time,
both can pass the initial check while the flag is still false, and both
end up writing a status-change record. The second (redundant) record
lands after the decoding start point the other slot already reserved,
so that slot scans it and trips the elog() above.

REPACK CONCURRENTLY makes this easy to hit because each operation
creates a temporary logical slot (max_repack_replication_slots), so firing
many REPACKs in parallel from the disabled state gives you many backends
racing to perform the very first activation. But nothing here is
REPACK-specific.

To reproduce this i have added a test using logical-decoding-activation
injection point in 051_effective_wal_level.pl

I also hit it with a stress script [1]/messages/by-id/18351-f6e06364b3a2e669@postgresql.org which was again related to the
BUG #19519, instead of vacuum full i replaced it with REPACK (concurrently).

To fix this I re-checked the flag after re-acquiring the lock and skipped
the state change/WAL write if another backend had already completed
the transition.

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);

if (LogicalDecodingCtl->logical_decoding_enabled)
{
LogicalDecodingCtl->pending_disable = false;
LWLockRelease(LogicalDecodingControlLock);
return;
}

START_CRIT_SECTION();

With this, only the backend that actually performs the disabled->enabled
transition writes a record, and every slot's start point stays at or after
it, restoring the case xlog_decode() depends on.

Patch attached. It fixes logicalctl.c and adds a regression test to
051_effective_wal_level.pl reusing the existing injection-point
infrastructure. With the fix the test passes; without it it fails with
the exact error above.

[0]: /messages/by-id/19519-fe02d8ff679d834d@postgresql.org
/messages/by-id/19519-fe02d8ff679d834d@postgresql.org
[1]: /messages/by-id/18351-f6e06364b3a2e669@postgresql.org
/messages/by-id/18351-f6e06364b3a2e669@postgresql.org

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

On Thu, Jul 9, 2026 at 9:23 PM Srinath Reddy Sadipiralla
<srinath2133@gmail.com> wrote:

---------- Forwarded message ---------
From: Srinath Reddy Sadipiralla <srinath2133@gmail.com>
Date: Fri, Jul 10, 2026 at 9:48 AM
Subject: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>

Hi,

While stress-testing REPACK CONCURRENTLY, I was in that
area looking at BUG #19519 (REPACK failing with "missingchunk
number N for toast value" [0]; note that one reproduces with plain
REPACK too, since it's the shared copy_table_data() path, so
it's independent of what follows), I hit a different, reproducible
failure under high concurrency:
ERROR: unexpected logical decoding status change 1

I believe this is a race bug in the on-demand logical decoding
activation machinery (logicalctl.c), exposed by
REPACK CONCURRENTLY but not actually specific to it, as
this can be reproduced using pg_create_logical_replication_slot and CREATE_REPLICATION_SLOT.

At wal_level = 'replica', logical decoding is toggled on the first time a
logical slot is created. EnableLogicalDecoding() sets the shared
logical_decoding_enabled flag and writes an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record so standbys
learn about the change. On the decoding side, xlog_decode() treats
that record as unreachable:
elog(ERROR, "unexpected logical decoding status change %d", ...);

The reasoning (per the comment there) is that no running decoder can ever
have such a record within the LSN range it scans: there is exactly one
enable record per disabled->enabled transition, and it precedes the
decoding start point reserved by any slot.

That case does not hold under concurrency. EnableLogicalDecoding()
does:

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
if (logical_decoding_enabled) /* already on? -> done */
{ ... return; }
LWLockRelease(...);

WaitForProcSignalBarrier(...); /* lock released here */

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
logical_decoding_enabled = true;
write_logical_decoding_status_update_record(true); /* the record */
LWLockRelease(...);

The "already enabled?" check and the record write happen under two
separate lock acquisitions, with the barrier wait in between. The lock
must be dropped across the barrier; backends absorbing the barrier
take LogicalDecodingControlLock in shared mode (via
IsXLogLogicalInfoEnabled()), so holding it across WaitForProcSignalBarrier()
would deadlock.

So if two backends create the first logical slot(s) at the same time,
both can pass the initial check while the flag is still false, and both
end up writing a status-change record. The second (redundant) record
lands after the decoding start point the other slot already reserved,
so that slot scans it and trips the elog() above.

REPACK CONCURRENTLY makes this easy to hit because each operation
creates a temporary logical slot (max_repack_replication_slots), so firing
many REPACKs in parallel from the disabled state gives you many backends
racing to perform the very first activation. But nothing here is REPACK-specific.

To reproduce this i have added a test using logical-decoding-activation
injection point in 051_effective_wal_level.pl

I also hit it with a stress script [1] which was again related to the
BUG #19519, instead of vacuum full i replaced it with REPACK (concurrently).

To fix this I re-checked the flag after re-acquiring the lock and skipped
the state change/WAL write if another backend had already completed
the transition.

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);

if (LogicalDecodingCtl->logical_decoding_enabled)
{
LogicalDecodingCtl->pending_disable = false;
LWLockRelease(LogicalDecodingControlLock);
return;
}

START_CRIT_SECTION();

With this, only the backend that actually performs the disabled->enabled
transition writes a record, and every slot's start point stays at or after
it, restoring the case xlog_decode() depends on.

Patch attached. It fixes logicalctl.c and adds a regression test to
051_effective_wal_level.pl reusing the existing injection-point
infrastructure. With the fix the test passes; without it it fails with
the exact error above.

Thank you for the report and the patch!

I agree with your analysis and the patch basically looks good to me.
Here are some review comments:

+ # Let the released backend finish creating its slot: feed running-xacts
+ # records until it reaches a consistent point (poll_query_until re-runs the
+ # query, so pg_log_standby_snapshot() is called until the slot is created).
+ $primary->poll_query_until(
+ 'postgres', qq[
+select (pg_log_standby_snapshot() is not null)
+  and exists (select 1 from pg_replication_slots
+              where slot_name = 'slot_stray' and confirmed_flush_lsn
is not null)
+]);

I don't think we need to call pg_log_standby_snapshot() until the slot
is created since the slot creation writes the running-xacts record
during the slot creation.

---
+ # Decoding the first slot must not stumble over a stray status-change record.
+ my ($decode_rc, $decode_out, $decode_err) = $primary->psql(
+ 'postgres',
+ qq[select count(*) from pg_logical_slot_get_changes('slot_first',
null, null)],
+ on_error_die => 0);
+ is($decode_rc, 0, "decoding a concurrently-created slot succeeds");
+ unlike(
+ $decode_err,
+ qr/unexpected logical decoding status change/,
+ "no redundant status-change record was decoded");

We can use safe_psql() to check if the query successfully completes.

I've made some cosmetic changes to the comment and the new test
including the above comments. Please review it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jul 14, 2026 at 2:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Jul 9, 2026 at 9:23 PM Srinath Reddy Sadipiralla
<srinath2133@gmail.com> wrote:

---------- Forwarded message ---------
From: Srinath Reddy Sadipiralla <srinath2133@gmail.com>
Date: Fri, Jul 10, 2026 at 9:48 AM
Subject: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>

Hi,

While stress-testing REPACK CONCURRENTLY, I was in that
area looking at BUG #19519 (REPACK failing with "missingchunk
number N for toast value" [0]; note that one reproduces with plain
REPACK too, since it's the shared copy_table_data() path, so
it's independent of what follows), I hit a different, reproducible
failure under high concurrency:
ERROR: unexpected logical decoding status change 1

I believe this is a race bug in the on-demand logical decoding
activation machinery (logicalctl.c), exposed by
REPACK CONCURRENTLY but not actually specific to it, as
this can be reproduced using pg_create_logical_replication_slot and CREATE_REPLICATION_SLOT.

At wal_level = 'replica', logical decoding is toggled on the first time a
logical slot is created. EnableLogicalDecoding() sets the shared
logical_decoding_enabled flag and writes an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record so standbys
learn about the change. On the decoding side, xlog_decode() treats
that record as unreachable:
elog(ERROR, "unexpected logical decoding status change %d", ...);

The reasoning (per the comment there) is that no running decoder can ever
have such a record within the LSN range it scans: there is exactly one
enable record per disabled->enabled transition, and it precedes the
decoding start point reserved by any slot.

That case does not hold under concurrency. EnableLogicalDecoding()
does:

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
if (logical_decoding_enabled) /* already on? -> done */
{ ... return; }
LWLockRelease(...);

WaitForProcSignalBarrier(...); /* lock released here */

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
logical_decoding_enabled = true;
write_logical_decoding_status_update_record(true); /* the record */
LWLockRelease(...);

The "already enabled?" check and the record write happen under two
separate lock acquisitions, with the barrier wait in between. The lock
must be dropped across the barrier; backends absorbing the barrier
take LogicalDecodingControlLock in shared mode (via
IsXLogLogicalInfoEnabled()), so holding it across WaitForProcSignalBarrier()
would deadlock.

So if two backends create the first logical slot(s) at the same time,
both can pass the initial check while the flag is still false, and both
end up writing a status-change record. The second (redundant) record
lands after the decoding start point the other slot already reserved,
so that slot scans it and trips the elog() above.

REPACK CONCURRENTLY makes this easy to hit because each operation
creates a temporary logical slot (max_repack_replication_slots), so firing
many REPACKs in parallel from the disabled state gives you many backends
racing to perform the very first activation. But nothing here is REPACK-specific.

To reproduce this i have added a test using logical-decoding-activation
injection point in 051_effective_wal_level.pl

I also hit it with a stress script [1] which was again related to the
BUG #19519, instead of vacuum full i replaced it with REPACK (concurrently).

To fix this I re-checked the flag after re-acquiring the lock and skipped
the state change/WAL write if another backend had already completed
the transition.

LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);

if (LogicalDecodingCtl->logical_decoding_enabled)
{
LogicalDecodingCtl->pending_disable = false;
LWLockRelease(LogicalDecodingControlLock);
return;
}

START_CRIT_SECTION();

With this, only the backend that actually performs the disabled->enabled
transition writes a record, and every slot's start point stays at or after
it, restoring the case xlog_decode() depends on.

Patch attached. It fixes logicalctl.c and adds a regression test to
051_effective_wal_level.pl reusing the existing injection-point
infrastructure. With the fix the test passes; without it it fails with
the exact error above.

Thank you for the report and the patch!

I agree with your analysis and the patch basically looks good to me.
Here are some review comments:

+ # Let the released backend finish creating its slot: feed running-xacts
+ # records until it reaches a consistent point (poll_query_until re-runs the
+ # query, so pg_log_standby_snapshot() is called until the slot is created).
+ $primary->poll_query_until(
+ 'postgres', qq[
+select (pg_log_standby_snapshot() is not null)
+  and exists (select 1 from pg_replication_slots
+              where slot_name = 'slot_stray' and confirmed_flush_lsn
is not null)
+]);

I don't think we need to call pg_log_standby_snapshot() until the slot
is created since the slot creation writes the running-xacts record
during the slot creation.

---
+ # Decoding the first slot must not stumble over a stray status-change record.
+ my ($decode_rc, $decode_out, $decode_err) = $primary->psql(
+ 'postgres',
+ qq[select count(*) from pg_logical_slot_get_changes('slot_first',
null, null)],
+ on_error_die => 0);
+ is($decode_rc, 0, "decoding a concurrently-created slot succeeds");
+ unlike(
+ $decode_err,
+ qr/unexpected logical decoding status change/,
+ "no redundant status-change record was decoded");

We can use safe_psql() to check if the query successfully completes.

I've made some cosmetic changes to the comment and the new test
including the above comments. Please review it.

While reviewing concurrency aspects of this code before pushing the
fix, I found other race conditions in the same area: logical decoding
can be deactivated while a logical slot is being created on a standby.

On standbys, logical decoding can be deactivated while a logical slot
is being created: either by replaying an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by the end-of-recovery
transition upon promotion, which deactivates logical decoding if no
valid logical slot exists. When logical decoding is deactivated, all
logical slots on the standby are invalidated, but this cannot find a
slot that is not visible yet. Therefore, a slot creation whose status
check interleaved with the deactivation could continue based on a
stale status. This affects two paths:

For regular slot creation, EnsureLogicalDecodingEnabled() assumed that
logical decoding must still be enabled during recovery since the
caller had already checked it. If a promotion interleaves as described
above, the backend creating the slot fails with:

TRAP: failed Assert("IsLogicalDecodingEnabled()"), File: "logicalctl.c"

For slot synchronization, the local slot could be created and
persisted based on the remote slot information fetched before the
deactivation was replayed, leaving a valid slot whose restart_lsn
precedes the deactivation. Decoding such a slot after a failover fails
with:

ERROR: unexpected logical decoding status change 0

These races are confined to the narrow window between checking the
logical decoding status and the new slot becoming visible; once the
slot is visible, the invalidation performed by the deactivation
already covers it. So the fix is simple: re-check the logical decoding
status after the new slot becomes visible. Regular slot creation
raises an error and slot synchronization skips persisting the slot. If
the deactivation happens after the recheck instead, it is guaranteed
to invalidate the now-visible slot as usual. The attached 0002
implements this.

0001 is the fix for the originally reported issue, unchanged from the
previous version.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi Masahiko-San,

On Wed, Jul 15, 2026 at 2:33 AM Masahiko Sawada <sawada.mshk@gmail.com>
wrote:

+ # Let the released backend finish creating its slot: feed running-xacts
+ # records until it reaches a consistent point (poll_query_until re-runs
the
+ # query, so pg_log_standby_snapshot() is called until the slot is
created).
+ $primary->poll_query_until(
+ 'postgres', qq[
+select (pg_log_standby_snapshot() is not null)
+  and exists (select 1 from pg_replication_slots
+              where slot_name = 'slot_stray' and confirmed_flush_lsn
is not null)
+]);

I don't think we need to call pg_log_standby_snapshot() until the slot
is created since the slot creation writes the running-xacts record
during the slot creation.

---
+ # Decoding the first slot must not stumble over a stray status-change
record.
+ my ($decode_rc, $decode_out, $decode_err) = $primary->psql(
+ 'postgres',
+ qq[select count(*) from pg_logical_slot_get_changes('slot_first',
null, null)],
+ on_error_die => 0);
+ is($decode_rc, 0, "decoding a concurrently-created slot succeeds");
+ unlike(
+ $decode_err,
+ qr/unexpected logical decoding status change/,
+ "no redundant status-change record was decoded");

We can use safe_psql() to check if the query successfully completes.

makes sense.

I've made some cosmetic changes to the comment and the new test
including the above comments. Please review it.

LGTM.

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

Hi Masahiko-san,

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada <sawada.mshk@gmail.com>
wrote:

While reviewing concurrency aspects of this code before pushing the
fix, I found other race conditions in the same area: logical decoding
can be deactivated while a logical slot is being created on a standby.

On standbys, logical decoding can be deactivated while a logical slot
is being created: either by replaying an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by the end-of-recovery
transition upon promotion, which deactivates logical decoding if no
valid logical slot exists. When logical decoding is deactivated, all
logical slots on the standby are invalidated, but this cannot find a
slot that is not visible yet. Therefore, a slot creation whose status
check interleaved with the deactivation could continue based on a
stale status. This affects two paths:

For regular slot creation, EnsureLogicalDecodingEnabled() assumed that
logical decoding must still be enabled during recovery since the
caller had already checked it. If a promotion interleaves as described
above, the backend creating the slot fails with:

TRAP: failed Assert("IsLogicalDecodingEnabled()"), File: "logicalctl.c"

For slot synchronization, the local slot could be created and
persisted based on the remote slot information fetched before the
deactivation was replayed, leaving a valid slot whose restart_lsn
precedes the deactivation. Decoding such a slot after a failover fails
with:

ERROR: unexpected logical decoding status change 0

These races are confined to the narrow window between checking the
logical decoding status and the new slot becoming visible; once the
slot is visible, the invalidation performed by the deactivation
already covers it. So the fix is simple: re-check the logical decoding
status after the new slot becomes visible. Regular slot creation
raises an error and slot synchronization skips persisting the slot. If
the deactivation happens after the recheck instead, it is guaranteed
to invalidate the now-visible slot as usual. The attached 0002
implements this.

i have looked into these conditions and they make sense and reviewed the
v3-0002 patch, LGTM.

while reviewing this, I had a thought (it's not related to these race
issues), but
if we disable logical decoding in primary by removing all the slots when
wal_level = replica; it directly invalidates the private slots of the
standby which
seems unfair cause there might be some consumers using it, but then suddenly
they get an error to either change the wal_level = logical on primary or
add a slot
on the primary, but instead i think we can make primary aware of the private
slots of standby and keep logical decoding on, during the
XLOG_LOGICAL_DECODING_STATUS_CHANGE record redo, thoughts?

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
"The truth is... I am Iron Man."

On Mon, Jul 20, 2026 at 9:47 AM Srinath Reddy Sadipiralla
<srinath2133@gmail.com> wrote:

Hi Masahiko-san,

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

While reviewing concurrency aspects of this code before pushing the
fix, I found other race conditions in the same area: logical decoding
can be deactivated while a logical slot is being created on a standby.

On standbys, logical decoding can be deactivated while a logical slot
is being created: either by replaying an
XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by the end-of-recovery
transition upon promotion, which deactivates logical decoding if no
valid logical slot exists. When logical decoding is deactivated, all
logical slots on the standby are invalidated, but this cannot find a
slot that is not visible yet. Therefore, a slot creation whose status
check interleaved with the deactivation could continue based on a
stale status. This affects two paths:

For regular slot creation, EnsureLogicalDecodingEnabled() assumed that
logical decoding must still be enabled during recovery since the
caller had already checked it. If a promotion interleaves as described
above, the backend creating the slot fails with:

TRAP: failed Assert("IsLogicalDecodingEnabled()"), File: "logicalctl.c"

For slot synchronization, the local slot could be created and
persisted based on the remote slot information fetched before the
deactivation was replayed, leaving a valid slot whose restart_lsn
precedes the deactivation. Decoding such a slot after a failover fails
with:

ERROR: unexpected logical decoding status change 0

These races are confined to the narrow window between checking the
logical decoding status and the new slot becoming visible; once the
slot is visible, the invalidation performed by the deactivation
already covers it. So the fix is simple: re-check the logical decoding
status after the new slot becomes visible. Regular slot creation
raises an error and slot synchronization skips persisting the slot. If
the deactivation happens after the recheck instead, it is guaranteed
to invalidate the now-visible slot as usual. The attached 0002
implements this.

i have looked into these conditions and they make sense and reviewed the
v3-0002 patch, LGTM.

Thank you for reviewing the patch!

while reviewing this, I had a thought (it's not related to these race issues), but
if we disable logical decoding in primary by removing all the slots when
wal_level = replica; it directly invalidates the private slots of the standby which
seems unfair cause there might be some consumers using it, but then suddenly
they get an error to either change the wal_level = logical on primary or add a slot
on the primary, but instead i think we can make primary aware of the private
slots of standby and keep logical decoding on, during the
XLOG_LOGICAL_DECODING_STATUS_CHANGE record redo, thoughts?

IIUC it's too late to inform the primary during
XLOG_LOGICAL_DECODING_STATUS_CHANGE redo; the standby replays that
record only after the primary has disabled logical decoding, so WAL
lacking the information required for logical decoding has already been
generated. Once such a gap exists, the standby's slots cannot decode
past it even if the primary re-enabled logical decoding in response,
so we would have to invalidate them anyway.

Alternatively, standbys could proactively tell the primary about their
slots (like hot_standby_feedback), but I don't think this can be made
reliable. With cascaded standbys the information has to be propagated
up through each level, and the propagation lag leaves an unavoidable
race: by the time the primary learns that a downstream server still
needs logical WAL, its slots may already be gone, or a new slot could
be created right after the primary decided to disable.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jul 21, 2026 at 6:12 AM Masahiko Sawada <sawada.mshk@gmail.com>
wrote:

IIUC it's too late to inform the primary during
XLOG_LOGICAL_DECODING_STATUS_CHANGE redo; the standby replays that
record only after the primary has disabled logical decoding, so WAL
lacking the information required for logical decoding has already been
generated. Once such a gap exists, the standby's slots cannot decode
past it even if the primary re-enabled logical decoding in response,
so we would have to invalidate them anyway.

Alternatively, standbys could proactively tell the primary about their
slots (like hot_standby_feedback), but I don't think this can be made
reliable. With cascaded standbys the information has to be propagated
up through each level, and the propagation lag leaves an unavoidable
race: by the time the primary learns that a downstream server still
needs logical WAL, its slots may already be gone, or a new slot could
be created right after the primary decided to disable.

makes sense.

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
" Maybe that's what Batman is about. Not winning. But failing, and getting
back up. "

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Tue, Jul 14, 2026 at 2:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

These races are confined to the narrow window between checking the
logical decoding status and the new slot becoming visible; once the
slot is visible, the invalidation performed by the deactivation
already covers it. So the fix is simple: re-check the logical decoding
status after the new slot becomes visible.

*
- * CheckLogicalDecodingRequirements() must have already errored out if
- * logical decoding is not enabled since we cannot enable the logical
- * decoding status during recovery.
+ * The caller has already checked that logical decoding is enabled via
+ * CheckLogicalDecodingRequirements(), but the status could have been
+ * disabled concurrently before our slot being created: either by
+ * replaying an XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by
+ * UpdateLogicalDecodingStatusEndOfRecovery() upon promotion. We
+ * cannot enable logical decoding during recovery, so raise an error.
  */
- Assert(IsLogicalDecodingEnabled());
+ if (!IsLogicalDecodingEnabled())
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical decoding on standby requires \"effective_wal_level\"

= \"logical\" on the primary"),

+ errdetail("Logical decoding was concurrently disabled during the
logical replication slot creation.")));

It is not clear after reading the comment above this check what kind
of interlocking would save us from concurrent deactivation by two ways
mentioned by you immediately after this check?

--
With Regards,
Amit Kapila.

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0001 is the fix for the originally reported issue, unchanged from the
previous version.

0001 LGTM. This is a separate fix than 0002, so we can proceed with
the commit of this one unless you want to combine the fix for both the
issues and want to make a single commit.

--
With Regards,
Amit Kapila.

On Wed, Jul 22, 2026 at 7:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

*
- * CheckLogicalDecodingRequirements() must have already errored out if
- * logical decoding is not enabled since we cannot enable the logical
- * decoding status during recovery.
+ * The caller has already checked that logical decoding is enabled via
+ * CheckLogicalDecodingRequirements(), but the status could have been
+ * disabled concurrently before our slot being created: either by
+ * replaying an XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by
+ * UpdateLogicalDecodingStatusEndOfRecovery() upon promotion. We
+ * cannot enable logical decoding during recovery, so raise an error.
*/
- Assert(IsLogicalDecodingEnabled());
+ if (!IsLogicalDecodingEnabled())
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical decoding on standby requires \"effective_wal_level\"

= \"logical\" on the primary"),

+ errdetail("Logical decoding was concurrently disabled during the
logical replication slot creation.")));

It is not clear after reading the comment above this check what kind
of interlocking would save us from concurrent deactivation by two ways
mentioned by you immediately after this check?

I checked that before reaching here, we would have marked the slot as
in_use, so the invalidation will invalidate such a slot.

The other observation I had while looking at this patch was: On a
standby, InvalidateObsoleteReplicationSlots() can disable logical
decoding whenever it invalidates the last valid logical slot via
following check:

if (invalidated_logical && !found_valid_logicalslot)
RequestDisableLogicalDecoding();
RequestDisableLogicalDecoding() isn't guarded against recovery, so a
purely local invalidation (RS_INVAL_HORIZON/WAL_REMOVED/IDLE_TIMEOUT)
makes the standby disable decoding even though the primary still has
it enabled. The standby's status is supposed to follow the primary via
XLOG_LOGICAL_DECODING_STATUS_CHANGE replay, and there's no
self-healing since the primary never sends an "enable". So, won't that
be a problem because after that no new slots will be allowed to be
created on standby and slotsync also won't be able perform sync. Am, I
missing something? If not then probable the above check needs
additional check: "!RecoveryInProgress()".

--
With Regards,
Amit Kapila.

On Fri, Jul 24, 2026 at 12:20 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Wed, Jul 22, 2026 at 7:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

*
- * CheckLogicalDecodingRequirements() must have already errored out if
- * logical decoding is not enabled since we cannot enable the logical
- * decoding status during recovery.
+ * The caller has already checked that logical decoding is enabled via
+ * CheckLogicalDecodingRequirements(), but the status could have been
+ * disabled concurrently before our slot being created: either by
+ * replaying an XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by
+ * UpdateLogicalDecodingStatusEndOfRecovery() upon promotion. We
+ * cannot enable logical decoding during recovery, so raise an error.
*/
- Assert(IsLogicalDecodingEnabled());
+ if (!IsLogicalDecodingEnabled())
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical decoding on standby requires \"effective_wal_level\"

= \"logical\" on the primary"),

+ errdetail("Logical decoding was concurrently disabled during the
logical replication slot creation.")));

It is not clear after reading the comment above this check what kind
of interlocking would save us from concurrent deactivation by two ways
mentioned by you immediately after this check?

I checked that before reaching here, we would have marked the slot as
in_use, so the invalidation will invalidate such a slot.

Right. Will update the comment to make it clearer.

The other observation I had while looking at this patch was: On a
standby, InvalidateObsoleteReplicationSlots() can disable logical
decoding whenever it invalidates the last valid logical slot via
following check:

if (invalidated_logical && !found_valid_logicalslot)
RequestDisableLogicalDecoding();
RequestDisableLogicalDecoding() isn't guarded against recovery, so a
purely local invalidation (RS_INVAL_HORIZON/WAL_REMOVED/IDLE_TIMEOUT)
makes the standby disable decoding even though the primary still has
it enabled. The standby's status is supposed to follow the primary via
XLOG_LOGICAL_DECODING_STATUS_CHANGE replay, and there's no
self-healing since the primary never sends an "enable". So, won't that
be a problem because after that no new slots will be allowed to be
created on standby and slotsync also won't be able perform sync. Am, I
missing something? If not then probable the above check needs
additional check: "!RecoveryInProgress()".

In this case, the checkpointer will try to disable logical decoding
but does nothing as it's still in recovery. See the
RecoveryInProgress() check in DisableLogicalDecodingIfNecessary().

If we skip calling RequestDisableLogicalDecoding() in this case, we
would end up missing the disable request as it can be interleaved with
the promotion process. Imagine:

1. the startup process begins the promotion, decides the
new_status=true as there is a logical slot (in
UpdateLogicalDecodingStatusEndOfRecovery()).
2. a backend process drops the last logical slot and skips calling
RequestDisableLogicalDecoding() as the recovery status is still
in-progress.
3. the startup completes the
UpdateLogicalDecodingStatusEndOfRecovery(). The logical decoding is
enabled even though there is no logical slot.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jul 23, 2026 at 4:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0001 is the fix for the originally reported issue, unchanged from the
previous version.

0001 LGTM. This is a separate fix than 0002, so we can proceed with
the commit of this one unless you want to combine the fix for both the
issues and want to make a single commit.

Agreed and pushed the 0001 patch.

I've updated the comments in the previous-0002 patch, please review it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jul 27, 2026 at 9:36 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Fri, Jul 24, 2026 at 12:20 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

The other observation I had while looking at this patch was: On a
standby, InvalidateObsoleteReplicationSlots() can disable logical
decoding whenever it invalidates the last valid logical slot via
following check:

if (invalidated_logical && !found_valid_logicalslot)
RequestDisableLogicalDecoding();
RequestDisableLogicalDecoding() isn't guarded against recovery, so a
purely local invalidation (RS_INVAL_HORIZON/WAL_REMOVED/IDLE_TIMEOUT)
makes the standby disable decoding even though the primary still has
it enabled. The standby's status is supposed to follow the primary via
XLOG_LOGICAL_DECODING_STATUS_CHANGE replay, and there's no
self-healing since the primary never sends an "enable". So, won't that
be a problem because after that no new slots will be allowed to be
created on standby and slotsync also won't be able perform sync. Am, I
missing something? If not then probable the above check needs
additional check: "!RecoveryInProgress()".

In this case, the checkpointer will try to disable logical decoding
but does nothing as it's still in recovery. See the
RecoveryInProgress() check in DisableLogicalDecodingIfNecessary().

If we skip calling RequestDisableLogicalDecoding() in this case, we
would end up missing the disable request as it can be interleaved with
the promotion process. Imagine:

1. the startup process begins the promotion, decides the
new_status=true as there is a logical slot (in
UpdateLogicalDecodingStatusEndOfRecovery()).
2. a backend process drops the last logical slot and skips calling
RequestDisableLogicalDecoding() as the recovery status is still
in-progress.
3. the startup completes the
UpdateLogicalDecodingStatusEndOfRecovery(). The logical decoding is
enabled even though there is no logical slot.

Thanks, I missed the check in DisableLogicalDecodingIfNecessary(). But
will it be better to add a comment atop that check and also atop
RequestDisableLogicalDecoding()? If so, how about something like
attached?

--
With Regards,
Amit Kapila.

On Mon, Jul 27, 2026 at 10:10 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Agreed and pushed the 0001 patch.

I've updated the comments in the previous-0002 patch, please review it.

In the comments you mentioned 'already-visible slot' as follows: "Our
already-visible slot guarantees that this check doesn't miss ..." but
we normally don't use visibility term for slots (it is used primarily
for tuples). I think you mean to say that its in_use flag is set, I
tried to slightly modify that part of the comment in the attached.
See, if that looks okay to you.

Apart from that, I ran the added test on my Windows machine and it
failed, attached find the required log.

--
With Regards,
Amit Kapila.

On Wed, Jul 29, 2026 at 3:21 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Mon, Jul 27, 2026 at 10:10 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Agreed and pushed the 0001 patch.

I've updated the comments in the previous-0002 patch, please review it.

In the comments you mentioned 'already-visible slot' as follows: "Our
already-visible slot guarantees that this check doesn't miss ..." but
we normally don't use visibility term for slots (it is used primarily
for tuples). I think you mean to say that its in_use flag is set, I
tried to slightly modify that part of the comment in the attached.
See, if that looks okay to you.

Fair point. I've updated the comments and commit message not to use
the visibility term.

Apart from that, I ran the added test on my Windows machine and it
failed, attached find the required log.

Thank you for testing it. It seems to be reproducible also on CI, and
I believe I've fixed the issue.

I've attached the patch. Please review it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jul 30, 2026 at 12:37 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Thank you for testing it. It seems to be reproducible also on CI, and
I believe I've fixed the issue.

I've attached the patch. Please review it.

LGTM.

--
With Regards,
Amit Kapila.

On Thu, Jul 30, 2026 at 2:00 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Thu, Jul 30, 2026 at 12:37 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Thank you for testing it. It seems to be reproducible also on CI, and
I believe I've fixed the issue.

I've attached the patch. Please review it.

LGTM.

Thank you for reviewing the patch. Pushed.

... but some buildfarm members are unhappy. I'm working on it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jul 30, 2026 at 1:58 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Jul 30, 2026 at 2:00 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Thu, Jul 30, 2026 at 12:37 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Thank you for testing it. It seems to be reproducible also on CI, and
I believe I've fixed the issue.

I've attached the patch. Please review it.

LGTM.

Thank you for reviewing the patch. Pushed.

... but some buildfarm members are unhappy. I'm working on it.

I've reproduced the issue on local and confirmed the root cause; two
background psql sessions in the TAP test run with ON_ERROR_STOP=1 and
got canceled by pg_cancel_backend(), so psql exited as soon as the
cancelation error arrived. I've attached the patch that fixes the
problem. I'm waiting for CI to check if the tests still pass on other
environments.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Masahiko Sawada <sawada.mshk@gmail.com> writes:

I've reproduced the issue on local and confirmed the root cause; two
background psql sessions in the TAP test run with ON_ERROR_STOP=1 and
got canceled by pg_cancel_backend(), so psql exited as soon as the
cancelation error arrived. I've attached the patch that fixes the
problem. I'm waiting for CI to check if the tests still pass on other
environments.

The buildfarm looks like this has made 051_effective_wal_level.pl
less stable, not more so. The failures all look like

[18:37:44.952](0.014s) # injection_point 'logical-decoding-activation' is reached
[18:37:44.982](0.030s) ok 35 - the activation process aborted
[18:37:44.982](0.001s) # die: ack Broken pipe: write( 13, '\\q
# ' ) at /usr/share/perl5/vendor_perl/IPC/Run/IO.pm line 550.
[18:37:44.982](0.000s) 1..35
ack Broken pipe: write( 13, '\\q
' ) at /usr/share/perl5/vendor_perl/IPC/Run/IO.pm line 550.
# Postmaster PID for node "primary" is 81208
### Stopping node "primary" using mode immediate

regards, tom lane