Follow-up fixes for online wal_level change

Started by Masahiko Sawada3 months ago11 messageshackers
Jump to latest
#1Masahiko Sawada
sawada.mshk@gmail.com

Hi all,

While reviewing the online wal_level change feature (commit
67c20979ce7), I found an assertion failure during crash recovery.
Also, I noticed that pg_controldata doesn't show the logical decoding
status. Patches attached.

0001 fixes the assertion failure that happens in the following scenario:

1. Run the server with wal_level = 'replica' and create a logical
slot, which writes an XLOG_LOGICAL_DECODING_STATUS_CHANGE record that
activates logical decoding.
2. Drop the slot, and crash the server before the checkpointer
deactivates logical decoding.
3. Restart the server with wal_level = 'minimal'.

Crash recovery replays the status change record and activates logical
decoding, then UpdateLogicalDecodingStatusEndOfRecovery() fails the
assertion (Assert(!IsXLogLogicalInfoEnabled() &&
!IsLogicalDecodingEnabled())) that logical decoding is never active
with wal_level='minimal'.

I think that replaying the status change record itself is correct, as
it reflects the status at the time the record was written. The problem
is that the end-of-recovery code assumed that this cannot happen,
instead of adjusting the status. Therefore, the fix removes the
wal_level='minimal' special case from
UpdateLogicalDecodingStatusEndOfRecovery() so that it recomputes the
status as usual.

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Both patches are intended to be backpatched to v19.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v1-0001-Correct-logical-decoding-status-at-end-of-recover.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Correct-logical-decoding-status-at-end-of-recover.patchDownload+9-12
v1-0002-pg_controldata-Show-logical-decoding-status.patchtext/x-patch; charset=US-ASCII; name=v1-0002-pg_controldata-Show-logical-decoding-status.patchDownload+2-1
#2Guoqing Yang
yanggq1988@126.com
In reply to: Masahiko Sawada (#1)
Re: Follow-up fixes for online wal_level change

On 7/16/2026 8:09 AM, Masahiko Sawada wrote:

Hi all,

While reviewing the online wal_level change feature (commit
67c20979ce7), I found an assertion failure during crash recovery.
Also, I noticed that pg_controldata doesn't show the logical decoding
status. Patches attached.

0001 fixes the assertion failure that happens in the following scenario:

1. Run the server with wal_level = 'replica' and create a logical
slot, which writes an XLOG_LOGICAL_DECODING_STATUS_CHANGE record that
activates logical decoding.
2. Drop the slot, and crash the server before the checkpointer
deactivates logical decoding.
3. Restart the server with wal_level = 'minimal'.

I can reproduce this crash following above procedure. But in step 3, I also needed to set max_wal_senders=0, otherwise the serve failed to start with error: FATAL: WAL streaming ("max_wal_senders" > 0) requires "wal_level" to be "replica" or “logical".

Crash recovery replays the status change record and activates logical
decoding, then UpdateLogicalDecodingStatusEndOfRecovery() fails the
assertion (Assert(!IsXLogLogicalInfoEnabled() &&
!IsLogicalDecodingEnabled())) that logical decoding is never active
with wal_level='minimal'.

I think that replaying the status change record itself is correct, as
it reflects the status at the time the record was written. The problem
is that the end-of-recovery code assumed that this cannot happen,
instead of adjusting the status. Therefore, the fix removes the
wal_level='minimal' special case from
UpdateLogicalDecodingStatusEndOfRecovery() so that it recomputes the
status as usual.

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Both patches are intended to be backpatched to v19.

Regards,

Both 0001 and 0002 LGTM.

Regards,
Guoqing Yang

#3Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: Guoqing Yang (#2)
RE: Follow-up fixes for online wal_level change

Dear Sawada-san,

I think that replaying the status change record itself is correct, as
it reflects the status at the time the record was written. The problem
is that the end-of-recovery code assumed that this cannot happen,
instead of adjusting the status. Therefore, the fix removes the
wal_level='minimal' special case from
UpdateLogicalDecodingStatusEndOfRecovery() so that it recomputes the
status as usual.

Good catch. I could reproduce.

I initially thought that we could just skip redoing for
XLOG_LOGICAL_DECODING_STATUS_CHANGE records. I confirmed it could also solve the
issue. Also, I think it could slightly reduce the redoing time because it does
not have to emit/wait the ProcSignalBarrier. But your point is also valid:
either way is fine for me.

I reviewed your patches and I could not find issues.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

#4Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#3)
Re: Follow-up fixes for online wal_level change

On Thu, Jul 16, 2026 at 2:42 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:

Dear Sawada-san,

I think that replaying the status change record itself is correct, as
it reflects the status at the time the record was written. The problem
is that the end-of-recovery code assumed that this cannot happen,
instead of adjusting the status. Therefore, the fix removes the
wal_level='minimal' special case from
UpdateLogicalDecodingStatusEndOfRecovery() so that it recomputes the
status as usual.

Good catch. I could reproduce.

I initially thought that we could just skip redoing for
XLOG_LOGICAL_DECODING_STATUS_CHANGE records. I confirmed it could also solve the
issue. Also, I think it could slightly reduce the redoing time because it does
not have to emit/wait the ProcSignalBarrier. But your point is also valid:
either way is fine for me.

I reviewed your patches and I could not find issues.

Thank you for reviewing these patches.

I've just pushed both patches.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#5Fujii Masao
masao.fujii@gmail.com
In reply to: Masahiko Sawada (#1)
Re: Follow-up fixes for online wal_level change

On Thu, Jul 16, 2026 at 9:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Should this status be exposed also by pg_control_checkpoint()?

Regards,

--
Fujii Masao

#6Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: Fujii Masao (#5)
RE: Follow-up fixes for online wal_level change

Dear Fujii-san,

Should this status be exposed also by pg_control_checkpoint()?

Oh I didn't know the function, yes, seems to be exposed as well.

While checking the function, I found that CheckPoint:: wal_level is not
shown by the code function as well. Should we handle that?
I think pg_controldata command does not have to output the attribute because
ControlFileData::wal_level is also printed, but not sure the SQL function.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

#7Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fujii Masao (#5)
Re: Follow-up fixes for online wal_level change

On Thu, Jul 16, 2026 at 4:46 PM Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Jul 16, 2026 at 9:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Should this status be exposed also by pg_control_checkpoint()?

Indeed. I'll prepare the patch. Thank you for pointing it out.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#8Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Masahiko Sawada (#7)
Re: Follow-up fixes for online wal_level change

On Fri, Jul 17, 2026 at 12:57 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Jul 16, 2026 at 4:46 PM Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Jul 16, 2026 at 9:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Should this status be exposed also by pg_control_checkpoint()?

Indeed. I'll prepare the patch. Thank you for pointing it out.

I've attached the patch. Feedback is very welcome.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v1-0001-Add-logical-decoding-status-to-pg_control_checkpo.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Add-logical-decoding-status-to-pg_control_checkpo.patchDownload+23-21
#9Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Masahiko Sawada (#8)
Re: Follow-up fixes for online wal_level change

On Fri, Jul 17, 2026 at 10:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Fri, Jul 17, 2026 at 12:57 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Jul 16, 2026 at 4:46 PM Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Jul 16, 2026 at 9:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Should this status be exposed also by pg_control_checkpoint()?

Indeed. I'll prepare the patch. Thank you for pointing it out.

I've attached the patch. Feedback is very welcome.

Forgot to update the documentation. Attached v2 patch.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v2-0001-Add-logical-decoding-status-to-pg_control_checkpo.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Add-logical-decoding-status-to-pg_control_checkpo.patchDownload+28-21
#10Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#6)
Re: Follow-up fixes for online wal_level change

On Thu, Jul 16, 2026 at 8:35 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:

Dear Fujii-san,

Should this status be exposed also by pg_control_checkpoint()?

Oh I didn't know the function, yes, seems to be exposed as well.

While checking the function, I found that CheckPoint:: wal_level is not
shown by the code function as well. Should we handle that?
I think pg_controldata command does not have to output the attribute because
ControlFileData::wal_level is also printed, but not sure the SQL function.

ControlFileData.wal_level and ControlFileData.checkPointCopy.wal_level
are different and the only former is currently shown in pg_controldata
and the pg_control_XXX() function family. Considering the
ControlFileData.wal_level is effectively the same as wal_level GUC
value, I think we don't need to show it in pg_control_XXX() functions.
As for the latter, it might be useful for debugging to expose it in a
few cases and we might want to show it in both places if we want. But
I'm not sure it's worth the confusion it could cause. I think it
should be for PG20 anyway.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#11Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Masahiko Sawada (#9)
Re: Follow-up fixes for online wal_level change

On Fri, Jul 17, 2026 at 10:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Fri, Jul 17, 2026 at 10:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Fri, Jul 17, 2026 at 12:57 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Jul 16, 2026 at 4:46 PM Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Jul 16, 2026 at 9:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

0002 adds the logical decoding status stored in the checkpoint record
to the pg_controldata output, which I found useful while investigating
the above issue.

Should this status be exposed also by pg_control_checkpoint()?

Indeed. I'll prepare the patch. Thank you for pointing it out.

I've attached the patch. Feedback is very welcome.

Forgot to update the documentation. Attached v2 patch.

Pushed.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com