BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

Started by PG Bug reporting form10 days ago5 messagesbugs
Jump to latest
#1PG Bug reporting form
noreply@postgresql.org

The following bug has been logged on the website:

Bug reference: 19897
Logged by: Haruka Takatsuka
Email address: harukat@sraoss.co.jp
PostgreSQL version: 15.0
Operating system: Linux
Description:

When using the following procedure to recover a base backup taken from a
standby,
the service would sometimes fail to start up with log messages in the
following strange sequence:

LOG: consistent recovery state reached at ...
LOG: invalid record length at ...
FATAL: recovery ended before configured recovery target was reached
LOG: database system is ready to accept read-only connections
LOG: startup process (PID 1234) exited with exit code 1

The base backup had the same result after repeated attempts.

(recovery steps)
$ rm -f data/15.5reco/log/*
$ rm data/15.5reco/standby.signal
$ touch data/15.5reco/recovery.signal
$ cat >> data/15.5reco/postgresql.conf <<EOS
recovery_target = 'immediate'
restore_command = 'false' ## mimics not to be able to get valid wal and
history
recovery_target_action = 'pause'

EOS

$ pg_ctl start -D data/15.5reco
pg_ctl: could not start server

$ cat data/15.5/reco/log/*
2026-11-10 16:16:55.811 JST [3810] LOG: starting PostgreSQL 15.5 on
x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20250617 (Red Hat
14.3.1-2), 64-bit
2026-11-10 16:16:55.812 JST [3810] LOG: listening on IPv6 address "::1",
port 5434
2026-11-10 16:16:55.812 JST [3810] LOG: listening on IPv4 address
"127.0.0.1", port 5434
2026-11-10 16:16:55.817 JST [3810] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5434"
2026-11-10 16:16:55.822 JST [3814] LOG: database system was interrupted
while in recovery at log time 2026-09-24 09:40:18 JST
2026-11-10 16:16:55.822 JST [3814] HINT: If this has occurred more than
once some data might be corrupted and you might need to choose an earlier
recovery target.
2026-11-10 16:16:56.448 JST [3814] LOG: starting point-in-time recovery to
earliest consistent point
2026-11-10 16:16:56.458 JST [3814] LOG: redo starts at 0/12000028
2026-11-10 16:16:56.963 JST [3814] LOG: consistent recovery state reached
at 0/1470A6B0
2026-11-10 16:16:56.963 JST [3814] LOG: invalid record length at
0/1470A6B0: wanted 24, got 0
2026-11-10 16:16:56.963 JST [3814] LOG: redo done at 0/1470A678 system
usage: CPU: user: 0.00 s, system: 0.40 s, elapsed: 0.50 s
2026-11-10 16:16:56.963 JST [3814] LOG: last completed transaction was at
log time 2026-09-24 09:42:16.241123+09
2026-11-10 16:16:56.963 JST [3814] FATAL: recovery ended before configured
recovery target was reached
2026-11-10 16:16:56.963 JST [3810] LOG: database system is ready to accept
read-only connections
2026-11-10 16:16:56.965 JST [3810] LOG: startup process (PID 3814) exited
with exit code 1
2026-11-10 16:16:56.965 JST [3810] LOG: terminating any other active server
processes
2026-11-10 16:16:56.965 JST [3810] LOG: shutting down due to startup
process failure
2026-11-10 16:16:56.967 JST [3810] LOG: database system is shut down

When I checked the source code, I found that recoveryStopsAfter() returned
false
at the condition "if (rmid != RM_XACT_ID) return false;" before reaching
the condition "if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)".

After modifying the order of decisions as follows, I can now boot from the
same backup properly.
It looks like all PostgreSQL branches have the same problem.

diff --git a/src/backend/access/transam/xlogrecovery.c
b/src/backend/access/transam/xlogrecovery.c
index 52ac6d58423..796226efad4 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record)
                return true;
        }
+       /* Check if we should stop as soon as reaching consistency */
+       if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
+       {
+               ereport(LOG,
+                               (errmsg("recovery stopping after reaching
consistency")));
+
+               recoveryStopAfter = true;
+               recoveryStopXid = InvalidTransactionId;
+               recoveryStopTime = 0;
+               recoveryStopLSN = InvalidXLogRecPtr;
+               recoveryStopName[0] = '\0';
+               return true;
+       }
+
        if (rmid != RM_XACT_ID)
                return false;

@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
}
}

- /* Check if we should stop as soon as reaching consistency */
- if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
- {
- ereport(LOG,
- (errmsg("recovery stopping after reaching
consistency")));
-
- recoveryStopAfter = true;
- recoveryStopXid = InvalidTransactionId;
- recoveryStopTime = 0;
- recoveryStopLSN = InvalidXLogRecPtr;
- recoveryStopName[0] = '\0';
- return true;
- }
-
return false;
}

#2Srinath Reddy Sadipiralla
srinath2133@gmail.com
In reply to: PG Bug reporting form (#1)
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

Hi,

Thanks for the report and the analysis. I can reproduce this on master.

On Fri, Nov 13, 2026 at 2:27 PM PG Bug reporting form <
noreply@postgresql.org> wrote:

diff --git a/src/backend/access/transam/xlogrecovery.c
b/src/backend/access/transam/xlogrecovery.c
index 52ac6d58423..796226efad4 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record)
return true;
}
+       /* Check if we should stop as soon as reaching consistency */
+       if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
+       {
+               ereport(LOG,
+                               (errmsg("recovery stopping after reaching
consistency")));
+
+               recoveryStopAfter = true;
+               recoveryStopXid = InvalidTransactionId;
+               recoveryStopTime = 0;
+               recoveryStopLSN = InvalidXLogRecPtr;
+               recoveryStopName[0] = '\0';
+               return true;
+       }
+
if (rmid != RM_XACT_ID)
return false;

@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
}
}

- /* Check if we should stop as soon as reaching consistency */
- if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
- {
- ereport(LOG,
- (errmsg("recovery stopping after reaching
consistency")));
-
- recoveryStopAfter = true;
- recoveryStopXid = InvalidTransactionId;
- recoveryStopTime = 0;
- recoveryStopLSN = InvalidXLogRecPtr;
- recoveryStopName[0] = '\0';
- return true;
- }
-
return false;
}

Your proposed fix of moving the check above the rmgr test does fix the
reported case. But if the record that makes us consistent is a commit
or abort, it now returns before the SetLatestXTime() call, so the "last
completed transaction" time and pg_last_xact_replay_timestamp() would
miss that transaction. The attached 0001 instead goes back to the
pre-4f1b890b137 structure: the rmgr test becomes part of the condition
for the transaction-record block, so the SetLatestXTime still happens and
the immediate check is reached for every record. If this patch makes sense
i will write a TAP test, thoughts?

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/

Attachments:

t259192_2
v1-0001-Fix-recovery_target-immediate-when-WAL-ends-at-th.patchapplication/octet-stream; name=v1-0001-Fix-recovery_target-immediate-when-WAL-ends-at-th.patchDownload+9-8
#3Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Srinath Reddy Sadipiralla (#2)
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

Hello

I can also confirm the reported case, and also that the attached patch fixes it.

But I also found a few similar cases, that could be fixed together with it.

The first was pointed out by claude:

Take a data directory without backup_label after a crash (such as pg_ctl stop -m immediate), create recovery.signal and set:

recovery_target = 'immediate'
restore_command = 'false'
recovery_target_action = 'pause'

And then even with 0001, we see:

LOG: starting point-in-time recovery to earliest consistent point
LOG: redo starts at 0/0175BA40
LOG: invalid record length at 0/0177B8B8: expected at least 24, got 0
LOG: consistent recovery state reached at 0/0177B8B8
LOG: redo done at 0/0177B720 ...
FATAL: recovery ended before configured recovery target was reached

This scenario is explicitly mentioned in InitWalRecovery ("Or if you just kill a running primary server and put it into archive recovery by creating a recovery signal file"), but doesn't work, even with 0001 applied. An alternative patch fixes this and the original report both: adding a check right after the main redo loop, before the recovery_target_action switch.

And then there's a third issue, if instead of immediate, we use fast in the above repro, which again doesn't work. That needs some additional changes, and to avoid code duplication, I also did some refactorings around it.

Additionally, all the above was done on the primary, and we can do the same tests on a standby, where things are different. One case (backup on a standby + backup ending in a restore point) requires your fix from v1, so I think even through the reported case is also fixed by my version, we need both to cover everything.

Attached is a 2 part series, where 0001 is the same as before, while 0002 adds additional changes and test coverage. We need both patches for all tests to pass.

Attachments:

t259192_3
v2-0001-Fix-recovery_target-immediate-when-WAL-ends-at-th.patchtext/x-patchDownload+9-8
v2-0002-Fix-recovery_target-immediate-without-a-record-to.patchtext/x-patchDownload+216-34
#4Zsolt Parragi
zsolt.parragi@percona.com
In reply to: Zsolt Parragi (#3)
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

And after thinking about these changes, I realized that both of our patches broke standby promotion: both resulted in cases where we promoted the standby when we shouldn't have.

The attached v3 now doesn't modify the logic when we have a primary connection set, and contains both patches in one file instead.

Attachments:

t259192_4
v3-0001-Fix-immediate-recovery-targets-at-end-of-WAL.patchtext/x-patchDownload+299-34
#5TAKATSUKA Haruka
harukat@sraoss.co.jp
In reply to: Zsolt Parragi (#4)
Re: BUG #19897: Recovery fails after "LOG: consistent recovery state reached"

Thank you for creating the patch.

I applied the contents of this v3 patch to version 15.x
(with no special adjustments except for pg_fallthrough.)

I tested it on the pgdata where the issue was initially detected in 15.x
and confirmed that the problem has been resolved.

I also verified that the regular regression tests passed on 15.x.
But TAP test No.9 of 003_recovery_targets.pl which is commented as
"Check behavior when recovery ends before target is reached" is failed.
Is is expected "FATAL: recovery ended before configured recovery target was reached",
but after applying this patch the server now starts successfully.

+++ tap check in src/test/recovery +++
t/003_recovery_targets.pl .. 9/?
#   Failed test 'recovery end before target reached is a fatal error'
#   at t/003_recovery_targets.pl line 182.
t/003_recovery_targets.pl .. 29/? # Looks like you failed 1 test of 29.
t/003_recovery_targets.pl .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/29 subtests

Test Summary Report
-------------------
t/003_recovery_targets.pl (Wstat: 256 (exited 1) Tests: 29 Failed: 1)
Failed test: 9
Non-zero exit status: 1
Files=1, Tests=29, 202 wallclock secs ( 0.01 usr 0.01 sys + 2.01 cusr 3.90 csys = 5.93 CPU)
Result: FAIL
make: *** [Makefile:23: check] Error 1

$ tail tmp_check/log/003_recovery_targets_standby_8.log
2026-11-19 09:44:18.606 JST [6254] LOG: starting point-in-time recovery to "does_not_exist"
2026-11-19 09:44:18.609 JST [6254] LOG: restored log file "000000010000000000000003" from archive
2026-11-19 09:44:18.621 JST [6254] LOG: redo starts at 0/3000028
2026-11-19 09:44:18.625 JST [6254] LOG: restored log file "000000010000000000000004" from archive
2026-11-19 09:44:18.626 JST [6254] LOG: consistent recovery state reached at 0/3000100
2026-11-19 09:44:18.626 JST [6251] LOG: database system is ready to accept read-only connections
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
2026-11-19 09:44:18.635 JST [6254] LOG: pausing at the end of recovery
2026-11-19 09:44:18.635 JST [6254] HINT: Execute pg_wal_replay_resume() to promote.

I also tried on REL_15_STABLE head code and got same results.

with best regards,
Takatsuka Haruka / SRA OSS

On Sun, 15 Nov 2026 10:15:02 -0600
Zsolt Parragi <zsolt.parragi@percona.com> wrote:

Show quoted text

And after thinking about these changes, I realized that both of our
patches broke standby promotion: both resulted in cases where we
promoted the standby when we shouldn't have.

The attached v3 now doesn't modify the logic when we have a primary
connection set, and contains both patches in one file instead.