BUG #19897: Recovery fails after "LOG: consistent recovery state reached"
The following bug has been logged on the website:
Bug reference: 19897
Logged by: Haruka Takatsuka
Email address: harukat@sraoss.co.jp
PostgreSQL version: 15.0
Operating system: Linux
Description:
When using the following procedure to recover a base backup taken from a
standby,
the service would sometimes fail to start up with log messages in the
following strange sequence:
LOG: consistent recovery state reached at ...
LOG: invalid record length at ...
FATAL: recovery ended before configured recovery target was reached
LOG: database system is ready to accept read-only connections
LOG: startup process (PID 1234) exited with exit code 1
The base backup had the same result after repeated attempts.
(recovery steps)
$ rm -f data/15.5reco/log/*
$ rm data/15.5reco/standby.signal
$ touch data/15.5reco/recovery.signal
$ cat >> data/15.5reco/postgresql.conf <<EOS
recovery_target = 'immediate'
restore_command = 'false' ## mimics not to be able to get valid wal and
history
recovery_target_action = 'pause'
EOS
$ pg_ctl start -D data/15.5reco
pg_ctl: could not start server
$ cat data/15.5/reco/log/*
2026-11-10 16:16:55.811 JST [3810] LOG: starting PostgreSQL 15.5 on
x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20250617 (Red Hat
14.3.1-2), 64-bit
2026-11-10 16:16:55.812 JST [3810] LOG: listening on IPv6 address "::1",
port 5434
2026-11-10 16:16:55.812 JST [3810] LOG: listening on IPv4 address
"127.0.0.1", port 5434
2026-11-10 16:16:55.817 JST [3810] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5434"
2026-11-10 16:16:55.822 JST [3814] LOG: database system was interrupted
while in recovery at log time 2026-09-24 09:40:18 JST
2026-11-10 16:16:55.822 JST [3814] HINT: If this has occurred more than
once some data might be corrupted and you might need to choose an earlier
recovery target.
2026-11-10 16:16:56.448 JST [3814] LOG: starting point-in-time recovery to
earliest consistent point
2026-11-10 16:16:56.458 JST [3814] LOG: redo starts at 0/12000028
2026-11-10 16:16:56.963 JST [3814] LOG: consistent recovery state reached
at 0/1470A6B0
2026-11-10 16:16:56.963 JST [3814] LOG: invalid record length at
0/1470A6B0: wanted 24, got 0
2026-11-10 16:16:56.963 JST [3814] LOG: redo done at 0/1470A678 system
usage: CPU: user: 0.00 s, system: 0.40 s, elapsed: 0.50 s
2026-11-10 16:16:56.963 JST [3814] LOG: last completed transaction was at
log time 2026-09-24 09:42:16.241123+09
2026-11-10 16:16:56.963 JST [3814] FATAL: recovery ended before configured
recovery target was reached
2026-11-10 16:16:56.963 JST [3810] LOG: database system is ready to accept
read-only connections
2026-11-10 16:16:56.965 JST [3810] LOG: startup process (PID 3814) exited
with exit code 1
2026-11-10 16:16:56.965 JST [3810] LOG: terminating any other active server
processes
2026-11-10 16:16:56.965 JST [3810] LOG: shutting down due to startup
process failure
2026-11-10 16:16:56.967 JST [3810] LOG: database system is shut down
When I checked the source code, I found that recoveryStopsAfter() returned
false
at the condition "if (rmid != RM_XACT_ID) return false;" before reaching
the condition "if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)".
After modifying the order of decisions as follows, I can now boot from the
same backup properly.
It looks like all PostgreSQL branches have the same problem.
diff --git a/src/backend/access/transam/xlogrecovery.c
b/src/backend/access/transam/xlogrecovery.c
index 52ac6d58423..796226efad4 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record)
return true;
}
+ /* Check if we should stop as soon as reaching consistency */
+ if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
+ {
+ ereport(LOG,
+ (errmsg("recovery stopping after reaching
consistency")));
+
+ recoveryStopAfter = true;
+ recoveryStopXid = InvalidTransactionId;
+ recoveryStopTime = 0;
+ recoveryStopLSN = InvalidXLogRecPtr;
+ recoveryStopName[0] = '\0';
+ return true;
+ }
+
if (rmid != RM_XACT_ID)
return false;
@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
}
}
- /* Check if we should stop as soon as reaching consistency */
- if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
- {
- ereport(LOG,
- (errmsg("recovery stopping after reaching
consistency")));
-
- recoveryStopAfter = true;
- recoveryStopXid = InvalidTransactionId;
- recoveryStopTime = 0;
- recoveryStopLSN = InvalidXLogRecPtr;
- recoveryStopName[0] = '\0';
- return true;
- }
-
return false;
}
Hi,
Thanks for the report and the analysis. I can reproduce this on master.
On Fri, Nov 13, 2026 at 2:27 PM PG Bug reporting form <
noreply@postgresql.org> wrote:
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 52ac6d58423..796226efad4 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -2789,6 +2789,20 @@ recoveryStopsAfter(XLogReaderState *record) return true; }+ /* Check if we should stop as soon as reaching consistency */ + if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE && reachedConsistency) + { + ereport(LOG, + (errmsg("recovery stopping after reaching consistency"))); + + recoveryStopAfter = true; + recoveryStopXid = InvalidTransactionId; + recoveryStopTime = 0; + recoveryStopLSN = InvalidXLogRecPtr; + recoveryStopName[0] = '\0'; + return true; + } + if (rmid != RM_XACT_ID) return false;@@ -2867,20 +2881,6 @@ recoveryStopsAfter(XLogReaderState *record)
}
}- /* Check if we should stop as soon as reaching consistency */
- if (recoveryTarget == RECOVERY_TARGET_IMMEDIATE &&
reachedConsistency)
- {
- ereport(LOG,
- (errmsg("recovery stopping after reaching
consistency")));
-
- recoveryStopAfter = true;
- recoveryStopXid = InvalidTransactionId;
- recoveryStopTime = 0;
- recoveryStopLSN = InvalidXLogRecPtr;
- recoveryStopName[0] = '\0';
- return true;
- }
-
return false;
}
Your proposed fix of moving the check above the rmgr test does fix the
reported case. But if the record that makes us consistent is a commit
or abort, it now returns before the SetLatestXTime() call, so the "last
completed transaction" time and pg_last_xact_replay_timestamp() would
miss that transaction. The attached 0001 instead goes back to the
pre-4f1b890b137 structure: the rmgr test becomes part of the condition
for the transaction-record block, so the SetLatestXTime still happens and
the immediate check is reached for every record. If this patch makes sense
i will write a TAP test, thoughts?
--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
Hello
I can also confirm the reported case, and also that the attached patch fixes it.
But I also found a few similar cases, that could be fixed together with it.
The first was pointed out by claude:
Take a data directory without backup_label after a crash (such as pg_ctl stop -m immediate), create recovery.signal and set:
recovery_target = 'immediate'
restore_command = 'false'
recovery_target_action = 'pause'
And then even with 0001, we see:
LOG: starting point-in-time recovery to earliest consistent point
LOG: redo starts at 0/0175BA40
LOG: invalid record length at 0/0177B8B8: expected at least 24, got 0
LOG: consistent recovery state reached at 0/0177B8B8
LOG: redo done at 0/0177B720 ...
FATAL: recovery ended before configured recovery target was reached
This scenario is explicitly mentioned in InitWalRecovery ("Or if you just kill a running primary server and put it into archive recovery by creating a recovery signal file"), but doesn't work, even with 0001 applied. An alternative patch fixes this and the original report both: adding a check right after the main redo loop, before the recovery_target_action switch.
And then there's a third issue, if instead of immediate, we use fast in the above repro, which again doesn't work. That needs some additional changes, and to avoid code duplication, I also did some refactorings around it.
Additionally, all the above was done on the primary, and we can do the same tests on a standby, where things are different. One case (backup on a standby + backup ending in a restore point) requires your fix from v1, so I think even through the reported case is also fixed by my version, we need both to cover everything.
Attached is a 2 part series, where 0001 is the same as before, while 0002 adds additional changes and test coverage. We need both patches for all tests to pass.
And after thinking about these changes, I realized that both of our patches broke standby promotion: both resulted in cases where we promoted the standby when we shouldn't have.
The attached v3 now doesn't modify the logic when we have a primary connection set, and contains both patches in one file instead.
Thank you for creating the patch.
I applied the contents of this v3 patch to version 15.x
(with no special adjustments except for pg_fallthrough.)
I tested it on the pgdata where the issue was initially detected in 15.x
and confirmed that the problem has been resolved.
I also verified that the regular regression tests passed on 15.x.
But TAP test No.9 of 003_recovery_targets.pl which is commented as
"Check behavior when recovery ends before target is reached" is failed.
Is is expected "FATAL: recovery ended before configured recovery target was reached",
but after applying this patch the server now starts successfully.
+++ tap check in src/test/recovery +++
t/003_recovery_targets.pl .. 9/?
# Failed test 'recovery end before target reached is a fatal error'
# at t/003_recovery_targets.pl line 182.
t/003_recovery_targets.pl .. 29/? # Looks like you failed 1 test of 29.
t/003_recovery_targets.pl .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/29 subtests
Test Summary Report
-------------------
t/003_recovery_targets.pl (Wstat: 256 (exited 1) Tests: 29 Failed: 1)
Failed test: 9
Non-zero exit status: 1
Files=1, Tests=29, 202 wallclock secs ( 0.01 usr 0.01 sys + 2.01 cusr 3.90 csys = 5.93 CPU)
Result: FAIL
make: *** [Makefile:23: check] Error 1
$ tail tmp_check/log/003_recovery_targets_standby_8.log
2026-11-19 09:44:18.606 JST [6254] LOG: starting point-in-time recovery to "does_not_exist"
2026-11-19 09:44:18.609 JST [6254] LOG: restored log file "000000010000000000000003" from archive
2026-11-19 09:44:18.621 JST [6254] LOG: redo starts at 0/3000028
2026-11-19 09:44:18.625 JST [6254] LOG: restored log file "000000010000000000000004" from archive
2026-11-19 09:44:18.626 JST [6254] LOG: consistent recovery state reached at 0/3000100
2026-11-19 09:44:18.626 JST [6251] LOG: database system is ready to accept read-only connections
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
cp: cannot stat '/var/lib/pgsql/install/postgresql-15.5/src/test/recovery/tmp_check/t_003_recovery_targets_primary_data/archives/000000010000000000000005': No such file or directory
2026-11-19 09:44:18.635 JST [6254] LOG: pausing at the end of recovery
2026-11-19 09:44:18.635 JST [6254] HINT: Execute pg_wal_replay_resume() to promote.
I also tried on REL_15_STABLE head code and got same results.
with best regards,
Takatsuka Haruka / SRA OSS
On Sun, 15 Nov 2026 10:15:02 -0600
Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Show quoted text
And after thinking about these changes, I realized that both of our
patches broke standby promotion: both resulted in cases where we
promoted the standby when we shouldn't have.The attached v3 now doesn't modify the logic when we have a primary
connection set, and contains both patches in one file instead.