crash recovery vs partially written WAL
Hi,
A question from a colleague made me wonder if there are scenarios where
two subsequent crashes could lead to wrong WAL to be applied.
Imagine the following scenario
[ xlog page 1 ][ xlog page 2 ][ xlog page 3 ][ xlog page 4 ]
^flush ^write ^insert
if the machine crashes in this moment, we could end up with a situation
where page 1, 3, 4 made it out out to disk, but page 2 wasn't.
That itself is not a problem, when we perform crash recovery, we'll
detect the end of WAL. We'll zero out the invalid parts of page 2, and
log a end-of-recovery checkpoint (which has to fit either onto page 2 or
3).
What I am concerned about is what happens if after crash recovery we
fill up page 3 with new valid records, ending exactly at the page
boundary (i.e. .
[ xlog page 1 ][ xlog page 2 ][ xlog page 3 ][ xlog page 4 ]
^(flush,write)
^insert
if we crash now, we'll peform recovery from the end-fo-recovery record
somewhere on page 2 or 3, and replay the rest of page 3.
That's where I see/wonder about a problem: What guarantees that we find
the contents of xlog page 4 to be invalid? The page header will have the
appropriate xl_pageaddr/tli/info. and because the last record on page 3
ended precisely at the page boundary, there'll not be a xlp_rem_len
allowing us to detect this either.
While we zero out WAL pages in-memory before using them, this won't help
in this instance because a) nothing was inserted into page 4 b) page 4
was never written out.
WAL segment recycling doesn't cause similar problems because xlp_pageaddr
protects us against related issues.
Replaying the old records from page 4 is obviously wrong, since they may
rely on modifications the "old" records on page 2/3 would have performed
(but which got lost).
I don't immediately see a good fix for this. The most obvious thing
would be to explicitly zero-out all WAL files beyond the end-of-recovery
point that have a "correct" xlp_pageaddr, but that may reading a lot of
WAL due to WAL file recycling.
I hope I am missing some crosscheck making this a non-issue?
Greetings,
Andres Freund
On Wed, Dec 30, 2020 at 12:52:46PM -0800, Andres Freund wrote:
Hi,
A question from a colleague made me wonder if there are scenarios where
two subsequent crashes could lead to wrong WAL to be applied.Imagine the following scenario
[ xlog page 1 ][ xlog page 2 ][ xlog page 3 ][ xlog page 4 ]
^flush ^write ^insertif the machine crashes in this moment, we could end up with a situation
where page 1, 3, 4 made it out out to disk, but page 2 wasn't.
I don't see any flaw in your logic. Seems we have to zero out all
future WAL files, not just to the end of the current one, or at least
clear xlp_pageaddr on each future page.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
On Wed, Dec 30, 2020 at 12:52:46PM -0800, Andres Freund wrote:
A question from a colleague made me wonder if there are scenarios where
two subsequent crashes could lead to wrong WAL to be applied.Imagine the following scenario
[ xlog page 1 ][ xlog page 2 ][ xlog page 3 ][ xlog page 4 ]
^flush ^write ^insertif the machine crashes in this moment, we could end up with a situation
where page 1, 3, 4 made it out out to disk, but page 2 wasn't.I don't see any flaw in your logic. Seems we have to zero out all
future WAL files, not just to the end of the current one, or at least
clear xlp_pageaddr on each future page.
I've wondered before if we should be doing a timeline switch at the end
of crash recovery...
Thanks,
Stephen
On Thu, Dec 31, 2020 at 02:27:44PM -0500, Stephen Frost wrote:
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
On Wed, Dec 30, 2020 at 12:52:46PM -0800, Andres Freund wrote:
A question from a colleague made me wonder if there are scenarios where
two subsequent crashes could lead to wrong WAL to be applied.Imagine the following scenario
[ xlog page 1 ][ xlog page 2 ][ xlog page 3 ][ xlog page 4 ]
^flush ^write ^insertif the machine crashes in this moment, we could end up with a situation
where page 1, 3, 4 made it out out to disk, but page 2 wasn't.I don't see any flaw in your logic. Seems we have to zero out all
future WAL files, not just to the end of the current one, or at least
clear xlp_pageaddr on each future page.I've wondered before if we should be doing a timeline switch at the end
of crash recovery...
For a while we had trouble tracking timeline switches, but I think we
might be fine on that now.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee