WAL & RC1 status

I am *not* feeling good about pushing out an RC1 release candidate
today.

I've been going through the WAL code, trying to understand it and
document it. I've found a number of minor problems and several major
ones ("major" meaning "can't really fix without an incompatible file
format change, hence initdb"). I've reported the major problems to
the mailing lists but gotten almost no feedback about what to do.

In addition, I'm still looking for the bug that I originally went in to
find: Scott Parish's report of being unable to restart after a normal
shutdown of beta4. Examination of his WAL log shows some pretty serious
lossage (see attached dump). My current theory is that the
buffer-slinging logic in xlog.c dropped one or more whole buffers' worth
of log records, but I haven't figured out exactly how.

I want to veto putting out an RC1 until these issues are resolved...
comments?

I was not sure how to respond. Requiring an initdb at this stage seems
like it could be a pretty major blow to beta testers. However, if we
will have 7.1 problems with WAL that can not be fixed without a file
format change, we will have problems down the road. Is there a version
number in the WAL file? Can we put conditional code in there to create
new log file records with an updated format?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Is there a version number in the WAL file?

catversion.h will do fine, no?

Can we put conditional code in there to create
new log file records with an updated format?

The WAL stuff is *far* too complex already. I've spent a week studying
it and I only partially understand it. I will not consent to trying to
support multiple log file formats concurrently.

regards, tom lane

On Fri, 2 Mar 2001, Tom Lane wrote:

I am *not* feeling good about pushing out an RC1 release candidate
today.

I've been going through the WAL code, trying to understand it and
document it. I've found a number of minor problems and several major
ones ("major" meaning "can't really fix without an incompatible file
format change, hence initdb"). I've reported the major problems to
the mailing lists but gotten almost no feedback about what to do.

In addition, I'm still looking for the bug that I originally went in to
find: Scott Parish's report of being unable to restart after a normal
shutdown of beta4. Examination of his WAL log shows some pretty serious
lossage (see attached dump). My current theory is that the
buffer-slinging logic in xlog.c dropped one or more whole buffers' worth
of log records, but I haven't figured out exactly how.

I want to veto putting out an RC1 until these issues are resolved...
comments?

Will second it ... Vadim is supposed to be back on the 6th, and Peter has
a couple of changes to configure he wants to do this weekend for the JDBC
stuff ... Thomas and I are in SF the end of next week for some meetings,
so if you can pop off a summary of what you've found to either of us, and
assuming that Vadim doesn't get caught up by then, we can bring them up
"in person" at that time ... ?

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Is there a version number in the WAL file?

catversion.h will do fine, no?

Can we put conditional code in there to create
new log file records with an updated format?

The WAL stuff is *far* too complex already. I've spent a week studying
it and I only partially understand it. I will not consent to trying to
support multiple log file formats concurrently.

Well, I was thinking a few things. Right now, if we update the
catversion.h, we will require a dump/reload. If we can update just the
WAL version stamp, that will allow us to fix WAL format problems without
requiring people to dump/reload. I can imagine this would be valuable
if we find we need to make changes in 7.1.1, where we can not require
dump/reload.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, I was thinking a few things. Right now, if we update the
catversion.h, we will require a dump/reload. If we can update just the
WAL version stamp, that will allow us to fix WAL format problems without
requiring people to dump/reload.

Since there is not a separate WAL version stamp, introducing one now
would certainly force an initdb. I don't mind adding one if you think
it's useful; another 4 bytes in pg_control won't hurt anything. But
it's not going to save anyone's bacon on this cycle.

At least one of my concerns (single point of failure) would require a
change to the layout of pg_control, which would force initdb anyway.
Anyone want to propose a third version# for pg_control?

regards, tom lane

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, I was thinking a few things. Right now, if we update the
catversion.h, we will require a dump/reload. If we can update just the
WAL version stamp, that will allow us to fix WAL format problems without
requiring people to dump/reload.

Since there is not a separate WAL version stamp, introducing one now
would certainly force an initdb. I don't mind adding one if you think
it's useful; another 4 bytes in pg_control won't hurt anything. But
it's not going to save anyone's bacon on this cycle.

Having a version number of binary files has saved me many times because
I can add a little 'if' to allow upward binary compatibility without
breaking old binary files. I think we should have one.

I see our btree files, but I don't see one in heap. I am going to
recommend that for 7.2. All our files should have versions just in case
we ever need it. Some day, we may be able to skip dump/reload for major
versions.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

I've been going through the WAL code, trying to understand it and
document it. I've found a number of minor problems and several major
ones ("major" meaning "can't really fix without an incompatible file
format change, hence initdb"). I've reported the major problems to
the mailing lists but gotten almost no feedback about what to do.

Sorry for the "no feedback", but I've assumed that this will be more
productively discussed with Vadim in the loop. I don't disagree with
your observations, but of course that is from a position of happy
ignorance :)

... I want to veto putting out an RC1 until these issues are resolved...
comments?

OK with me.

- Thomas

From: Bruce Momjian [SMTP:pgman@candle.pha.pa.us]
Sent: Friday, March 02, 2001 9:54 AM
To: Tom Lane
Cc: pgsql-core@postgresql.org; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] WAL & RC1 status

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Is there a version number in the WAL file?

catversion.h will do fine, no?

Can we put conditional code in there to create
new log file records with an updated format?

While it may be unfortunate to have to do an initdb at this point in
the beta cycle, it is a beta and that is part of the deal. Postgre has the
reputation of being the highest quality opensource database and we should do
nothing to tarnish that. Release it when it's ready and not before.

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Well, I was thinking a few things. Right now, if we update the
catversion.h, we will require a dump/reload. If we can update just the
WAL version stamp, that will allow us to fix WAL format problems without
requiring people to dump/reload.

Since there is not a separate WAL version stamp, introducing one now
would certainly force an initdb. I don't mind adding one if you think
it's useful; another 4 bytes in pg_control won't hurt anything. But
it's not going to save anyone's bacon on this cycle.

At least one of my concerns (single point of failure) would require a
change to the layout of pg_control, which would force initdb anyway.
Anyone want to propose a third version# for pg_control?

I now remember Hiroshi complaining about major WAL problems also,
particularly corrupt WAL files preventing the database from starting.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Fri, Mar 02, 2001 at 10:54:04AM -0500, Bruce Momjian wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Is there a version number in the WAL file?

catversion.h will do fine, no?

Can we put conditional code in there to create
new log file records with an updated format?

The WAL stuff is *far* too complex already. I've spent a week studying
it and I only partially understand it. I will not consent to trying to
support multiple log file formats concurrently.

Well, I was thinking a few things. Right now, if we update the
catversion.h, we will require a dump/reload. If we can update just the
WAL version stamp, that will allow us to fix WAL format problems without
requiring people to dump/reload. I can imagine this would be valuable
if we find we need to make changes in 7.1.1, where we can not require
dump/reload.

It Seems to Me that after an orderly shutdown, the WAL files should be,
effectively, slag -- they should contain no deltas from the current
table contents. In practice that means the only part of the format that
*should* matter is whatever it takes to discover that they really are
slag.

That *should* mean that, at worst, a change to the WAL file format should
only require doing an orderly shutdown, and then (perhaps) running a simple
program to generate a new-format empty WAL. It ought not to require an
initdb.

Of course the details of the current implementation may interfere with
that ideal, but it seems a worthy goal for the next beta, if it's not
possible already. Given the opportunity to change the current WAL format,
it ought to be possible to avoid even needing to run a program to generate
an empty WAL.

Nathan Myers
ncm@zembu.com

It Seems to Me that after an orderly shutdown, the WAL files should be,
effectively, slag -- they should contain no deltas from the current
table contents. In practice that means the only part of the format that
*should* matter is whatever it takes to discover that they really are
slag.

That *should* mean that, at worst, a change to the WAL file format should
only require doing an orderly shutdown, and then (perhaps) running a simple
program to generate a new-format empty WAL. It ought not to require an
initdb.

Of course the details of the current implementation may interfere with
that ideal, but it seems a worthy goal for the next beta, if it's not
possible already. Given the opportunity to change the current WAL format,
it ought to be possible to avoid even needing to run a program to generate
an empty WAL.

This was my question too. If we are just changing WAL, why can't we
just have them stop the postmaster, install the new binaries, and
restart.

Tom told me on the phone that there was a magic number in the WAL log
file, and I see it now:

#define XLOG_PAGE_MAGIC 0x17345168

Couldn't we just have our new beta ignore WAL pages with this entry,
knowing that startup/shutdown creates new WAL files anyway,

Aside from inconveniencing the beta users, people can do testing easier
if we don't require a dump/reload for every WAL format change.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

ncm@zembu.com (Nathan Myers) writes:

It Seems to Me that after an orderly shutdown, the WAL files should be,
effectively, slag -- they should contain no deltas from the current
table contents. In practice that means the only part of the format that
*should* matter is whatever it takes to discover that they really are
slag.

That *should* mean that, at worst, a change to the WAL file format should
only require doing an orderly shutdown, and then (perhaps) running a simple
program to generate a new-format empty WAL. It ought not to require an
initdb.

Excellent point, considering that we were already thinking of making a
handy-dandy little utility to remove broken WAL files... Shouldn't take
much more than that to build something that also reformats pg_control.
Thanks for the suggestion!

regards, tom lane

I've reported the major problems to the mailing lists
but gotten almost no feedback about what to do.

I can't comment without access to code -:(

commit: 2001-02-26 17:19:57
0/0059996C: prv 0/00599948; xprv 0/00000000; xid 0;
RM 0 info 00 len 32
checkpoint: redo 0/0059996C; undo 0/00000000; sui 29;
nextxid 18903; nextoid 35195; online
-- this is the last normal-looking checkpoint record.
-- Judging from the commit timestamps surrounding prior
-- checkpoints, checkpoints were happening every five
-- minutes approximately on the 5-minute mark, so

You can't count on this: postmaster runs checkpoint
"maker" in 5 minutes *after* prev checkpoint was created,
not from the moment "maker" started. And checkpoint can
take *minutes*.

-- this one happened about 17:20.
-- (There really should be a timestamp
-- in the checkpoint records...)

Agreed.

commit: 2001-02-26 17:26:02
ReadRecord: record with zero len at 0/005A4B4C
-- My dump program is unhappy here because the rest
-- of the page is zero. Given that there is a
-- continuation record at the start of the next
-- page, there certainly should have been record(s)
-- here. But it's worse than that: check the commit
-- timestamps and the xid numbers before and after the
-- discontinuity. Did time go backwards here?

Commit timestamps are created *before* XLogInsert call,
which can suspend backend for some time (in multi-user
env). Random xid-s are also ok, generally.

-- Also notice the back-pointers in the first valid
-- record on the next page; they point not into the
-- zeroed space, which would suggest a mere failure
-- to write a buffer after filling it, but into the
-- middle of one of the valid records on the prior
-- page. It almost looks like page 5A6000 came from
-- a completely different run than page 5A4000.
Unexpected page info flags 0001 at offset 5A6000
Skipping unexpected continuation record at offset 5A6000
0/005A6904: prv 0/005A48B4(?); xprv 0/005A48B4; xid 19047;

^^^^^^^^^^ ^^^^^^^^^^
Same. So, TX 19047 really inserted record at 0/005A48B4
position.

-- What's even nastier (and the immediate cause of
-- Scott's inability to restart) is that the pg_control
-- file's checkPoint pointer points to 0/005AF9F0, which
-- is *not* the location of this checkpoint, but of
-- the record after it.

Well, well. Checkpoint position is taken from
MyLastRecord - I wonder how could this internal var
take "invalid" data from concurrent backend.

Ok, we're leaving Krasnoyarsk in 8 hrs and should
arrive SF Feb 5 ~ 10pm.

Vadim

-----------------------------------------------
FREE! The World's Best Email Address @email.com
Reserve your name now at http://www.email.com

Vadim Mikheev <vadim4o@email.com> writes:

-- Judging from the commit timestamps surrounding prior
-- checkpoints, checkpoints were happening every five
-- minutes approximately on the 5-minute mark, so

You can't count on this: postmaster runs checkpoint
"maker" in 5 minutes *after* prev checkpoint was created,
not from the moment "maker" started. And checkpoint can
take *minutes*.

Good point, although with so little going on (this is the *whole*
relevant section of the log), that seems unlikely.

-- here. But it's worse than that: check the commit
-- timestamps and the xid numbers before and after the
-- discontinuity. Did time go backwards here?

Commit timestamps are created *before* XLogInsert call,
which can suspend backend for some time (in multi-user
env). Random xid-s are also ok, generally.

Hmm ... maybe. Though again, this installation doesn't seem to have
been busy enough to cause a commit to be delayed for very long.

What I realized after posting that analysis is that the last checkpoint
record has SUI 30 whereas the earlier ones have SUI 29 ... so there was
a system restart in there somewhere. That still leaves me wondering
about the discontinuity and broken back-link, but it may account for
the "missing" checkpoint records --- perhaps they weren't generated
because the system wasn't up the entire interval.

-- What's even nastier (and the immediate cause of
-- Scott's inability to restart) is that the pg_control
-- file's checkPoint pointer points to 0/005AF9F0, which
-- is *not* the location of this checkpoint, but of
-- the record after it.

Well, well. Checkpoint position is taken from
MyLastRecord - I wonder how could this internal var
take "invalid" data from concurrent backend.

I have not been able to figure that one out either.

Ok, we're leaving Krasnoyarsk in 8 hrs and should
arrive SF Feb 5 ~ 10pm.

Have a safe trip!

regards, tom lane

Since there is not a separate WAL version stamp, introducing one now
would certainly force an initdb. I don't mind adding one if you think
it's useful; another 4 bytes in pg_control won't hurt anything. But
it's not going to save anyone's bacon on this cycle.

Yes, if initdb, that would probably be a good idea.
Imho the initdb now is not a real issue, since all beta testers
know that for serious issues there might be an initdb after beta started.

At least one of my concerns (single point of failure) would require a
change to the layout of pg_control, which would force initdb anyway.

Was that the "only one checkpoint back in time in pg_control" issue ?
One issue about too many checkpoints in pg_control, is that you then need
to keep more logs, and in my pgbench tests the log space was a real issue,
even for the one checkpoint case. I think a utility to recreate a busted pg_control
would add a lot more stability, than one more checkpoint in pg_control.

We should probably have additional criteria to time, that can trigger a
checkpoint, like N logs filled since last checkpoint. I do not think
reducing the checkpoint interval is a solution for once in a while heavy activity.

Andreas

Zeugswetter Andreas SB <ZeugswetterA@Wien.Spardat.at> writes:

At least one of my concerns (single point of failure) would require a
change to the layout of pg_control, which would force initdb anyway.

Was that the "only one checkpoint back in time in pg_control" issue ?

Yes.

One issue about too many checkpoints in pg_control, is that you then
need to keep more logs, and in my pgbench tests the log space was a
real issue, even for the one checkpoint case. I think a utility to
recreate a busted pg_control would add a lot more stability, than one
more checkpoint in pg_control.

Well, there is a big difference between 1 and 2 checkpoints stored in
pg_control. I don't intend to go further than 2. But I disagree about
a log-reset utility being more useful than an extra checkpoint. The
utility would be for manual recovery after a disaster, and it wouldn't
offer 100% recovery: you couldn't be sure that the last few transactions
had been applied atomically, ie, all or none. (Perhaps pg_log got
updated to show them committed, but not all of their tuple changes made
it to disk; how will you know?) If you can back up to the prior
checkpoint and then roll forward, you *do* have a shot at guaranteeing
a consistent database state after loss of the primary checkpoint.

We should probably have additional criteria to time, that can trigger a
checkpoint, like N logs filled since last checkpoint.

Perhaps. I don't have time to work on that now, but we can certainly
improve the strategy in future releases.

regards, tom lane

Zeugswetter Andreas SB <ZeugswetterA@Wien.Spardat.at> writes:

At least one of my concerns (single point of failure) would require a
change to the layout of pg_control, which would force initdb anyway.

Was that the "only one checkpoint back in time in pg_control" issue ?

Yes.

Is changing pg_control the thing that is going to require the initdb?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

One issue about too many checkpoints in pg_control, is that you then
need to keep more logs, and in my pgbench tests the log space was a
real issue, even for the one checkpoint case. I think a utility to
recreate a busted pg_control would add a lot more stability, than one
more checkpoint in pg_control.

Well, there is a big difference between 1 and 2 checkpoints stored in
pg_control. I don't intend to go further than 2. But I disagree about
a log-reset utility being more useful than an extra checkpoint.

Yes I agree, I thought there was already one additional checkpoint info in
pg_control.

The
utility would be for manual recovery after a disaster, and it wouldn't
offer 100% recovery: you couldn't be sure that the last few transactions
had been applied atomically, ie, all or none. (Perhaps pg_log got
updated to show them committed, but not all of their tuple changes made
it to disk; how will you know?) If you can back up to the prior
checkpoint and then roll forward, you *do* have a shot at guaranteeing
a consistent database state after loss of the primary checkpoint.

Yes, but a consistent db can only be guaranteed if all txlog logs up to the
crash are eighter rolled forward or at least the physical log pages are written
back to disk.

The consequence is imho, that a good utility to reset the logs should keep
all "physical log" pages, and only clear the log from all other records
[optionally starting at the position that hinders rollforward].

Andreas