PD_ALL_VISIBLE flag error on 9.0 alpha 4

On Tue, 2010-03-09 at 22:02 -0800, Josh Berkus wrote:

1. Set up 9.0a4 doing SR replication with a 2nd 9.0a4
2. Ran pgbench for a while.
3. Aborted pgbench with Ctl-C
4. Changed vacuum_defer_cleanup_age in postgresql.conf and reloaded
5. Ran pgbench again, and got:

Sidney-Stratton:pg90 josh$ pgbench -c 2 -T 300 bench
starting vacuum...WARNING: PD_ALL_VISIBLE flag was incorrectly set in
relation "pgbench_branches" page 0
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_branches" page 1
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 0
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 1

Understandable.

Time to remove vacuum_defer_cleanup_age, I think.

--
Simon Riggs www.2ndQuadrant.com

Simon Riggs wrote:

On Tue, 2010-03-09 at 22:02 -0800, Josh Berkus wrote:

1. Set up 9.0a4 doing SR replication with a 2nd 9.0a4
2. Ran pgbench for a while.
3. Aborted pgbench with Ctl-C
4. Changed vacuum_defer_cleanup_age in postgresql.conf and reloaded
5. Ran pgbench again, and got:

Sidney-Stratton:pg90 josh$ pgbench -c 2 -T 300 bench
starting vacuum...WARNING: PD_ALL_VISIBLE flag was incorrectly set in
relation "pgbench_branches" page 0
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_branches" page 1
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 0
WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 1

Understandable.

Time to remove vacuum_defer_cleanup_age, I think.

Umm, so what's the bug?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, 2010-03-10 at 23:08 +0200, Heikki Linnakangas wrote:

Time to remove vacuum_defer_cleanup_age, I think.

Umm, so what's the bug?

Whether you call it a bug or just an annoyance is debatable, but the
source of it is clear. Given the lack of effectiveness, I propose
removing it.

Would you agree or disagree with the suggested removal?

--
Simon Riggs www.2ndQuadrant.com

Simon Riggs <simon@2ndQuadrant.com> writes:

Time to remove vacuum_defer_cleanup_age, I think.

Umm, so what's the bug?

Whether you call it a bug or just an annoyance is debatable, but the
source of it is clear.

Maybe to you, but the rest of us would like to know.

Given the lack of effectiveness, I propose
removing it.

I read Josh's recent report at
http://archives.postgresql.org/message-id/4B973C3F.9070501@agliodbs.com
to say that it's quite effective. I think you're being way too hasty to
decide that it can just be dropped.

regards, tom lane

On Wed, 2010-03-10 at 17:55 -0500, Tom Lane wrote:

Simon Riggs <simon@2ndQuadrant.com> writes:

Time to remove vacuum_defer_cleanup_age, I think.

Umm, so what's the bug?

Whether you call it a bug or just an annoyance is debatable, but the
source of it is clear.

Maybe to you, but the rest of us would like to know.

If vacuum_defer_cleanup_age is set higher this causes the xmin to go
backwards, leading to the "PD_ALL_VISIBLE flag was incorrectly set"
warning.

Having this false xmin move backwards doesn't endanger the standby,
since the xids arrive and are checked normally. If they stop arriving
that is fine.

Having the false xmin going backwards is not a serious issue on primary
because the actual xmin does not go backwards. No observer loses
information as a result of this, it is only about whether cleanup
records are generated later than normal, or not.

Given the lack of effectiveness, I propose
removing it.

I read Josh's recent report at
http://archives.postgresql.org/message-id/4B973C3F.9070501@agliodbs.com
to say that it's quite effective. I think you're being way too hasty to
decide that it can just be dropped.

OK, that's enough to not remove it. I was aware of more negative
thoughts and conscious of my own feelings about it being a kluge.

--
Simon Riggs www.2ndQuadrant.com

On 3/10/10 3:26 PM, Simon Riggs wrote:

OK, that's enough to not remove it. I was aware of more negative
thoughts and conscious of my own feelings about it being a kluge.

Well, it *is* a kludge, but it may be the best one for people who want
to use HS/SR to support web applications. So I think we should work on
making it less kludgy.

Ultimately we're going to need publish-XID-to-master, but that's not
realistic for 9.0.

--Josh Berkus

On Wed, 2010-03-10 at 17:12 -0800, Josh Berkus wrote:

On 3/10/10 3:26 PM, Simon Riggs wrote:

OK, that's enough to not remove it. I was aware of more negative
thoughts and conscious of my own feelings about it being a kluge.

Well, it *is* a kludge, but it may be the best one for people who want
to use HS/SR to support web applications. So I think we should work on
making it less kludgy.

That's better, I was worried you'd gone all complimentary on me.

Ultimately we're going to need publish-XID-to-master, but that's not
realistic for 9.0.

Still some open items that I'm working on, but not that.

--
Simon Riggs www.2ndQuadrant.com

That's better, I was worried you'd gone all complimentary on me.

<grin>Never fear that!

Was that setting originally part of your design for HS? If so, why did
you back off from it?

--Josh

On Sat, 2010-03-13 at 11:29 -0800, Josh Berkus wrote:

That's better, I was worried you'd gone all complimentary on me.

<grin>Never fear that!

Was that setting originally part of your design for HS? If so, why did
you back off from it?

We all agreed its a kluge, that's why.

It's also my 3rd choice of solution behind fine-grained lock conflicts
(1st) which would avoid many issues and master/standby in lock step
(2nd).

Having said that I'm much in favour of providing a range of options and
then letting users tell us what works for them.

--
Simon Riggs www.2ndQuadrant.com

It's also my 3rd choice of solution behind fine-grained lock conflicts
(1st) which would avoid many issues and master/standby in lock step
(2nd).

Yeah, I just can't imagine you hunting down all of the corner cases for
fine-grained lock conflicts in time for 9.0. Given what I've been
looking at, it seems like a LOT of work.

--Josh Berkus

On Sat, 2010-03-13 at 15:29 -0800, Josh Berkus wrote:

It's also my 3rd choice of solution behind fine-grained lock conflicts
(1st) which would avoid many issues and master/standby in lock step
(2nd).

Yeah, I just can't imagine you hunting down all of the corner cases for
fine-grained lock conflicts in time for 9.0. Given what I've been
looking at, it seems like a LOT of work.

I can imagine and have done so. That patch was completed more than 6
weeks ago and can still be included in this release.

http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg145951.html

That is at least the second attempt at this feature and the design has
been refined over about 15 months.

--
Simon Riggs www.2ndQuadrant.com

I can imagine and have done so. That patch was completed more than 6
weeks ago and can still be included in this release.

See, this is why you're a committer and I'm not. ;-)

http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg145951.html

Cool! Will test. We'll make this thing work yet!

It would be a very different PostgreSQL without you, Simon. Thanks for
all the hard work.

--Josh Berkus