Remaining beta blockers
The schedule says we're going to wrap 9.3beta1 on Monday, but it doesn't
feel to me like we are anywhere near ready to ship a credible beta.
Of the items on the 9.3 open-items page,
https://wiki.postgresql.org/wiki/PostgreSQL_9.3_Open_Items
there are at least three that seem like absolute drop-dead stop-ship issues:
1. The matviews mess. Changing that will force initdb, more than
likely, so we need it resolved before beta1.
2. The checksum algorithm business. Again, we don't get to tinker with
that anymore once we're in beta.
3. The ProcessUtility restructuring problem. Surely we're not going to
ship a beta with persistent buildfarm failures, which even show up
sometimes on non-CLOBBER_CACHE_ALWAYS animals, eg today at
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=nightjar&dt=2013-04-27%2009%3A27%3A00
Can we get these resolved by Monday, or must we postpone beta?
As far as #1 goes, I think we have little choice at this point but to
remove the unlogged-matviews feature for 9.3. Various alternatives were
kicked around in the "matview scannability rehash" thread but they were
only marginally less klugy, and nobody's stepped up with a patch anyway.
I will undertake to remove unlogged matviews and replace isscannable-
as-a-file-size-property with isscannable-as-a-reloption (unless anyone
feels it would be better as a separate pg_class column?).
I haven't been paying too close attention to the checksum threads
so I'm not sure where we are on #2.
As for #3, there's a draft patch, who's going to take responsibility
for that?
Anything else that's "must fix"?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Apr 27, 2013 at 10:59:32AM -0400, Tom Lane wrote:
The schedule says we're going to wrap 9.3beta1 on Monday, but it doesn't
feel to me like we are anywhere near ready to ship a credible beta.
Of the items on the 9.3 open-items page,
https://wiki.postgresql.org/wiki/PostgreSQL_9.3_Open_Items
there are at least three that seem like absolute drop-dead stop-ship issues:
1. The matviews mess. Changing that will force initdb, more than
likely, so we need it resolved before beta1.
In similar discussions last year, we concluded that forcing initdb after beta
is fine so long as pg_upgrade can handle the change. Any of the proposals for
changing materialized view scannability are easy for pg_upgrade to handle.
As far as #1 goes, I think we have little choice at this point but to
remove the unlogged-matviews feature for 9.3. Various alternatives were
kicked around in the "matview scannability rehash" thread but they were
only marginally less klugy, and nobody's stepped up with a patch anyway.
I will undertake to remove unlogged matviews and replace isscannable-
as-a-file-size-property with isscannable-as-a-reloption (unless anyone
feels it would be better as a separate pg_class column?).
This perspective is all wrong. I hate to be blunt, but that thread ended with
your technical objections to the committed implementation breaking apart and
sinking. There was no consensus to change it on policy/UI grounds, either.
2. The checksum algorithm business. Again, we don't get to tinker with
that anymore once we're in beta.
Since pg_upgrade isn't in a position to migrate beta clusters to a new
checksum algorithm, I share the desire to settle this sooner rather than
later. However, if finalizing it before beta singularly entails slipping beta
by more than a week or two, I think we should cut the beta without doing so.
Then mention in its release notes that "initdb --data-checksums" beta clusters
may require dump/reload to upgrade to a release or later beta.
Anything else that's "must fix"?
Not to my knowledge.
nm
--
Noah Misch
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Noah Misch <noah@leadboat.com> writes:
On Sat, Apr 27, 2013 at 10:59:32AM -0400, Tom Lane wrote:
As far as #1 goes, I think we have little choice at this point but to
remove the unlogged-matviews feature for 9.3.
This perspective is all wrong. I hate to be blunt, but that thread ended with
your technical objections to the committed implementation breaking apart and
sinking. There was no consensus to change it on policy/UI grounds, either.
[ shrug... ] You and Kevin essentially repeated your claims that the
current implementation is OK; nobody else weighed in. I see absolutely
no reason to change my opinion on this. Keeping relisscannable state in
the form of is-the-file-of-nonzero-size is something we WILL regret, and
Pollyanna-ish refusal to believe that is not an adequate reason for
painting ourselves into a corner, especially not for a second-order
feature like unlogged matviews.
The way forward to unlogged matviews that behave the way Kevin wants
is to improve crash recovery so that we can update catalog state after
completing recovery, which is something there are other reasons to want
anyway. But it's far too late to do that in this cycle. In the
meantime I remain convinced that we're better off dropping the feature
until we have an implementation that's not going to bite us in the rear
in the future.
I also note that there are acknowledged-even-by-you bugs in the current
implementation, which no patch has emerged for, so *something's* got to
be done. I'm done waiting for something to happen, and am going to go
fix it in the way I think best.
2. The checksum algorithm business. Again, we don't get to tinker with
that anymore once we're in beta.
Since pg_upgrade isn't in a position to migrate beta clusters to a new
checksum algorithm, I share the desire to settle this sooner rather than
later. However, if finalizing it before beta singularly entails slipping beta
by more than a week or two, I think we should cut the beta without doing so.
Then mention in its release notes that "initdb --data-checksums" beta clusters
may require dump/reload to upgrade to a release or later beta.
Meh. If we want to reserve the right to do that, we'd better do
something about putting a checksum algorithm ID in pg_control after all.
Otherwise we'll be faced with not detecting the algorithm change, or
else changing pg_control format post-beta1, which would break beta
clusters for everybody not just those who'd experimented with checksums.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Apr 27, 2013 at 10:59 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
The schedule says we're going to wrap 9.3beta1 on Monday, but it doesn't
feel to me like we are anywhere near ready to ship a credible beta.
Of the items on the 9.3 open-items page,
https://wiki.postgresql.org/wiki/PostgreSQL_9.3_Open_Items
there are at least three that seem like absolute drop-dead stop-ship issues:
I completely agree. I think it's considerably premature to wrap a
beta at this point. We haven't resolved the issue of what to do about
accidental restores into pg_catalog either; nobody replied to my last
email on that thread.
1. The matviews mess. Changing that will force initdb, more than
likely, so we need it resolved before beta1.
I would like to rip out the whole notion of whether a materialized
view is scannable and am happy to do that on Monday if you're willing
to sit still for it. I think that's better than failing to support
unlogged relations, and I'm confident that the decision to put the
scannability flag in pg_class rather than the backing file is dead
wrong. At the same time, I *also* agree that using the file size as a
flag is untenable.
2. The checksum algorithm business. Again, we don't get to tinker with
that anymore once we're in beta.
I think it's pretty darn clear that we should change the algorithm,
and I think we've got a patch to do that. So we should be able to
resolve this relatively quickly. But +1 for adding a checksum
algorithm ID to pg_control anyway.
3. The ProcessUtility restructuring problem. Surely we're not going to
ship a beta with persistent buildfarm failures, which even show up
sometimes on non-CLOBBER_CACHE_ALWAYS animals, eg today at
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=nightjar&dt=2013-04-27%2009%3A27%3A00As for #3, there's a draft patch, who's going to take responsibility
for that?
I have been assuming that Alvaro was responsible for fixing this since
he (AIUI) was the one who committed what broke it. If that's not the
case, I should probably jump in, since I committed some earlier event
trigger patches. Or Dimitri, as the original author of said patches.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Sat, Apr 27, 2013 at 10:59 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Of the items on the 9.3 open-items page,
https://wiki.postgresql.org/wiki/PostgreSQL_9.3_Open_Items
there are at least three that seem like absolute drop-dead stop-ship issues:
I completely agree. I think it's considerably premature to wrap a
beta at this point. We haven't resolved the issue of what to do about
accidental restores into pg_catalog either; nobody replied to my last
email on that thread.
As far as that item goes, I agree it's must-fix, but I'm not sure it's
must-fix-before-beta.
1. The matviews mess. Changing that will force initdb, more than
likely, so we need it resolved before beta1.
I would like to rip out the whole notion of whether a materialized
view is scannable and am happy to do that on Monday if you're willing
to sit still for it.
That would actually be my druthers too; while I see that we're going to
want such a concept eventually, I'm not convinced that the current
feature is a reasonable (and upward-compatible) subset of what we'll
want later. However, when I proposed doing that earlier, Kevin
complained pretty strenuously. I'm willing to yield on the point,
as long as the implementation doesn't make use of storage-file size
to represent scannability.
I think that's better than failing to support
unlogged relations, and I'm confident that the decision to put the
scannability flag in pg_class rather than the backing file is dead
wrong. At the same time, I *also* agree that using the file size as a
flag is untenable.
Um, wait, it's *not* in pg_class now, and what I was about to do was
go put it there. Is there a typo in the above para, or are you saying
you don't like either approach? If the latter, what concept have you
got for an eventual implementation?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Apr 27, 2013 at 3:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
1. The matviews mess. Changing that will force initdb, more than
likely, so we need it resolved before beta1.I would like to rip out the whole notion of whether a materialized
view is scannable and am happy to do that on Monday if you're willing
to sit still for it.That would actually be my druthers too; while I see that we're going to
want such a concept eventually, I'm not convinced that the current
feature is a reasonable (and upward-compatible) subset of what we'll
want later. However, when I proposed doing that earlier, Kevin
complained pretty strenuously. I'm willing to yield on the point,
as long as the implementation doesn't make use of storage-file size
to represent scannability.I think that's better than failing to support
unlogged relations, and I'm confident that the decision to put the
scannability flag in pg_class rather than the backing file is dead
wrong. At the same time, I *also* agree that using the file size as a
flag is untenable.Um, wait, it's *not* in pg_class now, and what I was about to do was
go put it there. Is there a typo in the above para, or are you saying
you don't like either approach? If the latter, what concept have you
got for an eventual implementation?
If we're going to have it at all, I'd like to make it a flag in the
page header on page 0, or maybe have a dedicated metapage that stores
that detail, and perhaps other things.
But really, I'd rather not have it at all.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Sat, Apr 27, 2013 at 3:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Um, wait, it's *not* in pg_class now, and what I was about to do was
go put it there. Is there a typo in the above para, or are you saying
you don't like either approach? If the latter, what concept have you
got for an eventual implementation?
If we're going to have it at all, I'd like to make it a flag in the
page header on page 0, or maybe have a dedicated metapage that stores
that detail, and perhaps other things.
I cannot say that I find that idea attractive; the biggest problem with
it being that updating such a state flag will be nontransactional,
unless we go to a lot of effort to support rollbacks. ISTM that the
scannability property is a perfectly normal relation property and as
such *ought* to be in the pg_class row, or at worst some other catalog
entry. Why do you think differently?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 27 April 2013 20:23, Robert Haas <robertmhaas@gmail.com> wrote:
On Sat, Apr 27, 2013 at 10:59 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
2. The checksum algorithm business. Again, we don't get to tinker with
that anymore once we're in beta.I think it's pretty darn clear that we should change the algorithm,
and I think we've got a patch to do that. So we should be able to
resolve this relatively quickly.
I'll be working on this today.
But +1 for adding a checksum
algorithm ID to pg_control anyway.
Yes, seems best.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 27 April 2013 19:06, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Noah Misch <noah@leadboat.com> writes:
On Sat, Apr 27, 2013 at 10:59:32AM -0400, Tom Lane wrote:
As far as #1 goes, I think we have little choice at this point but to
remove the unlogged-matviews feature for 9.3.This perspective is all wrong. I hate to be blunt, but that thread ended with
your technical objections to the committed implementation breaking apart and
sinking. There was no consensus to change it on policy/UI grounds, either.[ shrug... ] You and Kevin essentially repeated your claims that the
current implementation is OK; nobody else weighed in.
On other patches, one committer objecting to something is seen as
enough of a blocker to require change. That should work in every
direction.
In any case, we shouldn't all need to line up and vote on stuff, its
so timewasting. Of course, we need to sometimes, but only when the
case is put clearly enough that it can be done sensibly, otherwise we
just end up voting ad hominem.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Apr 27, 2013 at 3:51 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Sat, Apr 27, 2013 at 3:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Um, wait, it's *not* in pg_class now, and what I was about to do was
go put it there. Is there a typo in the above para, or are you saying
you don't like either approach? If the latter, what concept have you
got for an eventual implementation?If we're going to have it at all, I'd like to make it a flag in the
page header on page 0, or maybe have a dedicated metapage that stores
that detail, and perhaps other things.I cannot say that I find that idea attractive; the biggest problem with
it being that updating such a state flag will be nontransactional,
unless we go to a lot of effort to support rollbacks. ISTM that the
scannability property is a perfectly normal relation property and as
such *ought* to be in the pg_class row, or at worst some other catalog
entry. Why do you think differently?
Mostly because of the issue with unlogged tables, I suppose. If
you've got a reasonable idea how to do catalog updates on restart,
though, I could probably be convinced to yield to that.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Sat, Apr 27, 2013 at 3:51 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I cannot say that I find that idea attractive; the biggest problem with
it being that updating such a state flag will be nontransactional,
unless we go to a lot of effort to support rollbacks. ISTM that the
scannability property is a perfectly normal relation property and as
such *ought* to be in the pg_class row, or at worst some other catalog
entry. Why do you think differently?
Mostly because of the issue with unlogged tables, I suppose. If
you've got a reasonable idea how to do catalog updates on restart,
though, I could probably be convinced to yield to that.
Well, it's fairly clear *how* to do it: add some more processing that
occurs after we've completed crash replay. We already have some of
that, eg completion of partial splits in btrees, so it's not that much
of a stretch; it's just a lot of code that's not been written yet.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Simon Riggs <simon@2ndQuadrant.com> writes:
On other patches, one committer objecting to something is seen as
enough of a blocker to require change. That should work in every
direction.
The bottom line here is that we have substantial disagreement on how
unlogged matviews should be implemented, and there's no longer enough
time for coming to a resolution that will satisfy everybody. I think
that means we have to pull the feature from 9.3. If it had not yet
been committed it would certainly not be getting in now over multiple
objections.
Given Robert's concerns, it may be that the same should be said for
scannability tracking. I think it's definitely the case that if we
don't have unlogged matviews then the need for system-level tracking
of scannability is greatly decreased. Kevin's already said that he
plans to work on a much more flexible notion of scannability for 9.4,
and I remain concerned that something we do in haste now might not
prove to be a good upward-compatible basis for that redesign.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 28 April 2013 16:55, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Simon Riggs <simon@2ndQuadrant.com> writes:
On other patches, one committer objecting to something is seen as
enough of a blocker to require change. That should work in every
direction.The bottom line here is that we have substantial disagreement on how
unlogged matviews should be implemented, and there's no longer enough
time for coming to a resolution that will satisfy everybody. I think
that means we have to pull the feature from 9.3. If it had not yet
been committed it would certainly not be getting in now over multiple
objections.
I've not said much good about Mat Views, that is true, but that was
aimed at not running with it as a headline feature without
qualification. I don't take that as far as thinking the feature should
be pulled completely; there is some good worth having in most things.
Is this issue worth pulling the whole feature on?
Given Robert's concerns, it may be that the same should be said for
scannability tracking. I think it's definitely the case that if we
don't have unlogged matviews then the need for system-level tracking
of scannability is greatly decreased. Kevin's already said that he
plans to work on a much more flexible notion of scannability for 9.4,
and I remain concerned that something we do in haste now might not
prove to be a good upward-compatible basis for that redesign.
Given that unlogged tables are somewhat volatile, unlogged matviews
wouldn't be missed much AFAICS. We can add that thought as a later
optimisation.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Simon Riggs <simon@2ndQuadrant.com> writes:
On 28 April 2013 16:55, Tom Lane <tgl@sss.pgh.pa.us> wrote:
The bottom line here is that we have substantial disagreement on how
unlogged matviews should be implemented, and there's no longer enough
time for coming to a resolution that will satisfy everybody. I think
that means we have to pull the feature from 9.3. If it had not yet
been committed it would certainly not be getting in now over multiple
objections.
I've not said much good about Mat Views, that is true, but that was
aimed at not running with it as a headline feature without
qualification. I don't take that as far as thinking the feature should
be pulled completely; there is some good worth having in most things.
Is this issue worth pulling the whole feature on?
I think you misread that. I'm only proposing that we remove *unlogged*
matviews, and perhaps scannability tracking for matviews.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 28 April 2013 21:06, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Simon Riggs <simon@2ndQuadrant.com> writes:
On 28 April 2013 16:55, Tom Lane <tgl@sss.pgh.pa.us> wrote:
The bottom line here is that we have substantial disagreement on how
unlogged matviews should be implemented, and there's no longer enough
time for coming to a resolution that will satisfy everybody. I think
that means we have to pull the feature from 9.3. If it had not yet
been committed it would certainly not be getting in now over multiple
objections.I've not said much good about Mat Views, that is true, but that was
aimed at not running with it as a headline feature without
qualification. I don't take that as far as thinking the feature should
be pulled completely; there is some good worth having in most things.
Is this issue worth pulling the whole feature on?I think you misread that. I'm only proposing that we remove *unlogged*
matviews, and perhaps scannability tracking for matviews.
Happily so.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sun, Apr 28, 2013 at 11:41 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
On Sat, Apr 27, 2013 at 3:51 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I cannot say that I find that idea attractive; the biggest problem with
it being that updating such a state flag will be nontransactional,
unless we go to a lot of effort to support rollbacks. ISTM that the
scannability property is a perfectly normal relation property and as
such *ought* to be in the pg_class row, or at worst some other catalog
entry. Why do you think differently?Mostly because of the issue with unlogged tables, I suppose. If
you've got a reasonable idea how to do catalog updates on restart,
though, I could probably be convinced to yield to that.Well, it's fairly clear *how* to do it: add some more processing that
occurs after we've completed crash replay. We already have some of
that, eg completion of partial splits in btrees, so it's not that much
of a stretch; it's just a lot of code that's not been written yet.
As far as I can see, that would require starting a separate backend
process for every database, and keeping track of which of those
completed their post-recovery work, and disallowing connections to any
given database until the post-recovery work for that database had been
completed. That seems to add quite a few failure modes that we don't
have today, which is why I haven't been much interested in going that
route.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Sun, Apr 28, 2013 at 11:41 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Well, it's fairly clear *how* to do it: add some more processing that
occurs after we've completed crash replay. We already have some of
that, eg completion of partial splits in btrees, so it's not that much
of a stretch; it's just a lot of code that's not been written yet.
As far as I can see, that would require starting a separate backend
process for every database, and keeping track of which of those
completed their post-recovery work, and disallowing connections to any
given database until the post-recovery work for that database had been
completed. That seems to add quite a few failure modes that we don't
have today, which is why I haven't been much interested in going that
route.
Well, I didn't say it would be easy or quick ;-). But you're presuming
quite a number of design elements that don't seem essential to me.
What I'd be inclined to think about as prerequisite work is fixing
things so that a process could reattach to a new database, after
flushing all its caches. That's a feature that's been requested plenty,
and could have applications in autovacuum or other places besides this,
and would certainly get lots of testing outside of crash recovery.
Having done that, we could imagine that the startup process itself
cycles through all the databases and does the fixup work, rather than
complicating the postmaster logic as suggested above. Potentially this
could replace the filesystem-scan-driven fixup logic that exists in it
now, if it turns out to be faster to search for unlogged tables via a
catalog scan rather than directory scans.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Tom Lane <tgl@sss.pgh.pa.us> wrote:
[ shrug... ] You and Kevin essentially repeated your claims that
the current implementation is OK; nobody else weighed in.
Many people weighed in on the need to differentiate between an
empty matview and one which has not been populated. Many have also
weighed in on the benefits of unlogged matviews.
I see absolutely no reason to change my opinion on this. Keeping
relisscannable state in the form of is-the-file-of-nonzero-size
is something we WILL regret
... or change in a subsequent major release. I see no reason why
using this technique now make it harder to do something else later.
Could you elaborate on the technical challenges you see to doing
so?
Pollyanna-ish refusal to believe that is not an adequate reason
for painting ourselves into a corner, especially not for a
second-order feature like unlogged matviews.
I would guess that about half the use-cases for materialized views
will stay with tables in spite of the added hassle, if they have to
degrade performance by adding logging where they currently have
none.
The way forward to unlogged matviews that behave the way Kevin
wants is to improve crash recovery so that we can update catalog
state after completing recovery, which is something there are
other reasons to want anyway.
That would certainly be for the best.
But it's far too late to do that in this cycle.
Yes it is.
In the meantime I remain convinced that we're better off dropping
the feature until we have an implementation that's not going to
bite us in the rear in the future.
I haven't caught up with you on how it will do that.
I also note that there are acknowledged-even-by-you bugs in the
current implementation, which no patch has emerged for, so
*something's* got to be done. I'm done waiting for something to
happen, and am going to go fix it in the way I think best.
Are you referring to the case where if you refresh a matview with
over 8GB and it winds up empty, that vacuum can make it look
invalid until the next REFRESH? This is one of many ways that
could be fixed.
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 8a1ffcf..b950b16 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -230,7 +230,13 @@ lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
*
* Don't even think about it unless we have a shot at releasing a goodly
* number of pages. Otherwise, the time taken isn't worth it.
+ *
+ * Leave a populated materialized view with at least one page.
*/
+ if (onerel->rd_rel->relkind == RELKIND_MATVIEW &&
+ vacrelstats->nonempty_pages == 0)
+ vacrelstats->nonempty_pages = 1;
+
possibly_freeable = vacrelstats->rel_pages - vacrelstats->nonempty_pages;
if (possibly_freeable > 0 &&
(possibly_freeable >= REL_TRUNCATE_MINIMUM ||The hysteria and FUD about using the currently-supported technique
for unlogged tables to implement unlogged matviews are very
discouraging. And the notion that we would release a matview
feature which allowed false results (in the form of treating a
never-populated matview as a legal empty matview) is truly scary to
me. If we can't tell the difference between those two things, I
don't think we should be releasing the feature.
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
* Kevin Grittner (kgrittn@ymail.com) wrote:
Many people weighed in on the need to differentiate between an
empty matview and one which has not been populated. Many have also
weighed in on the benefits of unlogged matviews.
Sure, I think I did that- we should be able to distinguish between those
two cases and unlogged matviews are great.
I see absolutely no reason to change my opinion on this. Keeping
relisscannable state in the form of is-the-file-of-nonzero-size
is something we WILL regret... or change in a subsequent major release. I see no reason why
using this technique now make it harder to do something else later.
Could you elaborate on the technical challenges you see to doing
so?
I've not followed this all that closely, to be honest, but I tend to
side with Tom on this, making PG depend on the file size for an
attribute of a relation is a bad idea. For one thing, what happens when
an admin figures out that they can 'hack' the system to do what they
want by truncating that file? Or we end up wanting to have that file be
non-zero and considered 'empty' later, but we don't want pg_upgrade
running around touching all of the existing files out there?
I would guess that about half the use-cases for materialized views
will stay with tables in spite of the added hassle, if they have to
degrade performance by adding logging where they currently have
none.
Life's tough. There's quite a few things that I wish had made it into
9.3 which didn't. One might also point out that a lot of individuals
may be, understandably, cautious about this new feature and not use it
in the first major rev it's released in anyway (I'm talking about
matviews entirely here..). Basically, I don't believe we should be
acting like we've promised unlogged-matviews will be in 9.3 just because
it's been committed.
The way forward to unlogged matviews that behave the way Kevin
wants is to improve crash recovery so that we can update catalog
state after completing recovery, which is something there are
other reasons to want anyway.That would certainly be for the best.
Then let's forgo having this specific feature in 9.3 and implement it
correctly for 9.4.
The hysteria and FUD about using the currently-supported technique
for unlogged tables to implement unlogged matviews are very
discouraging.
Perhaps I'm all wet here, but it's not obvious to me that what's done
for unlogged tables really equates to what's being done here.
And the notion that we would release a matview
feature which allowed false results (in the form of treating a
never-populated matview as a legal empty matview) is truly scary to
me. If we can't tell the difference between those two things, I
don't think we should be releasing the feature.
I agree that it's important to distinguish the two. I'm not sure that
I've heard anyone explicitly say otherwise..
Thanks,
Stephen
Stephen Frost <sfrost@snowman.net> wrote:
what happens when an admin figures out that they can 'hack' the
system to do what they want by truncating that file?
If they modified the heap files that way while the server was
running, the results would be somewhat unpredictable. If they did
it while the server was stopped, starting the server and attempting
to access the matview would generate:
ERROR: materialized view "matview_name" has not been populated
HINT: Use the REFRESH MATERIALIZED VIEW command.
Or we end up wanting to have that file be non-zero and considered
'empty' later, but we don't want pg_upgrade running around
touching all of the existing files out there?
I didn't follow this one; could you restate it, please?
--
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
