damage control mode

On Thu, Jan 07, 2010 at 08:57:15PM -0500, Robert Haas wrote:

On Thu, Jan 7, 2010 at 4:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Jan 7, 2010 at 3:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

I am tempted to say we should clamp down and go into damage control
mode sooner rather than later.

The more I see of the HS patch, the more I think the same. �But my
proposal for "damage control mode" would be to immediately punt
everything else to the next release and focus our energies exclusively
on HS *and* SR.

Hmm. �There's something to what you say, but what about the people who
were expecting their patches to be reviewed and perhaps committed in
the forthcoming CommitFest. �I proposed a schedule for this release
that involved only three CommitFests and it was rejected, so it seems
a bit unfair to pull the rug out from under people at the eleventh
hour. �Will we lose developers if we do this?

Well, I think we should put at least some effort into clearing out
the underbrush. �There's a lot of pretty small stuff in the January CF
list that I think we could go through in a short time. �The biggies,
IMO, are the ones you noted:

Unfortunately, there are some patches that I probably will not feel
confident to commit without your input - in particular, writeable
CTEs, listen/notify, more frame options in window functions -

and Teodor's GIN/GIST stuff. �If we feel that we are in schedule
trouble I think that bumping these to the next release is by far
the sanest response. �Bumping SR so we can get these in would be
completely misguided.

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door. This proposal
needs input from the community. The affected patches are:

I don't see any need to thump the panic button today.

If we *must* have SR and it's not in by the 15th, let's do another
Commitfest rather than jack the people who played by the rules.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Robert Haas wrote:

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door. This proposal
needs input from the community. The affected patches are:

- Listen/Notify Rewrite.
- Writeable CTEs.
- more frame options for window functions
- knngist
- rbtree

This strikes me as quite premature. Heiki just said he expects to have
SR committed next week.

Maybe we need a copy of the Hitchhiker's Guide to the Galaxy, or at
least its front cover
(<http://myhodgepodge.files.wordpress.com/2008/07/hitchhikers_guide_to_galaxy.jpg&gt;)

cheers

andrew

* David Fetter (david@fetter.org) wrote:

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door. This proposal
needs input from the community. The affected patches are:

I don't see any need to thump the panic button today.

If we *must* have SR and it's not in by the 15th, let's do another
Commitfest rather than jack the people who played by the rules.

Playing by the rules isn't a guarantee, sorry. We have to prioritize
at some point or we'll never get it done. It seems like typically we do
that when we're past the point where we had originally planned to
release and only do it because of that pressure. I would have flipped
the priority of the two top items (I'd rather have writeable CTE than a
new listen/notify), but that doesn't change that I think they're under
SR.

I'm gonna have to +1 bumping the lower priority items in favor of having
an on-time release. To be honest, and I don't intend this as a slight
to anyone, but I'm already worried that just getting HS into a state
where everyone is comfortable releasing it to the wild and having users
trust their data to 8.5 is going to be a serious effort between now and
release.

Perhaps if we discover HS and SR go in alot more easily than expected we
could revisit the decision to bump things, but I don't like encouraging
false hope. Would things really happen that way? I tend to doubt it,
since I think after we get HS and SR done there's going to be an
appropriate push for beta.

Just my 2c from reading the list and history. I'm happy to be wrong.

Thanks,

Stephen

On Thu, 2010-01-07 at 20:57 -0500, Robert Haas wrote:

- Listen/Notify Rewrite.
- Writeable CTEs.

...

Votes?

I'm not qualified to vote on how other people spend their time, but here
are my thoughts:

SR was submitted quite some time ago, so I don't see it as breaking the
rules to put it first in line. If we never give the big features the
serious attention required, then they will never get committed. However,
that's easy for me to say, because my feature made it; and we shouldn't
dismiss the frustration of following the rules and still missing the
boat.

Aside: I'll take this alarm as a very strong hint that I shouldn't push
the "range types" any more until the next development cycle.
Particularly because Tom is one of the people with opinions about it, so
I don't want to distract him from features submitted several commitfests
ago.

Regards,
Jeff Davis

On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:

This strikes me as quite premature. Heiki just said he expects to have SR
committed next week.

Getting it committed is not what I'm worried about. What I'm
concerned about is Tom's statement that he believes that HS is not
close to being in a state where it is stable enough to go to beta, and
the possibility that other large patches committed during this
CommitFest might have similar issues.

Maybe we need a copy of the Hitchhiker's Guide to the Galaxy, or at least
its front cover
(<http://myhodgepodge.files.wordpress.com/2008/07/hitchhikers_guide_to_galaxy.jpg&gt;)

I like to save my panicking for things that are worth it, which the
PostgreSQL release schedule is not. But I do think that is worth an
honest discussion of what is possible. Do you feel we can commit six
large patches in the next month, stabilize all of them plus Hot
Standby, and put out a release in June? If not, would you prefer to
slip some patches or slip the schedule?

...Robert

Robert Haas <robertmhaas@gmail.com> writes:

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door. This proposal
needs input from the community. The affected patches are:

- Listen/Notify Rewrite.
- Writeable CTEs.
- more frame options for window functions
- knngist
- rbtree

Looking at this list again, it strikes me that the listen/notify rewrite
might need to go in so that we have a sane framework for listen/notify
with HS. Treating pg_listener as a replicatable table isn't sane at
all, whereas with a message-driven notify mechanism we have at least got
the possibility of shipping the messages to the standby (via WAL) and
having listeners there. I don't want to say we'd actually implement
that in 8.5, but shipping pg_listener tuple updates is just completely
nuts.

The other four things have no apparent connection to HS/SR so I think
they could be punted without creating technical issues. Whether this
is really necessary from a schedule viewpoint is not clear yet.

My thought about it would be to put these four on the back burner;
not necessarily bounce them right away, but not put any effort into
them until we have dealt with the other stuff in the January CF.
At that point we should have a feel for where we are schedule-wise,
and in particular we'll know whether SR is in or not ;-)

regards, tom lane

On Thu, Jan 7, 2010 at 10:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

OK, we have a proposal on the table to bump some patches from this
CommitFest to free up more committer resources, particularly Tom, to
work on Hot Standby and Streaming Replication and attempt to
accelerate the process of getting 8.5 out the door.  This proposal
needs input from the community.  The affected patches are:

- Listen/Notify Rewrite.
- Writeable CTEs.
- more frame options for window functions
- knngist
- rbtree

Looking at this list again, it strikes me that the listen/notify rewrite
might need to go in so that we have a sane framework for listen/notify
with HS.  Treating pg_listener as a replicatable table isn't sane at
all, whereas with a message-driven notify mechanism we have at least got
the possibility of shipping the messages to the standby (via WAL) and
having listeners there.  I don't want to say we'd actually implement
that in 8.5, but shipping pg_listener tuple updates is just completely
nuts.

It's also related to this open issue, so possibly we could kill two
birds with one stone (although I guess that would still leave the
problem of what to do in the back branches).

http://archives.postgresql.org/pgsql-bugs/2009-12/msg00274.php

The other four things have no apparent connection to HS/SR so I think
they could be punted without creating technical issues.  Whether this
is really necessary from a schedule viewpoint is not clear yet.

My thought about it would be to put these four on the back burner;
not necessarily bounce them right away, but not put any effort into
them until we have dealt with the other stuff in the January CF.
At that point we should have a feel for where we are schedule-wise,
and in particular we'll know whether SR is in or not ;-)

That seems wishy-washy. If we're going to have any chance of getting
these patches in, we have to give the patch authors good feedback
early in the CommitFest so that they have time to make the necessary
revisions before the end of the CommitFest. If we think we can swing
it, I'm happy to handle these patches in the normal way; I'm also
happy to say we'll review them all but not commit them for fear of
destabilizing the tree; or we can punt them altogether. We can also
pick and choose. But saying we're going to postpone dealing with them
doesn't seem to me to solve anything.

I also think that postponement is not going to buy us very much time
anyway. Most of them are pretty small.

If SR is in, does that militate toward bumping the patches (because
now we have more potential bugs to fix) or against it (because now you
don't have to help make it committable)?

...Robert

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Jan 7, 2010 at 10:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Looking at this list again, it strikes me that the listen/notify rewrite
might need to go in so that we have a sane framework for listen/notify
with HS.

It's also related to this open issue, so possibly we could kill two
birds with one stone (although I guess that would still leave the
problem of what to do in the back branches).
http://archives.postgresql.org/pgsql-bugs/2009-12/msg00274.php

Uh, no, AFAICS that is an independent problem. The actual bug is that
our Windows signal implementation can lose signals. That spells trouble
in all kinds of contexts, not only NOTIFY.

My thought about it would be to put these four on the back burner;

That seems wishy-washy.

Fair enough ;-). But I don't feel a need to make a decision now,
either. We can at least wait a week and see if Heikki gets SR
committed.

regards, tom lane

On Thu, Jan 7, 2010 at 10:51 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Fair enough ;-).  But I don't feel a need to make a decision now,
either.  We can at least wait a week and see if Heikki gets SR
committed.

OK.

...Robert

Robert Haas wrote:

If we're going to have any chance of getting
these patches in, we have to give the patch authors good feedback
early in the CommitFest so that they have time to make the necessary
revisions before the end of the CommitFest. If we think we can swing
it, I'm happy to handle these patches in the normal way; I'm also
happy to say we'll review them all but not commit them for fear of
destabilizing the tree; or we can punt them altogether.

Presuming enough reviewers (which should be the case this time given the
expectation that submitters also review), the suggested pacing here now
has every patch passing through a round of review and potentially one
update within ten days. Given that, I'm not sure why you're looking to
special case anything here. Give everybody a fair initial run, just
with the reminder that the bar for "ready for committer" is higher than
normal on this last CF, which people have certainly gotten some warning
of. If your patch smells funny at all or seems like it will take a lot
of work from a committer to apply, it's out. Giving someone a review
but then telling them "it looks good, but we just don't want to commit
it right now" is more fair than not getting that author's patch a review
at all, right? So why bounce them prematurely?

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com

David Fetter <david@fetter.org> writes:

If we *must* have SR and it's not in by the 15th, let's do another
Commitfest rather than jack the people who played by the rules.

If we do add another Commitfest what we do is exactly jacking people who
played by the rules. Because all those patches that are already part of
alpha3 have been worked on by people expecting a 4 CF development cycle,
and adjusted their agenda, and want a mid-year release.

Now, I'll second Greg Smith and Tom here, in that I think we need to run
the last commitfest as usual, knowing that the outcome of the commitfest
for any given patch is not "it made it" but "we reviewed it". It's still
right for the project to bump a patch on resources ground rather than on
technical merit, at the end of the commitfest.

Why we can do it this way is because we're not starving on
reviewers. We're starving on commiters time. And seeing this:

https://commitfest.postgresql.org/action/commitfest_view?id=5

Status Summary. Needs Review: 19, Waiting on Author: 5, Ready for
Committer: 2, Committed: 9, Returned with Feedback: 4. Total: 39.

I don't see any reason not to consider all the 24 patches requiring our
attention.

Regards,
--
dim

On Fri, Jan 8, 2010 at 10:02, Dimitri Fontaine <dfontaine@hi-media.com> wrote:

David Fetter <david@fetter.org> writes:

If we *must* have SR and it's not in by the 15th, let's do another
Commitfest rather than jack the people who played by the rules.

If we do add another Commitfest what we do is exactly jacking people who
played by the rules. Because all those patches that are already part of
alpha3 have been worked on by people expecting a 4 CF development cycle,
and adjusted their agenda, and want a mid-year release.

Now, I'll second Greg Smith and Tom here, in that I think we need to run
the last commitfest as usual, knowing that the outcome of the commitfest
for any given patch is not "it made it" but "we reviewed it". It's still
right for the project to bump a patch on resources ground rather than on
technical merit, at the end of the commitfest.

+1.

Why we can do it this way is because we're not starving on
reviewers. We're starving on commiters time. And seeing this:

Well, we're actually somewhat starving on senior reviewers as well.
That can take on things like the index patches, Writable CTE or SR.
We're not starving on reviewers for small-to-medium patches.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:

Why we can do it this way is because we're not starving on
reviewers. We're starving on commiters time. And seeing this:

Well, we're actually somewhat starving on senior reviewers as well.
That can take on things like the index patches, Writable CTE or SR.
We're not starving on reviewers for small-to-medium patches.

We've been talking about having "specialized" reviewers, or multi
layered reviewing. There are several things we do in reviewing, and for
big enough patches there's no need to have the same reviewer do all of
them.

[...searching the archives for a proposal I did already send...]

http://archives.postgresql.org/pgsql-hackers/2009-08/msg00764.php

So this mail proposes we see those separate items to be handled in
review:

- patch (applies, merge, compiles, pass regression)
- code reading (looks like it was already there, no WTF?) [1]http://www.osnews.com/images/comics/wtfm.jpg
- documentation (covers code, targets users, is sufficient)
- testing (code behavior is what is documented, works well)
- creative testing (tried hard to crash it)
- perf testing (profiling, no regression in non optimized cases...)
- you name it

Now the senior reviewers you're talking about are required the most for
code reading. We certainly still can have an army of junior reviewers,
or not-wannabe-hackers reviewers checking the other points. That'd push
the bottleneck some.

Regards,
--
dim

[1]: http://www.osnews.com/images/comics/wtfm.jpg

Robert Haas wrote:

On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:

This strikes me as quite premature. Heiki just said he expects to have SR
committed next week.

Getting it committed is not what I'm worried about. What I'm
concerned about is Tom's statement that he believes that HS is not
close to being in a state where it is stable enough to go to beta, and
the possibility that other large patches committed during this
CommitFest might have similar issues.

Looking at our existing schedule, I think it is unlikely we will even
make a June 2010 release date for 8.5. Let me explain why --- here is
the ideal schedule:

Jan 15 start commitfest
Feb 15 stop commitfest
Apr 1 start beta
Jun 1 release release candidate (RC)
Jun 20 release 8.5

You can't move from commitfest to beta until all _known_ bugs are
fixed/addressed, and you can't move from beta to RC using the same
criteria.

Of course we rarely have an ideal schedule --- we should expect
unexpected problems. Now, I know people are going to say I am being
over pessimistic, but history will bear out my analysis. I think we
should anticipate a July or August release of 8.5.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> wrote:

here is the ideal schedule:

Jan 15 start commitfest
Feb 15 stop commitfest
Apr 1 start beta
Jun 1 release release candidate (RC)
Jun 20 release 8.5

Of course we rarely have an ideal schedule

So for a project which strives for an annual release, we have over
six months from initial submission of a *small* patch until the
release it goes in, if we meet the ideal schedule? Longer for large
patches or if we slip from the ideal schedule? That sounds like
there are still some opportunities for improvements in project
management, but it's not *so* bad if development for the next
release can overlap the tail end of that schedule. I'm not sure how
much that is anticipated or encouraged, though.

It also seems to call into question the wisdom of annual releases.
If we had a two-year cycle which had three times as much in it,
would that be an improvement, or not?

-Kevin

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote:

Bruce Momjian <bruce@momjian.us> wrote:

Jan 15 start commitfest

Jun 20 release 8.5

over six months

OK, so "over *five* months". Still....

-Kevin

* Kevin Grittner (Kevin.Grittner@wicourts.gov) wrote:

It also seems to call into question the wisdom of annual releases.
If we had a two-year cycle which had three times as much in it,
would that be an improvement, or not?

At the moment, my vote would be "how 'bout we discuss this post-8.5?".
Maybe if we postpone the discussion till then, we'll actually be able to
chew through everything and release close to on-time, otherwise I wonder
if a thread like this won't end up sucking up too much in terms of our
resources right at crunch time..

Thanks,

Stephen

On Jan 8, 2010, at 1:02 AM, Dimitri Fontaine wrote:

Now, I'll second Greg Smith and Tom here, in that I think we need to run
the last commitfest as usual, knowing that the outcome of the commitfest
for any given patch is not "it made it" but "we reviewed it". It's still
right for the project to bump a patch on resources ground rather than on
technical merit, at the end of the commitfest.

+1

David

On Fri, Jan 8, 2010 at 11:44 AM, Bruce Momjian <bruce@momjian.us> wrote:

Robert Haas wrote:

On Thu, Jan 7, 2010 at 9:11 PM, Andrew Dunstan <andrew@dunslane.net> wrote:

This strikes me as quite premature. Heiki just said he expects to have SR
committed next week.

Getting it committed is not what I'm worried about.  What I'm
concerned about is Tom's statement that he believes that HS is not
close to being in a state where it is stable enough to go to beta, and
the possibility that other large patches committed during this
CommitFest might have similar issues.

Looking at our existing schedule, I think it is unlikely we will even
make a June 2010 release date for 8.5.  Let me explain why --- here is
the ideal schedule:

       Jan 15 start commitfest
       Feb 15 stop commitfest
       Apr  1 start beta
       Jun  1 release release candidate (RC)
       Jun 20 release 8.5

You can't move from commitfest to beta until all _known_ bugs are
fixed/addressed, and you can't move from beta to RC using the same
criteria.

Hmm. For 8.4, I don't think we actually fixed "all" known bugs - I
think we made a decision about which ones had to be fixed and which
ones we were going to push off, and then did so.

Of course we rarely have an ideal schedule --- we should expect
unexpected problems.  Now, I know people are going to say I am being
over pessimistic, but history will bear out my analysis.  I think we
should anticipate a July or August release of 8.5.

I think so, too, but I'm actually afraid that if we don't start making
some tough decisions soon it's going to be even later than that. I'm
dismayed by the number of people who seem to think that the current
schedule is not already tight.

...Robert