Releasing in September

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 04:48:16PM +0100, Andres Freund wrote:

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

Well, when the commitfest extends to May, you are not going to fork the
next release, and since we haven't forked by PGCon in Ottawa, we try to
get in five commitfest, instead of four, and we then get the schedule we
have now.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 07:40 AM, Bruce Momjian wrote:

Our current 9.5/9.6 timing looks more like the 8.X series of release
dates. Everyone might be fine with that, but we had better be prepared
for November-February major release dates going forward.

Bruce,

Thank you for bringing this up it is a good time to discuss it. I think
we have a number of things we need to consider when it comes to releases:

1. When we release
2. What we release
3. When not to release

#1 September is actually a great time to release. However, I think that
we are going to run into trouble keeping that time frame. The reality is
PostgreSQL of today is a lot more complex than the PostgreSQL of even 3
years ago.

That means that unless we are willing to have stunted releases, we are
going to miss release dates. We need to be able to say, "Ompf, no whiz
bang features this release, mostly polish" or "These whiz bang features
need more testing, we are pushing our release". If we aren't willing to
do that then we need to reconsider having a fixed release date and
perhaps start a release time frame (9.6 will be released Q4 2016).

#2 This is a longer topic. I have been stewing in my head about releases
for years. I have even brought up the idea of an Ubuntu style release
cycle on list once or twice. The more I think about it, the more I think
this can help our community. We essentially would have two types of
releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS. This
allows two LTS versions per 5 year support cycle

There is more to say on this but this is already a long reply.

#3 The fact that 9.5 was delayed is a good thing, not a bad one. I
believe we need to be much more willing to push a button that says, "It
isn't done." Which goes back to the idea in #1 of having a release
"window" versus date. We aren't a company. We don't need to adhere to an
exact release schedule.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.  We
twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 10:55:07 -0500, Robert Haas wrote:

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

Said twiddling seems to largely happened from July to December. In which
the other branch was open, and no 9.5 commitfest was happening. If we
move the commitfests to earlier, but still have a half year of nothing
happening, we're still in a bad situation.

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

On 2016-01-20 10:55:07 -0500, Robert Haas wrote:

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

Said twiddling seems to largely happened from July to December. In which
the other branch was open, and no 9.5 commitfest was happening. If we
move the commitfests to earlier, but still have a half year of nothing
happening, we're still in a bad situation.

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

/Magnus

Magnus Hagander <magnus@hagander.net> writes:

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

We will not get back to on-schedule releases unless we can keep -hackers
working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

I'd suggest that the easiest process fix is to not fork a new devel branch
until we reach, say, RC status.

And that does mean losing at least one commitfest per release cycle ---
but it would be the currently-first one, not the currently-last one.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 11:19:28 -0500, Tom Lane wrote:

We will not get back to on-schedule releases unless we can keep -hackers
working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

Agreed. I'll note that the linux kernel used to have a similar
problem. Now there's a two week integration and a 10 week stabilization
period, with occasionally a week added/subtracted. If a feature patch is
submitted for integration (not just parallel review) into the main
branch outside the merge window you get yelled at. Now, we're at least
two magnitudes smaller, so not everything there applies to us. But I
think looking over that fence, to see how they scaled from smaller where
we are now, to way way bigger, isn't a bad idea.

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:

#2 This is a longer topic. I have been stewing in my head about
releases for years. I have even brought up the idea of an Ubuntu
style release cycle on list once or twice. The more I think about
it, the more I think this can help our community. We essentially
would have two types of releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS.
This allows two LTS versions per 5 year support cycle

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:53:32AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

Unfair or not, that is probably the effect. I can also see people
publishing git trees to avoid this roadblock.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 11:53:32 -0500, Robert Haas wrote:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

Yes, that's a problem. But I don't see how we can avoid dealing with it
directly. Avoiding being stalled by unfixed bugs of others by having a
separate branch to continue working, inevitably leads to delayed
releases. If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

You're assuming that the people who are hot to get started on the next
release are different from the people who don't have time to fix the bugs
they introduced previously. IME it's mostly the same people.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 08:51 AM, Bruce Momjian wrote:

On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:

#2 This is a longer topic. I have been stewing in my head about
releases for years. I have even brought up the idea of an Ubuntu
style release cycle on list once or twice. The more I think about
it, the more I think this can help our community. We essentially
would have two types of releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS.
This allows two LTS versions per 5 year support cycle

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

I understand but I think we are missing out on an opportunity here.
Notice that the shorter release cycle for STS will actually make some
things easier. Including:

* Increased test base (just like Fedora/Ubuntu)
* Increased early adopter testing (that is what early adopting is
really about for us anyway)
* Decreased concerns about upgrades and ability to extend upgrade status.

I am not in any way suggesting that this is a slam dunk but I do think
something along these lines can really bring forth a lot of
possibilities. Here is another example:

pg_audit was pulled from core, did it have to be? I can't say. I didn't
read the code. I was a proactive member stating we could pull it though
because it was an extension.

However, if it was an STS release it was going into we could be a little
more relaxed and say, "O.k. let's get this into the wild for some
general user testing".

There are a lot of people (think about the multixact problem) that will
run any software because it is new. Let's put the proper disclaimers on
there and let them run it.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Magnus Hagander <magnus@hagander.net> writes:

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6

I agree that it's not the same problem. I do believe that it is *a* problem
though, and a fairly significant one too. Because there's *never* any
downtime from CF mode, regardless of where in the cycle we are.

While not the same, we need to fix both.

We will not get back to on-schedule releases unless we can keep -hackers

working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

Agreed.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 01/20/2016 09:03 AM, Andres Freund wrote:

If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

We have been slowly moving to a more dictator based release anyway. It
used to be that we released "when it's done", then we moved to
commitfest, and now we are having yet another discussion on the same
topic of how to manage all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

Life is hard sometimes, we don't always get what we want.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote:

On 01/20/2016 09:03 AM, Andres Freund wrote:

If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

We have been slowly moving to a more dictator based release anyway. It used
to be that we released "when it's done", then we moved to commitfest, and
now we are having yet another discussion on the same topic of how to manage
all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

That's not really related to the discussion here tho? The debated
problem is things that have already been committed that have have bugs,
preventing beta/rc releases from being made. In some cases in the last
releases such bugs were in features integrated very early in the cycle.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Magnus Hagander <magnus@hagander.net> writes:

On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6

I agree that it's not the same problem. I do believe that it is *a* problem
though, and a fairly significant one too. Because there's *never* any
downtime from CF mode, regardless of where in the cycle we are.

True, we've been failing badly on the intention that there would be time
off from CF mode, and I'd like to see a fix for that. I do not think it's
directly related to the can't-get-a-release-out problem.

I'm not really sure why we've allowed CFs to drift on, though. Can't we
just arbitrarily decree them closed on the last day of the month? And
push unfinished work to the next one? Admittedly, this probably doesn't
work for the last CF of a release cycle, but that one's always been a
special case.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 09:17 AM, Andres Freund wrote:

On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote:

On 01/20/2016 09:03 AM, Andres Freund wrote:

If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

We have been slowly moving to a more dictator based release anyway. It used
to be that we released "when it's done", then we moved to commitfest, and
now we are having yet another discussion on the same topic of how to manage
all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

That's not really related to the discussion here tho? The debated
problem is things that have already been committed that have have bugs,
preventing beta/rc releases from being made. In some cases in the last
releases such bugs were in features integrated very early in the cycle.

Sorry if I was causing noise. I was under the impression that the
discussion was more than just things that already have bugs, but also
how the commitfests were working (scheduling etc..). I admit, I may have
grabbed your comment out of an unrelated portion of the thread.

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 12:18:28 -0500, Tom Lane wrote:

I'm not really sure why we've allowed CFs to drift on, though. Can't we
just arbitrarily decree them closed on the last day of the month? And
push unfinished work to the next one? Admittedly, this probably doesn't
work for the last CF of a release cycle, but that one's always been a
special case.

I think the problem there is the assumption that each CF entry deserves
some amount of feedback. It's not a big problem to close entries of
patches that have gotten a fair amount, but it's pretty common to have a
set of entries that have barely gotten any review. Usually either
because they're perceived as too boring (fair enough), because they're
too complex for the majority of non-busy people (uhhh), or because
they're seen as claimed.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 6:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Magnus Hagander <magnus@hagander.net> writes:

On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6

I agree that it's not the same problem. I do believe that it is *a*

problem

though, and a fairly significant one too. Because there's *never* any
downtime from CF mode, regardless of where in the cycle we are.

True, we've been failing badly on the intention that there would be time
off from CF mode, and I'd like to see a fix for that. I do not think it's
directly related to the can't-get-a-release-out problem.

In a way you could say they are two symptoms of the same underlying
problem, being that we've partially lost control over our development and
release schedule.

I'm not really sure why we've allowed CFs to drift on, though. Can't we
just arbitrarily decree them closed on the last day of the month? And
push unfinished work to the next one? Admittedly, this probably doesn't
work for the last CF of a release cycle, but that one's always been a
special case.

That's pretty much what I suggested :)

Except that we need to do it for the last one as well. With the only
exception that the last one might be a bit longer. But the fact is that the
recent of CFs *and* releases, we've taken the approach of closing the CF
when everything in it is done or explicitly reviewed-and-bumped, and tried
really really hard not to bump things because nobody had time to look at
them. That's what I'm suggesting we change, and actually just cut them.
Yes, one of the reasons for the CFs was to allow people a fair chance to
get reviewed.But given that there isn't actually a deadline in practice
doesn't help with that.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

I admit, I may have grabbed your comment out of an unrelated portion
of the thread.

Ceterum censeo Carthaginem esse delendam.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

Good point, totally agree.

So the project should try to think of new ideas on how to incentivise reviewing?

I have three ideas so far:

1. The way I see it, the honor system is based on being mentioned
in the commit message and the release notes.

Authors are always mentioned in the release notes,
but I see reviewers are mostly only mentioned in the commit messages.

Maybe more skilled developers would think it's cool to do reviewing
if they were "paid" by also being mentioned in the release notes?

At least some skilled people would probably be motivated by it,
in addition to the good feeling of doing something just because it's
fun or important.

2. Currently "Needs review" can mean a patch is in any of the phases
(Submission -> Usability -> Feature test -> Performance review ->
Coding review -> Architecture review -> Review review).
If you as a reviewer don't even know if the patch will be accepted and
committed by a committer,
there is a risk review-time will be wasted until the patch has been
accepted by the community.

I suggest we inject a new intermediate optional step "Needs Code
Review" to mean the feature has been discussed on pghackers
and there is a consensus among committers that they at least agree the
feature is something desired,
and that someone has at least reviewed the patch applies and the
feature works like expected.
And maybe renaming the "Needs review" step to "Needs Usability Review"
(or some other word to capture the Submission -> Usability -> Feature
test -> Performance review, phase before the Code review phase).

Then you as a reviewer would be confident the feature will get
accepted as long as the code is correct,
and the time you spent on code-reviewing will not be wasted.

3. For those who are partly motivated by money,
maybe this idea would work too to some extent:

Since Trustly's Bug Country program [1]https://trustly.com/se/postgresql-bug-bounty/ is aimed at motivating people
to find bugs in HEAD hasn't produced a single report so far,
maybe we should extend it to also cover reviews of commits
marked as "Ready For Committer" that shed light on a problem
that could have caused data corruption,
preventing a bad commit from being committed.

If we suddenly get hundreds of new reviewers who find hundreds of bugs
in uncommitted code, then maybe we have to change the terms again,
but I doubt neither of those will happen.

If people think this is a good idea, I can discuss internally if we can change
the conditions of the bug country program accordingly.

[1]: https://trustly.com/se/postgresql-bug-bounty/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Magnus Hagander <magnus@hagander.net> writes:

That's pretty much what I suggested :)

Except that we need to do it for the last one as well. With the only
exception that the last one might be a bit longer. But the fact is that the
recent of CFs *and* releases, we've taken the approach of closing the CF
when everything in it is done or explicitly reviewed-and-bumped, and tried
really really hard not to bump things because nobody had time to look at
them. That's what I'm suggesting we change, and actually just cut them.
Yes, one of the reasons for the CFs was to allow people a fair chance to
get reviewed.But given that there isn't actually a deadline in practice
doesn't help with that.

Yeah. It's certainly unfair if someone's patch doesn't get reviewed,
but there are only 24 hours in a day, and we have a limited pool of
reviewer and committer manpower. I think we just have to say that
sometimes life is unfair.

I think it might also be a good idea if we could somehow distinguish
"nobody had time for that yet" from "nobody is interested", with an eye
to flat-out rejecting patches that no one cares enough about to review.
Maybe we could reduce the workload by inventing some kind of up-front
filter whereby a patch isn't even a candidate to get reviewed unless, say,
at least two people besides the author say "this sounds like it's worth
pursuing".

One of the other things I do not like about the current CF process is that
it's created a default assumption that most submitted patches should get
accepted eventually. It is *very* hard to reject a patch altogether in
the CF process: you more or less have to show cause why it should not get
accepted. That default is backwards. Maybe this isn't the process' fault
exactly, but it sure seems like that's how we approach patches now.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 09:31 AM, Joel Jacobson wrote:

On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

Good point, totally agree.

So the project should try to think of new ideas on how to incentivise reviewing?

I have three ideas so far:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 09:42:28 -0800, Joshua D. Drake wrote:

Don't review patches? Don't submit patches.

Absolutely. While I think encouraging reviewers is a good thing, it
seems independent of requiring contributors to do quid-pro-quo
reviews. There'll always be people too busy or uninterested in reviewing
riding free on others work otherwise.

Obviously a drive by bugfix is something different than an actual new
feature.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Jan 20, 2016, at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other patches.

And there will often be patches that generate so little interest that they’ll never be reviewed no matter how many other patches the submitter reviews.

That said, it’s not a bad heuristic, and I suspect that someone who reviews patches is more likely to get their patch reviewed. But obviously there are no guarantees.

Best,

David

Magnus Hagander wrote:

Except that we need to do it for the last one as well. With the only
exception that the last one might be a bit longer. But the fact is that the
recent of CFs *and* releases, we've taken the approach of closing the CF
when everything in it is done or explicitly reviewed-and-bumped, and tried
really really hard not to bump things because nobody had time to look at
them. That's what I'm suggesting we change, and actually just cut them.
Yes, one of the reasons for the CFs was to allow people a fair chance to
get reviewed.But given that there isn't actually a deadline in practice
doesn't help with that.

We have a rule in the commitfest process, which I mentioned in some
other thread yesterday, which we have never enforced as far as I
remember -- which is that the last commitfest only accepts patches that
have already been submitted to previous commitfests. If we were to
enforce that one, I think we put a limit to the size of the last
commitfest, reducing the amount of work necessary to close it.

I definitely think we should be stricter about closing each commitfest
on the last day of the month. To improve on the fairness front we could
also be stricter about giving priority to reviewing patches that have
been punted from previous commitfest.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 09:48:24 -0800, David E. Wheeler wrote:

There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other patches.

I think that's actually getting less and less true. By now the really
desirable-enough features imply so much work by reviewer and committers,
over a prolonged time, that they're imo unlikely to be picked up just
because they're desirable, even when the author doesn't play by the
rules. I don't think an exceptional case or two is particularly
important.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 09:48 AM, David E. Wheeler wrote:

On Jan 20, 2016, at 9:42 AM, Joshua D. Drake <jd@commandprompt.com> wrote:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other patches.

Correct.

And there will often be patches that generate so little interest that they’ll never be reviewed no matter how many other patches the submitter reviews.

Correct.

That said, it’s not a bad heuristic, and I suspect that someone who reviews patches is more likely to get their patch reviewed. But obviously there are no guarantees.

Well a good portion of humanity is built on karma. I do for you, you do
for me. It isn't explicit, that is just bad taste but there is an
implicit desire that gets built up to help people when you are also
being helped.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Joshua D. Drake wrote:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

Here's one area where the commitfest app could help the CFM. I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Jan 20, 2016 18:57, "Alvaro Herrera" <alvherre@2ndquadrant.com> wrote:

Joshua D. Drake wrote:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

Here's one area where the commitfest app could help the CFM. I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

I assume you mean a summary review? You can already use the filter feature
to get this information if you want to look at an individual contributor,
but I'm guessing you're looking for a global overview?

And you want the actual patches, and not just a count?

/Magnus

Magnus Hagander wrote:

On Jan 20, 2016 18:57, "Alvaro Herrera" <alvherre@2ndquadrant.com> wrote:

Here's one area where the commitfest app could help the CFM. I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

I assume you mean a summary review? You can already use the filter feature
to get this information if you want to look at an individual contributor,
but I'm guessing you're looking for a global overview?

Yes, a summary. I tried using the filters but it's too cumbersome.

And you want the actual patches, and not just a count?

Yes, the actual patches, so that I can go and see the threads.
Otherwise it's easy to mark oneself as reviewer and not actually review
anything ...

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 19:00:05 +0100, Magnus Hagander wrote:

And you want the actual patches, and not just a count?

I think the actual patches makes sense, because reviewing one 200KB
patch obviously is something different than reviewing 3 onelinesrs.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund <andres@anarazel.de> writes:

On 2016-01-20 09:48:24 -0800, David E. Wheeler wrote:

There will always be patches desirable-enough that they will be reviewed whether or not the submitter reviewed other patches.

I think that's actually getting less and less true. By now the really
desirable-enough features imply so much work by reviewer and committers,
over a prolonged time, that they're imo unlikely to be picked up just
because they're desirable, even when the author doesn't play by the
rules. I don't think an exceptional case or two is particularly
important.

Thinking about that crystallized something for me that maybe is worth
considering. There is a difference between big patches (say, the
size of the GROUPING SETS one) and little patches (say, most of the
tab-completion fixes we get). We're currently trying to apply the
same process and criteria to both cases, and I'm thinking that that is
a bad idea. I am not sure what exactly ought to be different about
them, but probably something should. I think for small patches, we are
using the CF app mostly to be sure things don't fall through the cracks,
but maybe we don't need the whole process otherwise.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 12:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

You're assuming that the people who are hot to get started on the next
release are different from the people who don't have time to fix the bugs
they introduced previously. IME it's mostly the same people.

The last common commit between master and REL9_5_STABLE is dated June
30th. Here is the open items list as of that date:

https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&amp;oldid=25373

The only thing on that list that I had anything to do with is "Foreign
join pushdown vs EvalPlanQual". And that only would have mattered to
people who want to do out-of-core FDW join pushdown against 9.5 and
aren't comfortable restricting the feature to queries without
rowmarks, which is probably nobody, and "PGXS "check" target forcing
an install ?", which was a very minor issue that I didn't create but
did help fix. Similarly, I see nothing on that list that could be
attributed to, say, you.

I hate to name names, but it seems to me that where things came off
the rails for 9.5 is that (1) the row-level security feature was
absolutely riddled with bugs and that those bugs were not fixed in
anything like timely fashion; (2) the commit timestamp stuff was
pretty badly broken, too; and (3) Andres's multixact reworks landed
quite late and IMHO were not safe enough to back-patch, yet they
needed to be in the release. In my view, the last major fix for (1)
was in 7d8db3e8f37aec9d252353904e77381a18a2fa9f on September 30th, for
(2) was in commit 69e7235c93e2965cc0e17186bd044e4c54997c19 on December
11th, and for (3) was in aa29c1ccd9f785f9365809f5133e5491acc7ae53 on
September 26th. I think the row-security stuff is particularly bad
because that feature was originally committed on September 19, *2014*.
It's not good when you commit a security feature and are still
redefining the way that the privilege system interacts with that
feature a year later.

Also note the dates. We put out beta1 in the beginning of October,
right after those major fixes for (1) and (3) in late September. And
we put out rc1 just after (2) got patched up. There is of course a
chicken and egg problem here, because it's unclear to what degree the
commits drove the wrap date and to what degree the wrap date drove the
commits, but if we can't get fixes for those kinds of issues into the
tree on time, we are not going to release on time. Now maybe if we
refuse to branch until we get these kinds of issues fixed, we'll fix
them sooner (or revert the patches that caused the problems in the
first place, which is another way to go). That would be a positive
development. But if we end up waiting just as long but with nothing
substantial getting committed in the meantime, that will be awful.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 13:03:05 -0500, Tom Lane wrote:

I am not sure what exactly ought to be different about them [small
patches], but probably something should. I think for small patches,
we are using the CF app mostly to be sure things don't fall through
the cracks, but maybe we don't need the whole process otherwise.

Aren't we already treating them differently on an ad-hoc basis? Several
committers fast-track such patches through the process. We could mark
them differently in the CF app, but I'm not sure what we'd want to
achieve by doing that. Such small patches are frequently enough
buggy. Committers can most of the time can easily polish them up, but
that still takes time; so I think tests/reviews of those are still
worthwhile.

So the only real difference I see an explicit difference would make is
making it easier to spot such patches when looking for an easy thing to
commit, and to make life for the commitfest manager easier. Not sure if
such a labeling would save more time than doing the labeling itself
would cost.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 13:13:29 -0500, Robert Haas wrote:

(3) Andres's multixact reworks landed quite late and IMHO were not
safe enough to back-patch, yet they needed to be in the release. In
my view, the last major fix for

At least in that case I was, for a long while, basically hoping for more
input, probably unwisely so. I think the 9.4 jsonb compressability thing
essentially was in a somewhat similar state for months; the proposed
patch was mostly there, but we didn't press ahead. I think deadlines
and multiple people being blocked help tremendously with such issues.
They're often too complex for an individual and/or delayed due to lack
of concensus.

But if we end up waiting just as long but with nothing substantial
getting committed in the meantime, that will be awful.

That's true. Which I think means that we'll need to be agressive in
threatening reverts in such cases. That wouldn't have worked for 3), but
in the other two cases it'd have been doable.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 1:29 PM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-20 13:13:29 -0500, Robert Haas wrote:

(3) Andres's multixact reworks landed quite late and IMHO were not
safe enough to back-patch, yet they needed to be in the release. In
my view, the last major fix for

At least in that case I was, for a long while, basically hoping for more
input, probably unwisely so. I think the 9.4 jsonb compressability thing
essentially was in a somewhat similar state for months; the proposed
patch was mostly there, but we didn't press ahead. I think deadlines
and multiple people being blocked help tremendously with such issues.
They're often too complex for an individual and/or delayed due to lack
of concensus.

True.

But if we end up waiting just as long but with nothing substantial
getting committed in the meantime, that will be awful.

That's true. Which I think means that we'll need to be agressive in
threatening reverts in such cases. That wouldn't have worked for 3), but
in the other two cases it'd have been doable.

Yes. One problem is that it's very hard to get a patch reverted. Tom
opined that too often patches roll along toward commit and that it's
too hard to say "no, we want that". But that's nothing compared to
how hard it is to get a patch reverted. It's true that, if Tom calls
for the revert, it more often than not happens, but nobody else can
really pull that off without a major war.

And, you know, I did my time fighting major wars to try to compress
the release schedule, and honestly, it wasn't that much fun. The
process we have now is imperfect in many ways, but I no longer have
abuse heaped on me for wanting to boot a patch out of a CommitFest.
That may be bad for the project, but it's spared me a lot of grief
personally. I think that many other people are in the same situation.
Everybody would like somebody else to be the schedule enforcer ...
unless they have something that they still want to get in, in which
case they would like nobody to do it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 20 January 2016 at 15:40, Bruce Momjian <bruce@momjian.us> wrote:

Many people where happy with our consistent releasing major releases in
September, e.g. 9.0 to 9.3:

What is the point in having a special mailing list to discuss the release
schedule plus a face-to-face dev meeting to discuss this if we are going to
discuss it here?

ISTM the wrong starting point to discuss plans in an unplanned way and
assume that everyone has time to take part today, right now.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 20 January 2016 at 15:55, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de>
wrote:

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible.

+1

But this never works, because there are
always patches we want to slip in late.

I haven't seen that in the last few years, apart from maybe JSONB. Our
views on the state of patches differ, mainly because we've not all reviewed
every patch, so its hard to say whether a commit has been brewing for ages
and is right, or not.

Some bad stuff happened in 9.3 and we were scared to release early. Let's
get back on track and release on time (Sept).

The main problem is the length of the integration phase, which is mostly
where nothing happens. We need to manage that process just as we do with
CFs.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 20 January 2016 at 16:29, Andres Freund <andres@anarazel.de> wrote:

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

It failed because its never mentioned or even attempted to be enforced.

Some way to balance submissions against reviews is now essential.

If we believe in "peer review", we need peers that review.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 20 January 2016 at 17:03, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

You're assuming that the people who are hot to get started on the next
release are different from the people who don't have time to fix the bugs
they introduced previously. IME it's mostly the same people.

That sounds like all people who wanted to start the next release had bugs
that needed fixing. That was definitely not the case for 9.5.

I don't think we can allow the release to be slowed down by people that
need to perform actions and yet aren't available.

Ultimately, we should decide to simply turn off that feature and release
anyway.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:

What is the point in having a special mailing list to discuss the release
schedule plus a face-to-face dev meeting to discuss this if we are going to
discuss it here?

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:13 AM, Andres Freund <andres@anarazel.de> wrote:

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

I am not on the release list.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

andres@anarazel.de (Andres Freund) writes:

On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:

What is the point in having a special mailing list to discuss the release
schedule plus a face-to-face dev meeting to discuss this if we are going to
discuss it here?

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

In particular, we have several important people who won't be in Brussels
next week, so it's appropriate to hear their thoughts beforehand ...

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 2:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

andres@anarazel.de (Andres Freund) writes:

On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:

What is the point in having a special mailing list to discuss the release
schedule plus a face-to-face dev meeting to discuss this if we are going to
discuss it here?

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

In particular, we have several important people who won't be in Brussels
next week, so it's appropriate to hear their thoughts beforehand ...

As somebody who definitely won't be there, +1 from me. :-)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 20 January 2016 at 15:40, Bruce Momjian <bruce@momjian.us> wrote:

Many people were happy with our consistent releasing major releases in
September, e.g. 9.0 to 9.3:

9.5 2016-01-07
9.4 2014-12-18
9.3 2013-09-09 <--
9.2 2012-09-10 <--
9.1 2011-09-12 <--
9.0 2010-09-20 <--
8.4 2009-07-01
8.3 2008-02-04
8.2 2006-12-05
8.1 2005-11-08
8.0 2005-01-19

We have gotten off of that cycle in the last two major releases

Yes we have and I agree that it would be useful to return to it.

For 9.6 we have a commitfest starting in March 1. This will
never allow a September release.

If that is the case, then its too late to change.

March - last CF
April - integration
May - release Beta
Sept - release Prod

It seems perfectly OK for me to have Beta start at beginning May and for us
to release in September.

If the process takes more than 5 months (Apr - Sept) then there's something
wrong with that process, rather than there being a problem with the
schedule.

Our current 9.5/9.6 timing looks more like the 8.X series of release
dates. Everyone might be fine with that, but we had better be prepared
for November-February major release dates going forward.

I don't mind what month we pick, as long as we stick to the schedule.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 20 January 2016 at 19:45, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Jan 20, 2016 at 2:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

andres@anarazel.de (Andres Freund) writes:

On 2016-01-20 18:53:54 +0000, Simon Riggs wrote:

What is the point in having a special mailing list to discuss the

release

schedule plus a face-to-face dev meeting to discuss this if we are

going to

discuss it here?

That list exists to discuss concrete releases, and is separate so we
e.g. can mention there's security issues and such. This topic in
contrast quite validly merits input from more then the usual suspects
going to a developer meeting.

In particular, we have several important people who won't be in Brussels
next week, so it's appropriate to hear their thoughts beforehand ...

As somebody who definitely won't be there, +1 from me. :-)

OK, sorry, I forgot for a moment that you aren't going to be there.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 21/01/16 06:40, Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

That's pretty much what I suggested :)
Except that we need to do it for the last one as well. With the only
exception that the last one might be a bit longer. But the fact is that the
recent of CFs *and* releases, we've taken the approach of closing the CF
when everything in it is done or explicitly reviewed-and-bumped, and tried
really really hard not to bump things because nobody had time to look at
them. That's what I'm suggesting we change, and actually just cut them.
Yes, one of the reasons for the CFs was to allow people a fair chance to
get reviewed.But given that there isn't actually a deadline in practice
doesn't help with that.

Yeah. It's certainly unfair if someone's patch doesn't get reviewed,
but there are only 24 hours in a day, and we have a limited pool of
reviewer and committer manpower. I think we just have to say that
sometimes life is unfair.

I think it might also be a good idea if we could somehow distinguish
"nobody had time for that yet" from "nobody is interested", with an eye
to flat-out rejecting patches that no one cares enough about to review.
Maybe we could reduce the workload by inventing some kind of up-front
filter whereby a patch isn't even a candidate to get reviewed unless, say,
at least two people besides the author say "this sounds like it's worth
pursuing".

One of the other things I do not like about the current CF process is that
it's created a default assumption that most submitted patches should get
accepted eventually. It is *very* hard to reject a patch altogether in
the CF process: you more or less have to show cause why it should not get
accepted. That default is backwards. Maybe this isn't the process' fault
exactly, but it sure seems like that's how we approach patches now.

regards, tom lane

Possibly the emphasis should be on what patches should be ACCEPTED,
rather than on what patches should be REJECTED?

Then there is less stigma in a patch missing out, and people don't have
justify rejecting patches.

Cheers,
Gavin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 10:53 AM, Simon Riggs wrote:

On 20 January 2016 at 15:40, Bruce Momjian <bruce@momjian.us
<mailto:bruce@momjian.us>> wrote:

Many people where happy with our consistent releasing major releases in
September, e.g. 9.0 to 9.3:

What is the point in having a special mailing list to discuss the
release schedule plus a face-to-face dev meeting to discuss this if we
are going to discuss it here?

Because, like core although they (may) have the final word, taking solid
feedback from people not on that committee or having that discussion
publicly lends itself to:

* Community
* Transparency
* The Open Source way

ISTM the wrong starting point to discuss plans in an unplanned way and
assume that everyone has time to take part today, right now.

If it is important to them, they will make time. Everybody has a choice.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 20 January 2016 at 20:29, Joshua D. Drake <jd@commandprompt.com> wrote:

On 01/20/2016 10:53 AM, Simon Riggs wrote:

On 20 January 2016 at 15:40, Bruce Momjian <bruce@momjian.us
<mailto:bruce@momjian.us>> wrote:

Many people where happy with our consistent releasing major releases
in
September, e.g. 9.0 to 9.3:

What is the point in having a special mailing list to discuss the
release schedule plus a face-to-face dev meeting to discuss this if we
are going to discuss it here?

Because, like core although they (may) have the final word, taking solid
feedback from people not on that committee or having that discussion
publicly lends itself to:

* Community
* Transparency
* The Open Source way

I'm in agreement with your points.

It's clear I'd placed too much emphasis on the meeting and the time
reserved for it, not realising some key people would be excluded and the
effects that might have. I withdraw my earlier comments and apologize.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jan 20, 2016 at 6:57 PM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Joshua D. Drake wrote:

4. Submit a patch, review a patch.

Don't review patches? Don't submit patches.

Here's one area where the commitfest app could help the CFM. I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

Report added. You will find a new "Reports" button at the bottom of the CF
(as a CF admin) which has a link to it.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 1/20/16 1:44 PM, Robert Haas wrote:

And, you know, I did my time fighting major wars to try to compress
the release schedule, and honestly, it wasn't that much fun. The
process we have now is imperfect in many ways, but I no longer have
abuse heaped on me for wanting to boot a patch out of a CommitFest.
That may be bad for the project, but it's spared me a lot of grief
personally. I think that many other people are in the same situation.
Everybody would like somebody else to be the schedule enforcer ...
unless they have something that they still want to get in, in which
case they would like nobody to do it.

Yeah, a lot of the ideas in this thread, while reasonable, are of the
sort "We need to be better about ..." which sounds a lot like "I need to
be better about exercising". A system based purely on distributed
willpower isn't going to last very long, as we are finding out.

Sure, I'd like more than one party to review my patches, and I'd like to
spend more time doing additional reviews on other people's patches. But
I can barely keep up as it is. I know we need code reviews, but I think
it's never going to work well to the point that we are satisfied with
it. Just look around the world, in software, open or commercial, or
even academics, and tell me who has peer reviews figured out.

My feeble attempts at alleviating this problem a bit are adding more
testing and removing outdated functionality, but both of those are also
incredibly abuse-prone activities.

FWIW, I do apologize for proposing the fifth commitfest (PGCon 2014).

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Magnus Hagander wrote:

On Wed, Jan 20, 2016 at 6:57 PM, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:

Here's one area where the commitfest app could help the CFM. I would
like to have a report that shows, for each person, the list of patches
where they are author, and the list of patches where they are reviewer.
Then I can check his reviewer score by skimming a couple of threads and
say "you don't have enough author credits, go away".

Report added. You will find a new "Reports" button at the bottom of the CF
(as a CF admin) which has a link to it.

Excellent, this is very useful, thanks.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 1:37 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

Yeah, a lot of the ideas in this thread, while reasonable, are of the
sort "We need to be better about ..." which sounds a lot like "I need to
be better about exercising". A system based purely on distributed
willpower isn't going to last very long, as we are finding out.

I have often thought that contributors have a tendency to be
unrealistic about how effective procedural changes can be in
alleviating our problems. We have always relied heavily on individual
acts of heroism. I expect no great changes there.

For example, you can't really expect a CF app SQL query to give you
anything greater than an extremely noisy idea of how much "review
goodwill" somebody has banked. My oldest pending patch in the CF app
is a boring refactoring patch that no one asked me to write. Should
that be held against me?

My feeble attempts at alleviating this problem a bit are adding more
testing and removing outdated functionality, but both of those are also
incredibly abuse-prone activities.

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".
There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 09:12:07AM -0800, Joshua Drake wrote:

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

I understand but I think we are missing out on an opportunity here.
Notice that the shorter release cycle for STS will actually make
some things easier. Including:

* Increased test base (just like Fedora/Ubuntu)
* Increased early adopter testing (that is what early adopting is
really about for us anyway)
* Decreased concerns about upgrades and ability to extend upgrade status.

I can see LTS working for plugin change, but not server binary changes.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:

My feeble attempts at alleviating this problem a bit are adding more
testing and removing outdated functionality, but both of those are also
incredibly abuse-prone activities.

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".

-------------------------

There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

I do think that is effectively a lesson we learned, wrong or not.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 07:48:07PM +0000, Simon Riggs wrote:

If that is the case, then its too late to change.

March - last CF
April - integration
May - release Beta
Sept - release Prod

It seems perfectly OK for me to have Beta start at beginning May and for us to
release in September.

If the process takes more than 5 months (Apr - Sept) then there's something
wrong with that process, rather than there being a problem with the schedule.

Well, consider the last commitfest finishes at the end of April, I don't
se how we could do integration and get to beta in May --- at least we
have never been able to do that.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 06:31:58PM +0100, Joel Jacobson wrote:

On Wed, Jan 20, 2016 at 5:29 PM, Andres Freund <andres@anarazel.de> wrote:

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

Good point, totally agree.

So the project should try to think of new ideas on how to incentivise reviewing?

I have three ideas so far:

1. The way I see it, the honor system is based on being mentioned
in the commit message and the release notes.

Authors are always mentioned in the release notes,
but I see reviewers are mostly only mentioned in the commit messages.

Maybe more skilled developers would think it's cool to do reviewing
if they were "paid" by also being mentioned in the release notes?

At least some skilled people would probably be motivated by it,
in addition to the good feeling of doing something just because it's
fun or important.

FYI, the fact that feature authors appear in the release notes is an
artifact of how we track who wrote which stuff, and is not designed for
rewarding, though it has that effect. If we were to expand that to
cover reviewers, we would then be overburdinging that system and we
would probably end up removing all names from the release notes.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian <bruce@momjian.us> writes:

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".

-------------------------

There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

I do think that is effectively a lesson we learned, wrong or not.

I think a lesson we *should* have learned by now is that we need to put
more emphasis on testing. That includes not only spending more time on
it, but investing more in testing infrastructure. The buildfarm has been
a huge advance in our ability to find/fix portability issues quickly, but
it does nothing to, say, assess crash safety. Or identify performance
regressions.

As a concrete example, I recall that Heikki or someone had a tool for
checking WAL replay by comparing master and slave disk contents. We
should make an effort to get that into a state where anyone can use it.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 05:56:27PM -0500, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".

-------------------------

There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

I do think that is effectively a lesson we learned, wrong or not.

I think a lesson we *should* have learned by now is that we need to put
more emphasis on testing. That includes not only spending more time on
it, but investing more in testing infrastructure. The buildfarm has been
a huge advance in our ability to find/fix portability issues quickly, but
it does nothing to, say, assess crash safety. Or identify performance
regressions.

The two sobering blockers I can remember were multi-xact (9.3) and JSONB
compression (9.4).

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 18:02:20 -0500, Bruce Momjian wrote:

The two sobering blockers I can remember were multi-xact (9.3) and JSONB
compression (9.4).

I don't see how those compare. The multixact stuff was all discovered a
fair bit after the release, the compression stuff pre release.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 12:05:19AM +0100, Andres Freund wrote:

On 2016-01-20 18:02:20 -0500, Bruce Momjian wrote:

The two sobering blockers I can remember were multi-xact (9.3) and JSONB
compression (9.4).

I don't see how those compare. The multixact stuff was all discovered a
fair bit after the release, the compression stuff pre release.

They were both cases where we were surprised and where hasty action did
or could have caused big problems.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 3:26 PM, Bruce Momjian <bruce@momjian.us> wrote:

They were both cases where we were surprised and where hasty action did
or could have caused big problems.

IMV MulitXacts were not a special case because of how many bugs there
were in 9.3.0. Rather, they were a special case because, to the
surprise of many, the number of bugs in each point release (or their
severity) did not really taper off. There was a particular qualitative
nature to the bugs. This seems to be because the design was
excessively complicated.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 2:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I think a lesson we *should* have learned by now is that we need to put
more emphasis on testing. That includes not only spending more time on
it, but investing more in testing infrastructure. The buildfarm has been
a huge advance in our ability to find/fix portability issues quickly, but
it does nothing to, say, assess crash safety. Or identify performance
regressions.

I might add: Making testability an explicit goal of development from
the very beginning, during the design of the feature. In essence,
reducing the surface area for bugs when (not if) they happen, by being
able to easily simulate various stresses. Or better yet, making sure
that each mechanism modified has very little chance of interacting
with some other complicated mechanism in production. Things that
happen just once every 6 months on many production systems seemed to
be where MultiXacts kept giving trouble.

My pet example is how UPSERT doesn't change anything about the on-disk
representation of heap tuples, unless their xact aborted.

As a concrete example, I recall that Heikki or someone had a tool for
checking WAL replay by comparing master and slave disk contents. We
should make an effort to get that into a state where anyone can use it.

I think that that was initially Jeff Janes. Jeff also provided great
testing infrastructure for UPSERT, something that I was very grateful
for.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 8:51 AM, Peter Geoghegan <pg@heroku.com> wrote:

On Wed, Jan 20, 2016 at 2:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

As a concrete example, I recall that Heikki or someone had a tool for
checking WAL replay by comparing master and slave disk contents. We
should make an effort to get that into a state where anyone can use it.

I think that that was initially Jeff Janes. Jeff also provided great
testing infrastructure for UPSERT, something that I was very grateful
for.

You mean the full-page write consistency tool? Heikki initiated that,
a common interface to be able to mask pages and make them consistent
for comparison on the master and the standby was also part of the
deal. Another tool that we had better IMO put some efforts in porting
into core is sqlsmith, which would actually be a complete rewrite
because the upstream code is under GPL license and depends on libpqxx.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 12:59 AM, Bruce Momjian <bruce@momjian.us> wrote:

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.  We
twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

Here I agree. Five commit fests is proving to be a lot and CF managers
are facing a severe burnout. Though I guess it sounded like a fine
idea at the moment this was begun (disclaimer: I was not there). So we
may want to do back to 4, and avoid too much overlapping between CFs
and what would be a stability period, without any commit fests going
on. It seems to me that the problem regarding the fact that fixes got
committed late is that committer's, author's and reviewer's attention
are getting distracted with commit fests and the new features proposed
there. Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 1:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Thu, Jan 21, 2016 at 12:59 AM, Bruce Momjian <bruce@momjian.us> wrote:

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.  We
twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

Here I agree. Five commit fests is proving to be a lot and CF managers
are facing a severe burnout. Though I guess it sounded like a fine
idea at the moment this was begun (disclaimer: I was not there). So we
may want to do back to 4, and avoid too much overlapping between CFs
and what would be a stability period, without any commit fests going
on. It seems to me that the problem regarding the fact that fixes got
committed late is that committer's, author's and reviewer's attention
are getting distracted with commit fests and the new features proposed
there. Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.

Another idea popping to my mind in order to fix the CF manager burnout
and actually motivate people into becoming CF managers: say that one a
CF manager is done with a commit fest, folks on -hackers discuss if
the CFM has done a good job or not. If yes, he gets a travel paid to
the conference of his choice. That's assuming that there are actual
funds for that. Most people here perhaps may not care as their company
are usually willing to pay for such trip, but in some cases I am sure
that there are people who can be funded by their company *if* they do
a presentation to this conference. Just a random idea.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 8:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.

Or it might just accelerate the amount of time it takes to *declare*
having reached that milestone. Besides, do you really think that
people need to be able to commit something before seriously working on
it?

Rules that are expressly coercive are a *bad* idea.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 8:23 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Another tool that we had better IMO put some efforts in porting
into core is sqlsmith, which would actually be a complete rewrite
because the upstream code is under GPL license and depends on libpqxx.

Then Andreas Seltenreich better get a commit bit. I've seen him turn
around changes in sqlsmith to test new areas of Postgres in a couple
of days. That's a big reason why he has been so effective.

What benefit does porting sqlsmith for inclusion in core have? I can
only think of costs, including those that you mentioned.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:

On Wed, Jan 20, 2016 at 8:23 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Another tool that we had better IMO put some efforts in porting
into core is sqlsmith, which would actually be a complete rewrite
because the upstream code is under GPL license and depends on libpqxx.

Then Andreas Seltenreich better get a commit bit. I've seen him turn
around changes in sqlsmith to test new areas of Postgres in a couple
of days. That's a big reason why he has been so effective.

What benefit does porting sqlsmith for inclusion in core have? I can
only think of costs, including those that you mentioned.

We have automatic buildfarm coverage on many platforms. Perhaps we
could live without that with a buildfarm module though.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 9:37 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

We have automatic buildfarm coverage on many platforms. Perhaps we
could live without that with a buildfarm module though.

It's not clear that buildfarm coverage makes sense for sqlsmith. If it
does make sense, I see no reason why introducing it should be attached
to sqlsmith's inclusion in core.

--
Peter Geoghegan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Michael Paquier <michael.paquier@gmail.com> writes:

On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:

What benefit does porting sqlsmith for inclusion in core have? I can
only think of costs, including those that you mentioned.

We have automatic buildfarm coverage on many platforms. Perhaps we
could live without that with a buildfarm module though.

I do not think we should necessarily try to include every testing tool
in the core distribution. What is important is that they be readily
available: easy to find, easy to use, documented, portable. "Same
license as the PG core code" is not on that list.

An immediately relevant example is that the buildfarm server and client
code aren't in the core distribution, and AFAIR no one has suggested
that they need to be.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jan 21, 2016 at 10:11 AM, Michael Paquier <michael.paquier@gmail.com>
wrote:

On Thu, Jan 21, 2016 at 1:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Thu, Jan 21, 2016 at 12:59 AM, Bruce Momjian <bruce@momjian.us>

wrote:

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de>

wrote:

I think this has very little to do with commitfest schedules, and

much

more with the "early" forking of the new version branch. For both

9.4

and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.

We

twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

Here I agree. Five commit fests is proving to be a lot and CF managers
are facing a severe burnout. Though I guess it sounded like a fine
idea at the moment this was begun (disclaimer: I was not there). So we
may want to do back to 4, and avoid too much overlapping between CFs
and what would be a stability period, without any commit fests going
on. It seems to me that the problem regarding the fact that fixes got
committed late is that committer's, author's and reviewer's attention
are getting distracted with commit fests and the new features proposed
there. Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.

Another idea popping to my mind in order to fix the CF manager burnout
and actually motivate people into becoming CF managers: say that one a
CF manager is done with a commit fest, folks on -hackers discuss if
the CFM has done a good job or not. If yes, he gets a travel paid to
the conference of his choice.

I also think there should be some way to give credit to CFM, if it is
difficult to do anything related to money, then we can enforce that if
CFM submits any patches for next CF, then those should be prioritised.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Wed, Jan 20, 2016 at 4:40 PM, Bruce Momjian <bruce@momjian.us> wrote:

Many people where happy with our consistent releasing major releases in
September, e.g. 9.0 to 9.3:

Not sure why the commitfest process should be synchronized with the

release process. What if, when the release date comes, the currently
running commitfest (if there is one) gets put on hold, cleanup work is
done, release gets stamped, then commitfest gets resumed for the next
release?

On Wed, Jan 20, 2016 at 12:40:04PM -0500, Tom Lane wrote:

I think it might also be a good idea if we could somehow distinguish
"nobody had time for that yet" from "nobody is interested", with an eye
to flat-out rejecting patches that no one cares enough about to review.
Maybe we could reduce the workload by inventing some kind of up-front
filter whereby a patch isn't even a candidate to get reviewed unless, say,
at least two people besides the author say "this sounds like it's worth
pursuing".

One of the other things I do not like about the current CF process is that
it's created a default assumption that most submitted patches should get
accepted eventually. It is *very* hard to reject a patch altogether in
the CF process: you more or less have to show cause why it should not get
accepted. That default is backwards. Maybe this isn't the process' fault
exactly, but it sure seems like that's how we approach patches now.

These paragraphs point to different facets of the same problem: community
conventions make rejecting a patch a substantial project in its own right. +1
for experimenting with one or more ways to expedite that. There's your
two-ACKs idea above, and we floated[1]https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Meeting#9.6_Schedule a few more at the 2015-06-16 meeting:

This was followed by discussion of how we arrange the CFs. Noah suggested a
prioritization system. He suggested a system where various committers can
provide feedback on stuff as triage. Simon agreed and volunteered. Haas
suggested a 2-committer veto. Frost said we need a formal way to "nack"
something. We need to triage stuff earlier in the process. Haas says the big
problem is the endless arguments because we don't want to say "no" to people
because we have scarce committer time.

I recommend trying the two-ACKs plan for one CF. Keep it if it helps
markedly; otherwise, try another of these variants for the CF after that one.

Thanks,
nm

[1]: https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Meeting#9.6_Schedule

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 06:58:24PM +0000, Simon Riggs wrote:

The main problem is the length of the integration phase, which is mostly
where nothing happens.

The open items wiki page saw steady change from 30 April to 28 December[1]https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&amp;limit=500&amp;action=history;
the two longest quiet periods were an eleven-day gap from 21 August and a
nine-day gap from 16 December. Nineteen distinct contributors were finding,
fixing or otherwise editing data about 9.5 defects. The integration work is
too slow, yes, but this persistent myth of a dead period is false.

For my part, I spent 1 July to 11 December reviewing 9.5 commits and reporting
or fixing defective 9.5 work. (I did spoil myself with a few breaks fixing
pre-9.5 bugs.) I doubt the 9.5 new bugs would have run dry within the next
five months, either, but the release team picked a fair juncture to give up
and call it dot-zero.

nm

[1]: https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&amp;limit=500&amp;action=history

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/21/16 2:29 AM, Amit Kapila wrote:

I also think there should be some way to give credit to CFM, if it is
difficult to do anything related to money, then we can enforce that if
CFM submits any patches for next CF, then those should be prioritised.

Personally, I don't see why we have our scarcest resource doing what is
essentially a project management task, especially when at least one
commercial company has offered to donate paid staff time.

I don't think the last CF of 9.6 is the time to experiment, but I think
we should try using a PM for the first CF of 9.7.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/20/16 11:49 PM, Tom Lane wrote:

Michael Paquier <michael.paquier@gmail.com> writes:

On Thu, Jan 21, 2016 at 2:30 PM, Peter Geoghegan <pg@heroku.com> wrote:

What benefit does porting sqlsmith for inclusion in core have? I can
only think of costs, including those that you mentioned.

We have automatic buildfarm coverage on many platforms. Perhaps we
could live without that with a buildfarm module though.

I do not think we should necessarily try to include every testing tool
in the core distribution. What is important is that they be readily
available: easy to find, easy to use, documented, portable. "Same
license as the PG core code" is not on that list.

An immediately relevant example is that the buildfarm server and client
code aren't in the core distribution, and AFAIR no one has suggested
that they need to be.

Right. What I think would be far more useful is making it easier to
explicitly test things (better tools + design for test), and something
akin to buildfarm that will run automated testing on submitted patches.

Put another way: it's stupid that we even ask reviewers to waste time
running make check. That can be automated. Ideally reviewers shouldn't
be doing any testing, because the tests that are part of the patch
should answer every question they would have, but I don't see that
happening until we have a separate automation-only target that we don't
care how long it takes to run.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/20/16 11:40 AM, Tom Lane wrote:

Yeah. It's certainly unfair if someone's patch doesn't get reviewed,
but there are only 24 hours in a day, and we have a limited pool of
reviewer and committer manpower. I think we just have to say that
sometimes life is unfair.

I think that's a great way to ensure we shrink the pool of reviewers
when someone works on a patch and then it goes nowhere. I find it rather
difficult to get feedback on ideas before I spend the time to code
something, it's got to be even worse for someone the community doesn't
know. So if we're going to do this, I think there must be a mechanism
for a patch idea/design to be approved.

I think we also need to be careful about -hackers being the only place
feature desirability is measured. There's an entire world of users out
there that aren't even on -general. If some feature doesn't really
interest -hackers but there's 50 users that want it and someone willing
to work on it, ISTM we should make efforts to get it committed.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:46 PM, Peter Geoghegan <pg@heroku.com> wrote:

On Wed, Jan 20, 2016 at 8:32 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

Perhaps some people are more interested in implementing new
features than working on bugs and would just continue hacking and
arguing about new features, at least a stability period may attract
more committer attention into actual bug fixes, in short: no new
features can be committed until the previous versions has reached at
least beta2, rc, whatever. This may accelerate the stability process.

Or it might just accelerate the amount of time it takes to *declare*
having reached that milestone.

Yes. I think that's been a problem already, and closing development
for prolonged periods of time will make it worse.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/20/16 4:29 PM, Bruce Momjian wrote:

On Wed, Jan 20, 2016 at 09:12:07AM -0800, Joshua Drake wrote:

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

I understand but I think we are missing out on an opportunity here.
Notice that the shorter release cycle for STS will actually make
some things easier. Including:

* Increased test base (just like Fedora/Ubuntu)
* Increased early adopter testing (that is what early adopting is
really about for us anyway)
* Decreased concerns about upgrades and ability to extend upgrade status.

I can see LTS working for plugin change, but not server binary changes.

s/LTS/STS/?

In any case, I think JD is onto something here. As someone that focuses
more on user experience than "deep core" code, I already find yearly
releases to be quite inconvenient. It's hard to find the motivation to
make a minor improvement in something (especially knowing how hard it
will be to get the patch approved) knowing that it won't see the light
of day for a year, and realistically I won't be able to use it with any
clients that are in production for 2-3 years.

Given the high level of extensibility that we have, maybe it would be
good to logically segregate stuff into things that are deeply embedded
in the "core" code (ie: on-disk format) from things that are much easier
to change when necessary (like add-on functions or PLs). Things like new
JSON operators could be released much more rapidly that way.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian wrote:

FYI, the fact that feature authors appear in the release notes is an
artifact of how we track who wrote which stuff, and is not designed for
rewarding, though it has that effect.

I think you can claim this all you want, but I'd be surprised if anyone
actually believed it. You're saying that we do this so that we can
properly attribute code to developers so that we can pester the right
guy when things go wrong; but in reality when we investigate a bug we
never look at the release notes; what we do is look at the Git logs,
which already contain this information.

To the public at large I'm pretty sure having the names there is a
reward for having developed stuff, not a blame sign. Do real users go
back to 9.3 to see that "oh this Alverro dude wrote multixacts which had
so many bugs, we should really hang him by the bollocks"? I don't think
so. People like Robert, Andres, Noah are more likely to want me hung
because they worked so hard to get the bugs fixed.

If we were to expand that to
cover reviewers, we would then be overburdinging that system and we
would probably end up removing all names from the release notes.

To me, this reads just as a threat that "if you disturb the waters here
I will just remove all names and you will all be the poorer for it."
Please don't do that, it looks rude.

I think it is a disservice to the project to deny the reviewers the bit
of coolness factor derived from having their names in the release notes
-- or maybe just *somewhere*. If you want to remove the developer names
from the release notes to avoid bloating them, perhaps what we can do is
put names of authors and reviewers in a page in the website (probably
not the wiki, though, because it's easy to vandalize that and many
people may have concerns regarding the legitimacy of anyone listed
there.)

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Jan 22, 2016 at 10:43 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:

If we were to expand that to
cover reviewers, we would then be overburdinging that system and we
would probably end up removing all names from the release notes.

To me, this reads just as a threat that "if you disturb the waters here
I will just remove all names and you will all be the poorer for it."
Please don't do that, it looks rude.

For my part, I am not sure the names in the release notes are actually
all that helpful. What does and does not get credited there and how
it gets credited is more random than I would like. And it's often
hard to draw the line as to who contributed enough to deserve credit
and who did not. In commit messages you can try to spell out the
details, but as Bruce says, there's no room for that in the release
notes. If a feature is credited as (Robert Haas, X, Y, Z) it's clear
that I'm the principle author, but it's unclear whether I did 40% of
the work or 90% of the work. If it's credited as (X, Robert Haas),
that could mean X came up with the idea and wrote a patch which I then
completely re-wrote from scratch using a different design and
committed; or it could mean X came with an idea that was basically
good and I wrote and debugged a 1000-line patch, and I rewrote 10% of
it for some reason before committing. So even today, it's pretty hard
to use the release notes as a way to measure depth of contribution.
The problem is worse for people most of whose work is reviewing and
committing rather than authoring, but if you put everybody's name on
everything they reviewed, that wouldn't fix it. Instead, you'd tend
to overvalue people who do a lot of shallow reviews relative to people
who do a few very deep ones. I think we just have to resign ourselves
to the fact that judging depth of contribution from the release notes,
or even the commit log, is going to be rather difficult; and that
different people will have different opinions about whose work is most
meritorious.

In other words, I fear that putting names in the release notes is just
politicizing a process that ought to be about presenting the
community's work in the best way possible. A new release ought to be
about putting the whole community forward in the best light.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:

Ideally reviewers shouldn't be doing any testing, because the tests
that are part of the patch should answer every question they would
have, but I don't see that happening until we have a separate
automation-only target that we don't care how long it takes to run.

I think that's completely wrong.

Yes, more tests are good, and we need a place for longer running
tests. But assuming that every patch author will create a testsuite that
covers every angle is just about akin to assuming every submitter will
deliver perfect, bug free code. And we know how well that turns out.

I think actively trying to break a feature, and postgres in general, is
one of the most important tasks of reviewers and testers. And with that
I don't mean trying to run "make check". Look e.g. at the tests Jeff
Janes has performed, what the recent plug tests of Tomas Vondra brought
to light, or at what the full page write checker tool of Heikki's
showed.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 22 January 2016 at 16:34, Robert Haas <robertmhaas@gmail.com> wrote:

For my part, I am not sure the names in the release notes are actually
all that helpful.

It has one important effect of current interest: establishing the truth
that multiple people and multiple companies are involved in producing and
maintaining PostgreSQL. Whether the names are properly attributed will
always be a time-consuming task, but I will oppose any attempt to remove or
obscure evidence of who develops PostgreSQL, wherever that occurs.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-01-22 08:18:45 -0600, Jim Nasby wrote:

Personally, I don't see why we have our scarcest resource doing what is
essentially a project management task, especially when at least one
commercial company has offered to donate paid staff time.

Because so far all CFs that weren't managed by somebody involved on the
code level worked even less than the rest. You need to be able to judge
how complex and ready a patch is to some degree. You need to know who to
prod for a specific patch.

Yes, a PM can learn to do that. But it's not something a freshly hired
person can do. You need a fair bit of community involvement.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-22 08:50:15 -0600, Jim Nasby wrote:

I think that's a great way to ensure we shrink the pool of reviewers when
someone works on a patch and then it goes nowhere.

True, it really sucks. But what's your counter proposal? Commitfests
dragging on forever, and people burning out on continually feeling they
need to work on them? Hasn't worked out well.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Jan 22, 2016 at 7:19 PM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-22 08:18:45 -0600, Jim Nasby wrote:

Personally, I don't see why we have our scarcest resource doing what is
essentially a project management task, especially when at least one
commercial company has offered to donate paid staff time.

Because so far all CFs that weren't managed by somebody involved on the
code level worked even less than the rest. You need to be able to judge
how complex and ready a patch is to some degree. You need to know who to
prod for a specific patch.

Yes, a PM can learn to do that. But it's not something a freshly hired
person can do. You need a fair bit of community involvement.

I definitely agree.

However, if someone were to put forward a person who at least partially
qualifies towards that I'm all for giving it a try at least once. But so
far I don't recall seeing any actual propsal of that *directly*, more very
vague "there are things others could do". But if/when they do, I definitely
think it's worth trying.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 22 January 2016 at 05:07, Noah Misch <noah@leadboat.com> wrote:

On Wed, Jan 20, 2016 at 06:58:24PM +0000, Simon Riggs wrote:

The main problem is the length of the integration phase, which is mostly
where nothing happens.

The open items wiki page saw steady change from 30 April to 28 December[1];
the two longest quiet periods were an eleven-day gap from 21 August and a
nine-day gap from 16 December. Nineteen distinct contributors were
finding,
fixing or otherwise editing data about 9.5 defects. The integration work
is
too slow, yes, but this persistent myth of a dead period is false.

For my part, I spent 1 July to 11 December reviewing 9.5 commits and
reporting
or fixing defective 9.5 work. (I did spoil myself with a few breaks fixing
pre-9.5 bugs.) I doubt the 9.5 new bugs would have run dry within the next
five months, either, but the release team picked a fair juncture to give up
and call it dot-zero.

nm

[1]
https://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&amp;limit=500&amp;action=history

Thanks for explaining that, and thanks for that work. Had I known you were
doing that I wouldn't have said "nothing happens".

What I do observe is that we have N developers writing patches, C
committers committing patches and R people reviewing things during the
integration phase.

N > C > R so that N >> R

Meaning that during the integration phase our parallel efficiency drops
significantly, but as you say, not to zero. Given that the integration
phase is frequently at least as long as the development phase we end up
operating at about 0.5-0.6 of maximum efficiency. I would like to improve
on that, if possible.

I'll do what I can to shorten the integration phase, including the
prioritization/triage step I volunteered for, applied to the last CF.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jan 22, 2016 at 11:46 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

On 22 January 2016 at 16:34, Robert Haas <robertmhaas@gmail.com> wrote:

For my part, I am not sure the names in the release notes are actually
all that helpful.

It has one important effect of current interest: establishing the truth
that multiple people and multiple companies are involved in producing and
maintaining PostgreSQL. Whether the names are properly attributed will
always be a time-consuming task, but I will oppose any attempt to remove or
obscure evidence of who develops PostgreSQL, wherever that occurs.

Thats a valid point and I think one way to retain such an evidence
without adding name of author/reviewer/committer is to add a link to
commit/'s after each feature, something like whats done in some of
the other release notes [1]http://kernelnewbies.org/LinuxChanges.

[1]: http://kernelnewbies.org/LinuxChanges

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Fri, Jan 22, 2016 at 6:21 PM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-22 08:50:15 -0600, Jim Nasby wrote:

I think that's a great way to ensure we shrink the pool of reviewers when
someone works on a patch and then it goes nowhere.

True, it really sucks. But what's your counter proposal? Commitfests
dragging on forever, and people burning out on continually feeling they
need to work on them? Hasn't worked out well.

Well my point of view is that the entire reason for commitfests is to
ensure every patch gets at least some feedback. If we're going to
bounce patches without any feedback and only work on the patches that
catchh people's interest then that's back where we were before
commitfests. We used to see a lot of patches sit basically ignored
until release time when Tom or someone would pick them up and rewrite
them from scratch because that would be faster than trying to explain
what's wrong and waiting for the original author to rewrite it. It
really sucks for the original author to be ignored for a year and I
think it's how we lost a lot of potential contributors.

I think it's true that there are basically two successful lanes
patches can be in. a) They're of wide interest and need concerted
ongoing effort by multiple people to proceed and b) they're of narrow
interest but really just need a thumbs up or pointer to what direction
to head and the original author can proceed. The first clogs up the
commitfest and dominates people's time when it probably belongs more
in the normal development process. The latter is really what they were
designed for.

Perhaps it would make sense to alternate between "commitfests" that
are intended to return feedback to authors so they can work on their
patch and "developfests" where larger patches that need more
discussion get covered. The latter are expected to drag on but
shouldn't block other work whereas the commitfests are expected to get
relatively short reviews and be bounced as "returned with feedback"
quickly.

Or perhaps we should just have two different "returned with feedback",
one of which is "under discussion" which means the patch has seen some
discussion and therefore doesn't need to be in the commitfest any more
regardless of whether it actually resolved the issues authoritatively.
Making decisions in a consensus-driven community is just hard and we
could use some lessons in how to say no or how to resolve
irreconcilable conflicts but barring solving those issues it would at
least be nice to remove them from the critical path blocking other
patches and making the process feel interminable for bystanders.

--
greg

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 20.01.2016 22:37, Peter Eisentraut wrote:

On 1/20/16 1:44 PM, Robert Haas wrote:

And, you know, I did my time fighting major wars to try to compress
the release schedule, and honestly, it wasn't that much fun. The
process we have now is imperfect in many ways, but I no longer have
abuse heaped on me for wanting to boot a patch out of a CommitFest.
That may be bad for the project, but it's spared me a lot of grief
personally. I think that many other people are in the same situation.
Everybody would like somebody else to be the schedule enforcer ...
unless they have something that they still want to get in, in which
case they would like nobody to do it.

Yeah, a lot of the ideas in this thread, while reasonable, are of the
sort "We need to be better about ..." which sounds a lot like "I need to
be better about exercising". A system based purely on distributed
willpower isn't going to last very long, as we are finding out.

Sure, I'd like more than one party to review my patches, and I'd like to
spend more time doing additional reviews on other people's patches. But
I can barely keep up as it is. I know we need code reviews, but I think
it's never going to work well to the point that we are satisfied with
it. Just look around the world, in software, open or commercial, or
even academics, and tell me who has peer reviews figured out.

Nobody, but there are different solutions. And the same solutions works
different in quality and quantity in the different projects.
In FreeBSD for example there is an online tool for review
(http://review.freebsd.org) which was opened to public. There you can
review any code, left comments in the parts you wanted, submit different
users to it etc.
It is not perfect, but a huge step forward for the project. And
something i misses here often.
But as stated earlier in another thread: for a not-so-deep-involved
volunteer, it is often unclear *what* to review. The threads are long
and often there is no final description about how the patch is supposed
to work. That make testing quite hard and time consuming.

It is not just about the schedule, the reviewer and participants. Its
also some more basic.

Greetings,
Torsten

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 01:59:59PM -0800, Peter Geoghegan wrote:

I think that we've learned some lessons from the problems with 9.3. I
don't think that one of those lessons was "take more time to release".
There is reason to doubt that that would have changed matters one bit
with 9.3. It might be a good idea to formally state what those lessons
are.

Hindsight of 9.3 taught me to monitor three additional[1]They joined older signals, some too laborious to measure for every commit: - complexity of the area(s) patched - plausible scope of breakage from defects, given the area(s) patched - gravity of follow-on repair commits signals for every
commit. Each is weak individually, firing often on harmless commits.
However, if commit 0ac5ad5 landed again today, I would at least prioritize
further analysis based on all three signals being present:

1. Patch came together shortly before a deadline.

The tree sees a bunch of feature commits just before feature freeze. This
list sees many fresh patch versions just before the last CommitFest submission
deadline. For an important subset, time pressure led the author or committer
to reduce vigilance. This is an analog signal, and I write "came together" to
be deliberately vague. The more the various steps of development work
(design, implementation, review, commit) cluster near some deadline, the more
to worry.

2. Committer disclaimed knowledge of the patch's correctness.

From commit 0ac5ad5 procedural history:

Tom Lane said at one point "this is too complex to maintain". Several
times during the development I feared he might well be right. I am sure
he will be discovering many oversights and poor design choices, when
gets around to reviewing the code; hopefully some extra effort will be
all that's needed to make the whole thing work sanely and not eat
anyone's data. I just hope that nothing so serious comes up that the
patch will need to be reverted completely.

Author/committer personality is a large factor in determining whether this
signal arrives. If I do see it, I now believe the author about the patch
being dangerous.

3. Committer == author and no other committer certified the patch.

By "certify", I mean a direct statement like "This patch is correct." or "This
feature is now bug-free." etc. Posting a review does not, by itself, qualify,
and reviewers regularly do enough to earn a co-author credit without having
certified any version of the patch. Being listed as the patch's git
author/committer _does_ imply certification.

This signal is common, particularly for smaller patches, and it ought to
remain common. The product would advance slowly if every contrib module or
vacuumdb flag took full attention from two committers. Every release has a
few patches that do deserve this level of attention, though, and they rarely
attract it. Index-only scans did; commit a2822fb had a non-author committer
despite an author having a commit bit. Committers should recognize particular
self-authored work as too grave to self-certify, submit it for review, and not
commit it until and unless another committer certifies it. Indeed, I regard
that recognition of one's own limits as the central qualification for being a
committer. I have failed at it myself, though.

Watching those signals is the lesson I took from the PostgreSQL 9.3 aftermath.

Thanks,
nm

[1]: They joined older signals, some too laborious to measure for every commit: - complexity of the area(s) patched - plausible scope of breakage from defects, given the area(s) patched - gravity of follow-on repair commits
- complexity of the area(s) patched
- plausible scope of breakage from defects, given the area(s) patched
- gravity of follow-on repair commits

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Fri, Jan 22, 2016 at 1:16 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

It has one important effect of current interest: establishing the truth that
multiple people and multiple companies are involved in producing and
maintaining PostgreSQL. Whether the names are properly attributed will
always be a time-consuming task, but I will oppose any attempt to remove or
obscure evidence of who develops PostgreSQL, wherever that occurs.

Obscuring evidence of who develops PostgreSQL was not my intention.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Jan 25, 2016 at 9:51 AM, Noah Misch <noah@leadboat.com> wrote:

This signal is common, particularly for smaller patches, and it ought to
remain common. The product would advance slowly if every contrib module or
vacuumdb flag took full attention from two committers. Every release has a
few patches that do deserve this level of attention, though, and they rarely
attract it. Index-only scans did; commit a2822fb had a non-author committer
despite an author having a commit bit.

Yeah, I was a bit surprised when Tom committed that, but I was
pleased. I didn't have huge confidence in it at that point, and
having Tom come along and give a hand was a great relief. I think he
not only committed the base patch but improved a very healthy number
of things about it as well.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:

Nobody, but there are different solutions. And the same solutions works
different in quality and quantity in the different projects.
In FreeBSD for example there is an online tool for review
(http://review.freebsd.org) which was opened to public. There you can
review any code, left comments in the parts you wanted, submit different
users to it etc.
It is not perfect, but a huge step forward for the project. And
something i misses here often.
But as stated earlier in another thread: for a not-so-deep-involved
volunteer, it is often unclear *what* to review. The threads are long
and often there is no final description about how the patch is supposed
to work. That make testing quite hard and time consuming.

I agree better code review tooling could help a bit. The URL you post
above doesn't work at the moment (for me), though.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/26/2016 02:09 AM, Peter Eisentraut wrote:

On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:

In FreeBSD for example there is an online tool for review
(http://review.freebsd.org) which was opened to public. There you can
review any code, left comments in the parts you wanted, submit different
users to it etc.

I agree better code review tooling could help a bit. The URL you post
above doesn't work at the moment (for me), though.

That's a typo, it's reviews.freebsd.org.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

All,

So the proximate cause of late releases are the following:

1. patches are getting kicked down the road from one CF to another, creating a big pileup in the final CF. This is exactly like the huge pile of unreviewed patches we used to have before the CF system.

2. Once the last CF is closed, everyone goes off and starts working on the next version, leaving a few people to handle getting the production release integrated and debugged. This is partly because nobody likes doing that work, and partly because most hackers aren't clear how they can help.

So, I have a modest suggestion on how to change all this:

Let's do quarterly development releases and supported production releases every 18 months.

A 3-month release cycle would let people see their code go into the wild a lot faster; basically we'd do a CF then a development release. That would limit pileup issues around people losing interest/moving on, forgetting what was going on, and conflicts between various features in development. A shorter release/dev cycle is more manageable. By having supported production releases 50% less often, we could keep the overall number of versions we need to patch the same as it is now.

The alternative to this is an aggressive recruitment and mentorship program to create more major contributors who can do deep review of patches. But that doesn't seem to have happened in the last 5 years, and even if we started it now, it would be 2 years before it paid off.

--
Josh Berkus
Red Hat OSAS
(opinions are my own)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi,

On 2016-01-26 00:26:18 -0600, Joshua Berkus wrote:

Let's do quarterly development releases and supported production
releases every 18 months.

A 3-month release cycle would let people see their code go into the wild a lot faster; basically we'd do a CF then a development release.

There are reason for/against doing that, but why would it actually
change the problem significantly? It won't change the amount of
necessary review, debugging and bugfixing by a very significant
amount. One CF / 3 months might actually make it harder to get a patch
into a revieawable state, because the time-to-feedback is larger.

The alternative to this is an aggressive recruitment and mentorship
program to create more major contributors who can do deep review of
patches. But that doesn't seem to have happened in the last 5 years,
and even if we started it now, it would be 2 years before it paid off.

I think the amount of review and maintenance time is the largest part of
the problem here, and is mostly unaffected by the manner we actually
schedule releases. The discussions around dropping patches on the floor
are partially a question of that, and partially a question of not
acceppting to maintain things that are of low interest.

But I don't really see "aggressive recruitment and mentorship" really
fixing this, even though there are other good reasons to work more on
that side. I think more fundamentally the issue is that that doing a
"deep" review of a bigger patches takes so much time and knowledge, that
it's not realistic to do so in ones own time anymore. You can if you're
very dedicated over a long time, but realistically in most cases it
requires your employer having an interest in you doing that. Quite
obviously there's an increasing number of employers paying people to
submit things to postgres, whereas there seems no corresponding uptick
on the review side. Unless the "paying people to do review side" is
solved (which has a *long* lead time), I don't see more recruiting,
different CF schedules, or anything like that really fixing the critical
bottlenecks. Reducing the amount of work, i.e. dropping patches not
interesting or too complex, seems to be the only solution until then.

As a short aside: I think another side of the coin is that, if you're
dedicated to work on a cool open source project, these days you'll find
a paying open source job so quickly, that a lot of people that would
have worked on e.g. postgres in their evenings, now just hack something
they're paid both during the day and the evening.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 26.01.2016 02:09, Peter Eisentraut wrote:

On 1/25/16 2:48 AM, Torsten Zühlsdorff wrote:

Nobody, but there are different solutions. And the same solutions works
different in quality and quantity in the different projects.
In FreeBSD for example there is an online tool for review
(http://review.freebsd.org) which was opened to public. There you can
review any code, left comments in the parts you wanted, submit different
users to it etc.
It is not perfect, but a huge step forward for the project. And
something i misses here often.
But as stated earlier in another thread: for a not-so-deep-involved
volunteer, it is often unclear *what* to review. The threads are long
and often there is no final description about how the patch is supposed
to work. That make testing quite hard and time consuming.

I agree better code review tooling could help a bit. The URL you post
above doesn't work at the moment (for me), though.

I'm sorry, the url contains a typo. The correct one is:
https://reviews.freebsd.org/

Greetings,
Torsten

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Jan 26, 2016 at 1:01 PM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-26 00:26:18 -0600, Joshua Berkus wrote:

The alternative to this is an aggressive recruitment and mentorship
program to create more major contributors who can do deep review of
patches. But that doesn't seem to have happened in the last 5 years,
and even if we started it now, it would be 2 years before it paid off.

I think the amount of review and maintenance time is the largest part of
the problem here, and is mostly unaffected by the manner we actually
schedule releases. The discussions around dropping patches on the floor
are partially a question of that, and partially a question of not
acceppting to maintain things that are of low interest.

But I don't really see "aggressive recruitment and mentorship" really
fixing this, even though there are other good reasons to work more on
that side. I think more fundamentally the issue is that that doing a
"deep" review of a bigger patches takes so much time and knowledge, that
it's not realistic to do so in ones own time anymore. You can if you're
very dedicated over a long time, but realistically in most cases it
requires your employer having an interest in you doing that. Quite
obviously there's an increasing number of employers paying people to
submit things to postgres, whereas there seems no corresponding uptick
on the review side.

Yeah, I also share the same feeling, but I think there is more to it, even
if
the employer is ready to support for the review of a patch which takes more
amount of time, the credit to reviewers is always on much lower side as
compare to the original Author even though the effort put by reviewer is
comparable to Author especially for somewhat complex patches.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On 1/22/16 12:14 PM, Andres Freund wrote:

On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:

Ideally reviewers shouldn't be doing any testing, because the tests
that are part of the patch should answer every question they would
have, but I don't see that happening until we have a separate
automation-only target that we don't care how long it takes to run.

I think that's completely wrong.

Yes, more tests are good, and we need a place for longer running
tests. But assuming that every patch author will create a testsuite that
covers every angle is just about akin to assuming every submitter will
deliver perfect, bug free code. And we know how well that turns out.

I think actively trying to break a feature, and postgres in general, is
one of the most important tasks of reviewers and testers. And with that
I don't mean trying to run "make check". Look e.g. at the tests Jeff
Janes has performed, what the recent plug tests of Tomas Vondra brought
to light, or at what the full page write checker tool of Heikki's
showed.

IIRC Jeff's tests are scripted and obviously the page write checker is
as well. I don't recall the exact methodology Tomas was using but I
suspect it could also be scripted if you had a way to pull the plug via
software (via a power management unit or maybe kill -9 of a VM). All of
that is stuff that can and should be automated. Presumably it won't ever
be part of the Makefile tests, but that's fine. Heck, the test scripts
could stay in completely separate repos.

Where the code lives isn't the issue; it's getting stuff like this
automated so humans can go back do doing things that can't be automated.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers