Releasing in September

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 04:48:16PM +0100, Andres Freund wrote:

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

Well, when the commitfest extends to May, you are not going to fork the
next release, and since we haven't forked by PGCon in Ottawa, we try to
get in five commitfest, instead of four, and we then get the schedule we
have now.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 07:40 AM, Bruce Momjian wrote:

Our current 9.5/9.6 timing looks more like the 8.X series of release
dates. Everyone might be fine with that, but we had better be prepared
for November-February major release dates going forward.

Bruce,

Thank you for bringing this up it is a good time to discuss it. I think
we have a number of things we need to consider when it comes to releases:

1. When we release
2. What we release
3. When not to release

#1 September is actually a great time to release. However, I think that
we are going to run into trouble keeping that time frame. The reality is
PostgreSQL of today is a lot more complex than the PostgreSQL of even 3
years ago.

That means that unless we are willing to have stunted releases, we are
going to miss release dates. We need to be able to say, "Ompf, no whiz
bang features this release, mostly polish" or "These whiz bang features
need more testing, we are pushing our release". If we aren't willing to
do that then we need to reconsider having a fixed release date and
perhaps start a release time frame (9.6 will be released Q4 2016).

#2 This is a longer topic. I have been stewing in my head about releases
for years. I have even brought up the idea of an Ubuntu style release
cycle on list once or twice. The more I think about it, the more I think
this can help our community. We essentially would have two types of
releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS. This
allows two LTS versions per 5 year support cycle

There is more to say on this but this is already a long reply.

#3 The fact that 9.5 was delayed is a good thing, not a bad one. I
believe we need to be much more willing to push a button that says, "It
isn't done." Which goes back to the idea in #1 of having a release
"window" versus date. We aren't a company. We don't need to adhere to an
exact release schedule.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote:

We have gotten off of that cycle in the last two major releases, and
this isn't going to improve as long as we have commitfests starting
after January.

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 10:55:07AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 10:48 AM, Andres Freund <andres@anarazel.de> wrote:

I think this has very little to do with commitfest schedules, and much
more with the "early" forking of the new version branch. For both 9.4
and 9.5 we essentially spent a couple months twiddling our thumbs.

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

The bottom line is we can't sustain five commitfests and stay on time
--- we need to go back to four, which I think is what we used to do.  We
twiddled our thumbs back in the September-release years too, but had
consistency because twiddling was built into the schedule.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 10:55:07 -0500, Robert Haas wrote:

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

Said twiddling seems to largely happened from July to December. In which
the other branch was open, and no 9.5 commitfest was happening. If we
move the commitfests to earlier, but still have a half year of nothing
happening, we're still in a bad situation.

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

On 2016-01-20 10:55:07 -0500, Robert Haas wrote:

It's certainly true that we twiddled our thumbs quite a bit about
getting 9.5 ready to ship. However, the old process where nobody
could get anything committed for six months out of the year blew
chunks, too. Personally, I think that the solution is to cut off the
last CommitFest a lot sooner, and then reopen the tree for the next
release as soon as possible. But this never works, because there are
always patches we want to slip in late.

Said twiddling seems to largely happened from July to December. In which
the other branch was open, and no 9.5 commitfest was happening. If we
move the commitfests to earlier, but still have a half year of nothing
happening, we're still in a bad situation.

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

/Magnus

Magnus Hagander <magnus@hagander.net> writes:

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

We will not get back to on-schedule releases unless we can keep -hackers
working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

I'd suggest that the easiest process fix is to not fork a new devel branch
until we reach, say, RC status.

And that does mean losing at least one commitfest per release cycle ---
but it would be the currently-first one, not the currently-last one.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 11:19:28 -0500, Tom Lane wrote:

We will not get back to on-schedule releases unless we can keep -hackers
working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

Agreed. I'll note that the linux kernel used to have a similar
problem. Now there's a two week integration and a 10 week stabilization
period, with occasionally a week added/subtracted. If a feature patch is
submitted for integration (not just parallel review) into the main
branch outside the merge window you get yelled at. Now, we're at least
two magnitudes smaller, so not everything there applies to us. But I
think looking over that fence, to see how they scaled from smaller where
we are now, to way way bigger, isn't a bad idea.

I think one thing we should work on, is being absolutely religious about
requiring, say, 2 reviews for every nontrivial contribution. We
currently seem to have a significantly increased submission rate, and at
the same time the number of reviews per patch has gone down
substantially. I think the "honor" system has failed in that regard.

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:

#2 This is a longer topic. I have been stewing in my head about
releases for years. I have even brought up the idea of an Ubuntu
style release cycle on list once or twice. The more I think about
it, the more I think this can help our community. We essentially
would have two types of releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS.
This allows two LTS versions per 5 year support cycle

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 11:53:32AM -0500, Robert Haas wrote:

On Wed, Jan 20, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6
development branch on June 30th, with the expectation of releasing in
September, and then couldn't release in September because nobody had done
any significant amount of stabilization work. Instead we had the 2015-07
commitfest. And the 2015-09 commitfest. And the 2015-11 commitfest.

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

Unfair or not, that is probably the effect. I can also see people
publishing git trees to avoid this roadblock.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription                             +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 11:53:32 -0500, Robert Haas wrote:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

Yes, that's a problem. But I don't see how we can avoid dealing with it
directly. Avoiding being stalled by unfixed bugs of others by having a
separate branch to continue working, inevitably leads to delayed
releases. If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

But I'm not very sure that we're talking about the same set of people
here. If we're going to go to a system where nobody's allowed to
commit anything for the next release until the current release is
finalized, then we'd better have some procedure for making sure that
happens relatively quickly. And the procedure can't be that the
people who are hot to get started on the next release have to bat
cleanup for the people who don't have time to fix the bugs they
introduced previously. Because *that* would be manifestly unfair.

You're assuming that the people who are hot to get started on the next
release are different from the people who don't have time to fix the bugs
they introduced previously. IME it's mostly the same people.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 01/20/2016 08:51 AM, Bruce Momjian wrote:

On Wed, Jan 20, 2016 at 07:53:25AM -0800, Joshua Drake wrote:

#2 This is a longer topic. I have been stewing in my head about
releases for years. I have even brought up the idea of an Ubuntu
style release cycle on list once or twice. The more I think about
it, the more I think this can help our community. We essentially
would have two types of releases:

STS:

* Supported for 1 release cycle plus 6 months (18-24 months)
* Inline upgrades supported

LTS:

* Supported for standard 5 years
* Is allowed to break binary format from STS but not previous LTS.
This allows two LTS versions per 5 year support cycle

I just don't buy the Ubuntu release model for our database. Ubuntu is
trying to balance hot features vs stability, while we are really focused
on stability, similar to Debian.

I understand but I think we are missing out on an opportunity here.
Notice that the shorter release cycle for STS will actually make some
things easier. Including:

* Increased test base (just like Fedora/Ubuntu)
* Increased early adopter testing (that is what early adopting is
really about for us anyway)
* Decreased concerns about upgrades and ability to extend upgrade status.

I am not in any way suggesting that this is a slam dunk but I do think
something along these lines can really bring forth a lot of
possibilities. Here is another example:

pg_audit was pulled from core, did it have to be? I can't say. I didn't
read the code. I was a proactive member stating we could pull it though
because it was an extension.

However, if it was an STS release it was going into we could be a little
more relaxed and say, "O.k. let's get this into the wild for some
general user testing".

There are a lot of people (think about the multixact problem) that will
run any software because it is new. Let's put the proper disclaimers on
there and let them run it.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Magnus Hagander <magnus@hagander.net> writes:

On Jan 20, 2016 5:03 PM, "Andres Freund" <andres@anarazel.de> wrote:

FWIW, looking at the last few commitfests, aside heroic and
unsustainable efforts by individual CF managers, I haven't noticed any
effect of when fests started/stopped. Aside from a short time increase
in unfinished patches being posted the day before the next CFs starts.

Yeah, we seem to be firmly stuck at two month long commitfests started
every two months. The plan was for them to be one month..

Maybe we should try just very drastically cutting them at one month and
bumping everything left. No questions asked, no extra time for anybody.
Regardless of if it's the first or the last commitfest.

Just to see what happens. Because what we are doing now clearly doesn't
work..

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6

I agree that it's not the same problem. I do believe that it is *a* problem
though, and a fairly significant one too. Because there's *never* any
downtime from CF mode, regardless of where in the cycle we are.

While not the same, we need to fix both.

We will not get back to on-schedule releases unless we can keep -hackers

working on release testing/stabilization when it's time to do that,
rather than being distracted by shiny new stuff going into the next
release.

Agreed.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 01/20/2016 09:03 AM, Andres Freund wrote:

If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

We have been slowly moving to a more dictator based release anyway. It
used to be that we released "when it's done", then we moved to
commitfest, and now we are having yet another discussion on the same
topic of how to manage all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

Life is hard sometimes, we don't always get what we want.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote:

On 01/20/2016 09:03 AM, Andres Freund wrote:

If people don't fix the issues in time, there needs to be
direct pushback, leading to much less stuff getting in next time round.

We have been slowly moving to a more dictator based release anyway. It used
to be that we released "when it's done", then we moved to commitfest, and
now we are having yet another discussion on the same topic of how to manage
all of this.

It seems clear that we need to be able to say:

* "Your feature is awesome. Sorry, you are out of time"

That's not really related to the discussion here tho? The debated
problem is things that have already been committed that have have bugs,
preventing beta/rc releases from being made. In some cases in the last
releases such bugs were in features integrated very early in the cycle.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Magnus Hagander <magnus@hagander.net> writes:

On Wed, Jan 20, 2016 at 5:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I do not think commitfest length is the problem (though surely it's not
working as intended). What happened with 9.5 is we forked the 9.6

I agree that it's not the same problem. I do believe that it is *a* problem
though, and a fairly significant one too. Because there's *never* any
downtime from CF mode, regardless of where in the cycle we are.

True, we've been failing badly on the intention that there would be time
off from CF mode, and I'd like to see a fix for that. I do not think it's
directly related to the can't-get-a-release-out problem.

I'm not really sure why we've allowed CFs to drift on, though. Can't we
just arbitrarily decree them closed on the last day of the month? And
push unfinished work to the next one? Admittedly, this probably doesn't
work for the last CF of a release cycle, but that one's always been a
special case.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers