Getting a move on for 8.2 beta

My feeling is that we ought to bounce bitmap indexes and updatable views
as not being ready, accept all the contrib stuff, and try to get the
other items done in time for a beta at, say, the end of next week.

I'm not thrilled about postponing those two large items till 8.3, but
we are a month past feature freeze and we do not have credible patches
in hand for either.

I know there is a lot of "backend" work that has been done for 8.2 but
the two features you are suggesting we bounce, are probably the most
visible of marketable features we will have for this release. Especially
updateable views.

The alternative is to be willing to slip the
schedule until whenever they are ready, and we have learned the hard way
that that's not good project management.

Well I don't like the idea either, but if it isn't done, it isn't done.
How often do we hear the complaint about software that is shipped before
it is done?

Outside of feature freeze, we really haven't had AFAICS a "solid"
release schedule. Is it going to be push Beta for a month? Does that
push us till the end of the year?

I don't know which is the better solution, but you asked for comments :)

Sincerely,

Joshua D. Drake

Comments?

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Tom Lane wrote:

September is upon us and it doesn't seem like we are ready to ship
a beta. I think it's time to start making some hard choices.

In the main code I see these outstanding features/patches:

* bitmap indexes
* updatable views

IMHO these two should be bounced. They are big features that still seem
to need a lot of work and will probably take long before they are ready
for inclusion. Interested parties should continue working on them with
an eye to be included *at the start* of the 8.3 development cycle.

* GUC variable reload + refactoring (previously applied and reverted)
* plpython improvements (needs review by someone who knows plpython)
* plpgsql improvements for returning record types
* patch to build on VC
* make libpq default client_encoding setting from locale?

In contrib we've got:

* new ISBN/etc module
* hstore (finally proposed for inclusion)
* new sslinfo module
* pgstattuple changes
* removing the deadwood

These all seem to be manageable within reasonable time.

I can help with the contrib stuff.

My feeling is that we ought to bounce bitmap indexes and updatable views
as not being ready, accept all the contrib stuff, and try to get the
other items done in time for a beta at, say, the end of next week.

+1.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Tom,

September is upon us and it doesn't seem like we are ready to ship
a beta. I think it's time to start making some hard choices.

In the main code I see these outstanding features/patches:

* bitmap indexes
* updatable views
* GUC variable reload + refactoring (previously applied and reverted)
* plpython improvements (needs review by someone who knows plpython)
* plpgsql improvements for returning record types
* patch to build on VC

What's VC?

* make libpq default client_encoding setting from locale?

In contrib we've got:

* new ISBN/etc module
* hstore (finally proposed for inclusion)
* new sslinfo module
* pgstattuple changes
* removing the deadwood

This is just a delete. I've fixed the pgFoundry permissions issues and
will be loading the last CVS snapshot today. At that point, Bruce can
delete stuff.

Can you be a little clearer on which things are non-working patches, and
which things are pending due to lack of review? I'm all for bouncing
stuff that's not ready to go, but bouncing stuff because we haven't
recruited enough code reviewers is just bad practice.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus <josh@agliodbs.com> writes:

Can you be a little clearer on which things are non-working patches, and
which things are pending due to lack of review?

The only things I currently want to reject as not having provided a
timely patch are bitmap indexes and updatable views. The other stuff
I mentioned is in need of review, but I don't currently have a reason
to think it can't get in.

I'm all for bouncing
stuff that's not ready to go, but bouncing stuff because we haven't
recruited enough code reviewers is just bad practice.

Well, it's taken us the full month to get this far through the queue, so
I'd sure like to have more people on board reviewing next time. Alvaro
helped but I miss Neil Conway, and some other people have helped in the
past. However --- the present situation has nothing to do with lack of
reviewers, it's lack of time to finish the patches.

regards, tom lane

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

My feeling is that we ought to bounce bitmap indexes and updatable views
as not being ready, accept all the contrib stuff, and try to get the
other items done in time for a beta at, say, the end of next week.

I agree with bouncing these to get the beta out, though I hate having
to wait another year for them. If things change in some way and we have
the opportunity to fit one in this time, I'd rather see updatable views
than bitmap indexes, personally.

Overall, I really think 8.2 is going to be an excellent release. I wish
autovacuum could have been enabled by default and I'd just like to ask,
now and I'll try to remember again once 8.2 is out, please let's turn it
on by default for 8.3 (and early on so we get some good testing of it).

Thanks,

Stephen

"Joshua D. Drake" <jd@commandprompt.com> writes:

My feeling is that we ought to bounce bitmap indexes and updatable views
as not being ready, accept all the contrib stuff, and try to get the
other items done in time for a beta at, say, the end of next week.

I know there is a lot of "backend" work that has been done for 8.2 but
the two features you are suggesting we bounce, are probably the most
visible of marketable features we will have for this release. Especially
updateable views.

Josh said nearly the same in core's discussion, but come now. The
updateable-view patch (in its current form) does not offer *one single
thing* you can't do today, indeed for many PG releases past; it just
saves writing out some tedious rules. If that's the most marketable
feature in 8.2 we're already in trouble. I don't think it is anyway.
Looking at the CVS logs I see

INSERT/UPDATE/DELETE RETURNING
VALUES lists
CREATE INDEX CONCURRENTLY
pl/pgsql debugger (OK, not in core, but available)
restartable WAL recovery
constraint exclusion works on UPDATE/DELETE queries
multiple-argument aggregate functions
SQL2003-standard statistical aggregates
COPY (SELECT ...) TO ...
pg_dump selectivity options
customizable timezone names
GIN indexes
FILLFACTOR for indexes and tables

and that's just back to the beginning of July, so it probably represents
about a quarter of the work that's gone into 8.2, but I got tired of
reading the logs at that point.

We have this discussion every single release cycle: "hey, if we wait X
amount more time then we can have cool features Y and Z". But there's
always another X, Y, and Z. More importantly, this view ignores the
very real disadvantages of not getting already-finished cool features
A through W out to our users sooner.

regards, tom lane

Josh said nearly the same in core's discussion, but come now. The
updateable-view patch (in its current form) does not offer *one single
thing* you can't do today, indeed for many PG releases past; it just
saves writing out some tedious rules.

Sure, I can write the rules, you can write the rules but we can do
rollup queries using unions too, but we still don't have rollup :)

Perception is everything and we don't have Updateable views until we
don't have to write those rules (yes, its stupid)...

If that's the most marketable
feature in 8.2 we're already in trouble. I don't think it is anyway.
Looking at the CVS logs I see

INSERT/UPDATE/DELETE RETURNING
VALUES lists
CREATE INDEX CONCURRENTLY
constraint exclusion works on UPDATE/DELETE queries
multiple-argument aggregate functions
SQL2003-standard statistical aggregates
COPY (SELECT ...) TO ...
pg_dump selectivity options
customizable timezone names
GIN indexes
FILLFACTOR for indexes and tables

Let's take multiple-argument aggregate functions for example... The
marketable portion of that is limited. 85% of the people we are
marketing too don't even know what that means. Heck, I haven't even run
into the pervious limitation :)

Versus SQL2003-standard statistical aggregates. That wins hands down
just because of the SQL2003-standard part of the string :)

COPY (SELECT...) TO, is only marketable to current PostgreSQL users.
Those who are not users don't know the limitation and it is likely not
something they would look for when evaluating PostgreSQL.

For my current customers the things that are the most marketable (note
current customers) is probably:

* CREATE INDEX CONCURRENTLY (this is a huge in production boon)

* GIN indexes (after explanation)

* constraint exclusion works on UPDATE/DELETE queries (We are using
alot of tp now, so this is great)

* Bitmap Indexes

It does not mean all those features are useful, they definitely are. I
am just trying to look at it from at:

WHIZ* BANG* POW* perspective.

What makes the analyst (and I know this isn't the communities problem)
say, "Holy crap batman, we need to be talking to more people about this
database". I think those would be Updateable views, Bitmap Indexes and
the SQL2003-standard statistical aggregates.

Sincerely,

Joshua D. Drake

and that's just back to the beginning of July, so it probably represents
about a quarter of the work that's gone into 8.2, but I got tired of
reading the logs at that point.

We have this discussion every single release cycle: "hey, if we wait X
amount more time then we can have cool features Y and Z". But there's
always another X, Y, and Z. More importantly, this view ignores the
very real disadvantages of not getting already-finished cool features
A through W out to our users sooner.

regards, tom lane

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Josh Berkus wrote:

Tom,

September is upon us and it doesn't seem like we are ready to ship
a beta. I think it's time to start making some hard choices.

In the main code I see these outstanding features/patches:

* bitmap indexes
* updatable views
* GUC variable reload + refactoring (previously applied and reverted)
* plpython improvements (needs review by someone who knows plpython)
* plpgsql improvements for returning record types
* patch to build on VC

What's VC?

MS Visual C++

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:

I'm all for bouncing
stuff that's not ready to go, but bouncing stuff because we haven't
recruited enough code reviewers is just bad practice.

Well, it's taken us the full month to get this far through the queue, so
I'd sure like to have more people on board reviewing next time. Alvaro
helped but I miss Neil Conway, and some other people have helped in the
past. However --- the present situation has nothing to do with lack of
reviewers, it's lack of time to finish the patches.

Also, for next time it would be good to have the reviewers got through
the patches *before* feature freeze, so that developers get early
feedback. (Which in turn means more time to finish the patches).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Bruce,

What's VC?

MS Visual C++

Given that this could lead to us recruiting more developers out of our
Windows user base, I'd prioritize it.

Overall, I think this whole process is pointing up that there's a problem
with our historic Feature Freeze+1 month|beta|RCs cycle. Given the number
of developers and companies involved and the increasing size of the code,
we need to develop a new schedule that allows us to get draft patches (or
at least full specifications) eariler than the month before feature
freeze. In short,the issue is that Feature Freeze isn't just Feature
Freeze, it's "final patch submission".

If you look at the two "incomplete" patches, and the "misfired" one
(Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches
where the submitter had been working on them months ago, and might have
made the release (or let us know they weren't on schedule) if we'd held
them to an earlier deadline. As I recall, it was the same situation last
year: getting large patches dropped on us out of the blue the week before
feature freeze, which then didnt' make it in.

Therefore I propose that we adopt the following schedule for 8.3, assuming
an September 1 Beta date:

June 1: Specification Freeze: specifications for all new features due
July 1: Feature Freeze: Draft patches and any minor tweaks due
August 1: Final (completed, mostly debugged) patches due
September 1: First Beta

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Joshua D. Drake wrote:

It does not mean all those features are useful, they definitely are. I
am just trying to look at it from at:

WHIZ* BANG* POW* perspective.

Holy crap, Batman! This database can do

INSERT INTO foo VALUES (1,1, 'so long'), (42, 2, 'and thanks'),
(142857, 3, 'for all the fish')

now! We should be talking to more people about that!

That will make some MySQL users happy at least a bit :-)

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Can you be a little clearer on which things are non-working patches, and
which things are pending due to lack of review?

The only things I currently want to reject as not having provided a
timely patch are bitmap indexes and updatable views. The other stuff
I mentioned is in need of review, but I don't currently have a reason
to think it can't get in.

I'm all for bouncing
stuff that's not ready to go, but bouncing stuff because we haven't
recruited enough code reviewers is just bad practice.

Well, it's taken us the full month to get this far through the queue, so
I'd sure like to have more people on board reviewing next time. Alvaro
helped but I miss Neil Conway, and some other people have helped in the
past. However --- the present situation has nothing to do with lack of
reviewers, it's lack of time to finish the patches.

I did try to get us additional help in reviewing. Neil was unavailable,
and Alvaro could only give part of his time.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:

Let's take multiple-argument aggregate functions for example... The
marketable portion of that is limited. 85% of the people we are
marketing too don't even know what that means. Heck, I haven't even
run into the pervious limitation :)

As far as market analysis goes, I'd guess that 100% of the people who
already use PostgreSQL by definition don't need any new features and
would rather see incremental improvements across the board, which is
what this or any release would give them.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Josh Berkus wrote:

Bruce,

What's VC?

MS Visual C++

Given that this could lead to us recruiting more developers out of our
Windows user base, I'd prioritize it.

Overall, I think this whole process is pointing up that there's a problem
with our historic Feature Freeze+1 month|beta|RCs cycle. Given the number
of developers and companies involved and the increasing size of the code,
we need to develop a new schedule that allows us to get draft patches (or
at least full specifications) eariler than the month before feature
freeze. In short,the issue is that Feature Freeze isn't just Feature
Freeze, it's "final patch submission".

If you look at the two "incomplete" patches, and the "misfired" one
(Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches
where the submitter had been working on them months ago, and might have
made the release (or let us know they weren't on schedule) if we'd held
them to an earlier deadline. As I recall, it was the same situation last
year: getting large patches dropped on us out of the blue the week before
feature freeze, which then didnt' make it in.

Therefore I propose that we adopt the following schedule for 8.3, assuming
an September 1 Beta date:

June 1: Specification Freeze: specifications for all new features due
July 1: Feature Freeze: Draft patches and any minor tweaks due
August 1: Final (completed, mostly debugged) patches due
September 1: First Beta

No rejiggering is going to get people to complete things they didn't
complete under the old system. The plan you list above is what we did
for this release.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:

Well, it's taken us the full month to get this far through the queue,
so I'd sure like to have more people on board reviewing next time.
Alvaro helped but I miss Neil Conway, and some other people have
helped in the past. However --- the present situation has nothing to
do with lack of reviewers, it's lack of time to finish the patches.

Not that I've had particularly plenty of time to give, but what's
concerning me is that while we're moving toward an increasingly rigid
process, we don't really have the process support to go with it. How
would anyone even know what patches are pending review? Something to
think about ...

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

Well, it's taken us the full month to get this far through the queue, so
I'd sure like to have more people on board reviewing next time. Alvaro
helped but I miss Neil Conway, and some other people have helped in the
past. However --- the present situation has nothing to do with lack of
reviewers, it's lack of time to finish the patches.

I did try to get us additional help in reviewing. Neil was unavailable,
and Alvaro could only give part of his time.

It strikes me that setting feature freeze in midsummer might not be the
best strategy for having manpower available to review --- people tend to
be on vacation in August. Maybe the answer is just to move the dates a
bit one way or the other.

regards, tom lane

Therefore I propose that we adopt the following schedule for 8.3, assuming
an September 1 Beta date:

June 1: Specification Freeze: specifications for all new features due

I would suggest the above be:

June 1: Specification Freeze: specifications for all new features due &
approved.

The last thing we want it to spend all of june arguing about how
something should be implemented and thus no real coding gets done.

July 1: Feature Freeze: Draft patches and any minor tweaks due
August 1: Final (completed, mostly debugged) patches due
September 1: First Beta

other then that w00t!

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Josh Berkus <josh@agliodbs.com> writes:

Bruce,

What's VC?

MS Visual C++

Given that this could lead to us recruiting more developers out of our
Windows user base, I'd prioritize it.

Yeah, that's why I listed it as a major item. I haven't had a chance to
look at the patch, but if it's not too ugly I would like to get it in
this time rather than next --- that could lead directly to having more
people available next time.

If you look at the two "incomplete" patches, and the "misfired" one
(Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches
where the submitter had been working on them months ago, and might have
made the release (or let us know they weren't on schedule) if we'd held
them to an earlier deadline.

Perhaps, but I'm not sure what we can or should do about it. Moving
deadlines up will either create a dead zone where we all sit around
twiddling our thumbs, or people will keep on coding till the last minute
anyway.

I think having a few patches that don't make the deadline isn't a bad
thing: it means we didn't have people sitting idle. It's not like the
work will go to waste --- those things can still get in in the next
devel cycle.

regards, tom lane

Alvaro Herrera wrote:

Joshua D. Drake wrote:

It does not mean all those features are useful, they definitely are. I
am just trying to look at it from at:

WHIZ* BANG* POW* perspective.

Holy crap, Batman! This database can do

INSERT INTO foo VALUES (1,1, 'so long'), (42, 2, 'and thanks'),
(142857, 3, 'for all the fish')

now! We should be talking to more people about that!

That will make some MySQL users happy at least a bit :-)

To be fair, it will make some of our customers happy to.. they can now
get rid of the weird union hack they had to do instead :)

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/