Feature freeze progress report

In summary, for a patch to be applied, someone has to understand the
patch and the subsystem it modifies. In the past, most complex patches
came from experienced developers, so even if no one but the author fully
understood the patch, we could rely on the author to some extent. With
new people developing complex patches, we don't have the same experience
level of the authors, so we have to do extra work to verify the patch
isn't going to break things. That is the crux of the difficulty during
the 8.3 feature freeze period.

---------------------------------------------------------------------------

bruce wrote:

Now that we are half-way though the scheduled feature freeze, I wanted
to share my thoughts about this period.

Having just pushed all open items into the patches queue or 8.4 hold
queue, I am seeing that we have many more in-process patches than we
normally do at this stage of the process. I think there are three
reasons for this:

1) The patches are not necessarily larger, but are more complex because
much most of the easy TODO items have already been written for previous
PostgreSQL releases.

2) We have a number of new developers who took on some of these complex
TODO items, and some of the TODO items were significantly beyond the
developer capabilities at the start of the process.

3) Many of the complex patches are hard to review because they deal
with very complex areas of the code, like reliability or transaction
semantics.

Our community could probably handle a few of these complex patches, but
the volume for this release is significantly higher than previous
releases. The community is doing a good job of giving patch writers
feedback and new patch versions are getting generated. However, this
amount of patch churn is not normal.

There are a few possible results that might come out of this:

1) Feature freeze will be much longer.
2) More patches will be postponed for later releases than usual.
3) Most patches will be included but beta will be longer because
of bug fixing.
4) Most patches will be included but beta will not be any longer.

I think we all hope for #4, but right now, I don't know the probability
of that. We are going to have to think creatively in the coming weeks
to increase the likelihood of a #4 result. However, right now, I can't
think of what we can do to improve the odds. I think the community has
to come up with ideas on how to accomplish this.

[ FYI, I leave on a 2-week trip tomorrow/Friday.]

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Thu, 2007-04-26 at 23:13 -0400, Bruce Momjian wrote:

Our community could probably handle a few of these complex patches, but
the volume for this release is significantly higher than previous
releases. The community is doing a good job of giving patch writers
feedback and new patch versions are getting generated. However, this
amount of patch churn is not normal.

I think it is probably going to be normal from now on. We now a
significant number of reasonably prolific developers.

I think the community has to come up with ideas on how to accomplish this.

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

With those two suggestions, the prod release would freeze on Sep 30 and
the dev release on Mar 31. This would then put us into the same
situation as Linux, where odd-numbered releases are dev and
even-numbered are main releases. Everyone would understand our decision
to take this action, as well as immediately understanding how this
process will work in the future.

By agreeing this now, we can then punt a reasonable number of patches
back to the main release, later this year. The de-selected patches still
have a second chance of making it into a release available in 2007 and
this will diffuse the various tensions and difficulties we now have.

Without this, we face a long wait. 8.2 took 4 months to go through beta
and release, so 8.3 could well take 6 months. If we allow 8.3 to be a
mega-release then it might take much longer than that: increases in
complexity have a non-linear effect on software quality/robustness.
Adding reviewers or committers isn't going to dramatically change that.
IMHO the only sensible thing to do is to make releases more frequent so
that each one is still a manageable size.

The alternative is to somehow select patches to wait until the next
release, a full year away. That is unlikely to be an easy process and
nobody really wants to volunteer their own or others' patches.

Realistically, people won't speed up the frequency they upgrade their
software and we certainly don't want to increase the number of
production releases in circulation that we must support. This set of
proposals is a realistic way forward from where we are and will be
easily explained to people only briefly in touch with our project.

Whether or not this is accepted, I'm happy to offer assistance wherever
the core team directs to improve the current situation.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

Simon Riggs wrote:

On Thu, 2007-04-26 at 23:13 -0400, Bruce Momjian wrote:

Our community could probably handle a few of these complex patches, but
the volume for this release is significantly higher than previous
releases. The community is doing a good job of giving patch writers
feedback and new patch versions are getting generated. However, this
amount of patch churn is not normal.

I think it is probably going to be normal from now on. We now a
significant number of reasonably prolific developers.

I think the community has to come up with ideas on how to accomplish this.

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

I'm not really convinced that this is good idea at all - it would lead
to further fragmentation of developer resources (likely more versions to
support and more frequent releases which to put quite a load on
developers by itself). And a lot of issues only get found in the field
and I'm unsure if a releases labled "dev" might get the critical mass in
terms of testing (if that would be true we would find most of the bugs
during beta imho).
99% of the people only use what is declared "stable" and already has a
number of minor releases - and the other 1% is already reading -hackers ...

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

I would rather like to see patches we don't are confident enough in to
be dropped from 8.3 and moved to 8.4 - the goal should not be jamming as
much patches into a single release s we can (because they are proposed)
but rather putting those in that meet the quality bar and we trust in.

With those two suggestions, the prod release would freeze on Sep 30 and
the dev release on Mar 31. This would then put us into the same
situation as Linux, where odd-numbered releases are dev and
even-numbered are main releases. Everyone would understand our decision
to take this action, as well as immediately understanding how this
process will work in the future.

I don't want to see the current linux model - which is basically that
2.6 is a continous development trunk with snapshots (ie "releases") done
every few months that only get supported until the next release or so.
Note that all the distributions spend considerable amount of time
stabilizing those kernels and have to put additional resources into
maintaining those over the years because upstream is not doing that any
more.
Look into the recent discussion about releasing 2.6.21 with a number of
KNOWN regressions for example.

By agreeing this now, we can then punt a reasonable number of patches
back to the main release, later this year. The de-selected patches still
have a second chance of making it into a release available in 2007 and
this will diffuse the various tensions and difficulties we now have.

Without this, we face a long wait. 8.2 took 4 months to go through beta
and release, so 8.3 could well take 6 months. If we allow 8.3 to be a
mega-release then it might take much longer than that: increases in
complexity have a non-linear effect on software quality/robustness.
Adding reviewers or committers isn't going to dramatically change that.
IMHO the only sensible thing to do is to make releases more frequent so
that each one is still a manageable size.

again - the bottleneck right now seems to be reviewer capacity coupled
with a large number of intrusive patches. So the most logical answer is
to drop those patches we are not fully confident in and try to get them
in during 8.4.

The alternative is to somehow select patches to wait until the next
release, a full year away. That is unlikely to be an easy process and
nobody really wants to volunteer their own or others' patches.

see above

Stefan

On 4/28/07, Simon Riggs <simon@2ndquadrant.com> wrote:

I think the community has to come up with ideas on how to accomplish this.

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

This would mean we would have to have a very well tested upgrade path
for odd releases (8.2 -> 8.4).

Also it probably would mean that analytical functions or recursive queries
should be postponed until 8.5 (as they didn't end up inside 8.3, and 8.4
would be "stable" release).

I think that with introducing stable/devel version we are risking that devel
versions will be less used in production environments (meaning less testing)
and as a result they can lengthen the development cycle. Currently every
release is stable, therefore we don't accept "experimental patches" unless
they are really good idea. Then there is beta sequence, and then a stable
release. With introducing dev release, we give green light to more
"experimental"
patches, and then devote dev release as a ripening period for them (equivalent
of current pre-releases, I imagine). And then we release stable relese (without
"experimental" patches; experimental patches are postponed until devel release,
and devel release twice the number of experimental patches).

I think we should not go into stable/devel release cycle without carefully
thinking if it will serve us good. I am afraid this will make many people
stick with stable releases and will make upgrades harder (do you remember
how hard it was to go from Linux 2.2 to 2.4, and from 2.4 to 2.6?).

Regards,
Dawid

Simon Riggs wrote:

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

I like the idea of draining the patch queue mid-way through the release
cycle. That'll hopefully encourage people to submit patches earlier in
the release cycle, knowing they will be reviewed. It'll also give people
working on external projects, drivers and tools, a checkpoint to sync with.

But I don't like the idea of making a release out of it. Who would use
such a release? No one in production. Making a release comes with a
cost, even if it's just a dev release.

One could also argue that we don't need the mid-cycle checkpoint, if we
just keep the patch queue empty all the time. In the end, it comes down
to how many people we have actively reviewing patches and giving
feedback (I agree that it's not a linear relationship as you pointed out
later in your mail, though). I believe a mid-cycle checkpoint would help
by directing efforts to review, just like the pre-release feature freeze
does.

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

I have no reason to believe that the next release will have less patches
in it, so if we went down that path we could never release a stable
release. If we have reasonable doubts about the stability of a patch, it
should not be included. That said, all patches come with a risk.

With those two suggestions, the prod release would freeze on Sep 30 and
the dev release on Mar 31. This would then put us into the same
situation as Linux, where odd-numbered releases are dev and
even-numbered are main releases. Everyone would understand our decision
to take this action, as well as immediately understanding how this
process will work in the future.

We're having a short 8.3 cycle because we wanted to shift our release
schedule from Autumn to Spring. That would get us back to releasing in
Autumn.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Stefan Kaltenbrunner wrote:

Simon Riggs wrote:

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

I would rather like to see patches we don't are confident enough in to
be dropped from 8.3 and moved to 8.4 - the goal should not be jamming as
much patches into a single release s we can (because they are proposed)
but rather putting those in that meet the quality bar and we trust in.

Yeah; the agreement we had was that 8.3 would be a short release. So if
we're going to take too long to review and apply the outstanding patches
we have, we should rather push them to 8.4, get 8.3 released quickly and
then go on with the regular annual release. The postponed patches can
be reviewed and committed early in 8.4, instead of at the last minute in
8.3. Sounds like a smarter, safer move.

(The only complication would be the pgindent changes which could cause
merge problems for some patches. It would be good to have a mechanism
to "update" a patch over pgindent easily.)

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

"Heikki Linnakangas" <heikki@enterprisedb.com> writes:

I like the idea of draining the patch queue mid-way through the release cycle.
That'll hopefully encourage people to submit patches earlier in the release
cycle, knowing they will be reviewed. It'll also give people working on
external projects, drivers and tools, a checkpoint to sync with.

But I don't like the idea of making a release out of it. Who would use such a
release? No one in production. Making a release comes with a cost, even if it's
just a dev release.

On other projects people use these snapshots or dev releases because checking
stuff out from CVS is likely to get you a source tree that won't even build
let alone run cleanly. It's also nice to have everyone using the same
checkouts when report bugs or submitting patches.

But it's not because we're afraid some user will run a CVS checkout that
Postgres CVS is kept clean. Postgres CVS is kept clean because that's just the
way the Postgres developers think it should work. Doing regular snapshot
releases isn't going to cause that to get worse.

One could also argue that we don't need the mid-cycle checkpoint, if we just
keep the patch queue empty all the time. In the end, it comes down to how many
people we have actively reviewing patches and giving feedback

I would argue that. In fact I would argue it would be *easier* to keep the
patch queue empty all the time than to spend months reviewing and merging
six-month-old patches once a year. But I still have hope this is a problem
that will fix itself naturally with time.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

"Alvaro Herrera" <alvherre@commandprompt.com> writes:

The postponed patches can be reviewed and committed early in 8.4, instead of
at the last minute in 8.3.

Given that some of those patches have been in the queue since last September
is there any reason to think they'll get reviewed early in this cycle if they
weren't last cycle?

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Alvaro Herrera wrote:

Yeah; the agreement we had was that 8.3 would be a short release. So if
we're going to take too long to review and apply the outstanding patches
we have, we should rather push them to 8.4, get 8.3 released quickly and
then go on with the regular annual release. The postponed patches can
be reviewed and committed early in 8.4, instead of at the last minute in
8.3. Sounds like a smarter, safer move.

Hmm, I do not have an overview on this, but like Alvaro mentions, the
shorter release cycles for 8.3 was done because we felt that a number of
patches that were originally slated for 8.2 were almost but not quite
ready for 8.2. So are all of those patches from back then ready to go
into 8.3? If not then it would indicate that fast tracking a release
cycle for patches there are not quite there yet is not paying off?

Otherwise, if all/most of the patches originally planned for 8.2 have
made it into 8.3, everything is good. If new additions are not yet ready
then they will just get bumped to 8.4, just like the changes that got
bumped to 8.3.

regards,
Lukas

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

Simon Riggs wrote:

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point.

I'm not really convinced that this is good idea at all - it would lead
to further fragmentation of developer resources (likely more versions to
support and more frequent releases which to put quite a load on
developers by itself).

That's my reaction too. The overhead of a "dev" release would be just
as high as a full release, and we don't really have enough manpower
to do two releases a year. We *definitely* haven't got enough manpower
to double the number of back branches we are trying to keep patched.
So this could only work if dev releases are abandoned from a support
perspective when the next full release comes out, and that will entirely
guarantee that no DBA will use one in production.

regards, tom lane

Lukas Kahwe Smith <smith@pooteeweet.org> writes:

Hmm, I do not have an overview on this, but like Alvaro mentions, the
shorter release cycles for 8.3 was done because we felt that a number of
patches that were originally slated for 8.2 were almost but not quite
ready for 8.2. So are all of those patches from back then ready to go
into 8.3? If not then it would indicate that fast tracking a release
cycle for patches there are not quite there yet is not paying off?

In fact several of the major ones are still not ready, so I think that
experience is evidence that we couldn't handle a six-month release cycle
anyway. We'll still stick to the announced 8.3 schedule though, since
part of the reason for it was to rotate around to a spring release time
instead of a fall release time, on the thought that that might work more
conveniently for many people.

regards, tom lane

Heikki Linnakangas wrote:

I like the idea of draining the patch queue mid-way through the release
cycle. That'll hopefully encourage people to submit patches earlier in
the release cycle, knowing they will be reviewed. It'll also give people
working on external projects, drivers and tools, a checkpoint to sync with.

This is important for me - the pgAdmin development cycle follows that of
PostgreSQL's for very obvious reasons, however *after* we enter 'feature
freeze' we can still end up adding significant amounts of new code. Why?
Becvause during PostgreSQL's feature freeze, patches are applied which
add new functionality we need to support. We can't code for the new
features when patches are submitted because we don't know if they'll go
in, or how much they'll change when they do.

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

Sooner or later with things working the way they are now, we *will*
reach a breaking point at which pgAdmin simply won't be ready when
PostgreSQL is released.

But I don't like the idea of making a release out of it. Who would use
such a release? No one in production. Making a release comes with a
cost, even if it's just a dev release.

Agreed. That would have the opposite effect of what should happen.

I like the idea of having a sync point mid cycle, however, what I'd like
to see even more is an improved system in which we put less pressure on
the few committers we have, and give them more freedom to commit patches
they may not understand fully themselves by having an improved community
review and feedback process to give the patches the approval they need.
Doing so might allow us to keep the queue of a more or less fixed, short
length throughout the cycle. There would be a few advantages to this:

- The problem described above practically vanishes.
- Developers see their work accepted more quickly, giving them the
confidence to produce more.
- Developers are able to build on their earlier work, knowing that it's
believed to be reasonably sound, unlike now when they may not know for
months.

I believe we can achieve this with a couple of relatively minor changes:

1) *Slowly* introduce more committers. The are developers gaining
experience all the time, and as they become trusted by the existing
committers/core they can be 'promoted' to alleviate some of the pressure
on the existing guys.

2) Introduce a new patch management system. I suggest a web interface
through which patches be submitted. This would assign them an ID number,
and forward them to the patches list. The system would track any
responses to the initial email, logging the thread automatically and
making it available through the web interface. Posts from
trusted/experienced developers might be highlighted so that committers
can see at a glance if any of the more experienced guys have commented
on the patch yet. A status flag could easily include a status flag to
mark them as "won't accept", "committed", "under review" or "under
revision". If left at "under review" for too long, the patch might be
highlighted, and if at "under revision" for too long, the patch author
might be automatically requested to send a status report.

There are potentially a number of benefits to such a system:

- No patches will get lost
- Bruce's time is feed up from the mundane work of tracking patches, to
allow him to spend time developing, reviewing/committing and all the
other great jobs he does for the community.
- Status of patches can be seen at a glance.
- *All* discussion of a patch can be logged in one place, allowing the
committer to put more reliance on the knowledge and experience of his
peers, rather than being expected to fully understand the minutae of
every patch - thus allowing him to commit more.

Well, I'll stop there as this is getting long winded - I'm sure others
will have their own ideas about how we can improve our processes for
future releases; one thing I'm certain of though, is that we absolutely
must strive to improve them somehow as whilst they has served us well in
the past, the current process is starting to show that it just won't
scale as the project grows.

Regards, Dave

Dave Page wrote:

Heikki Linnakangas wrote:

I like the idea of draining the patch queue mid-way through the
release cycle. That'll hopefully encourage people to submit patches
earlier in the release cycle, knowing they will be reviewed. It'll
also give people working on external projects, drivers and tools, a
checkpoint to sync with.

This is important for me - the pgAdmin development cycle follows that of
PostgreSQL's for very obvious reasons, however *after* we enter 'feature
freeze' we can still end up adding significant amounts of new code. Why?
Becvause during PostgreSQL's feature freeze, patches are applied which
add new functionality we need to support. We can't code for the new
features when patches are submitted because we don't know if they'll go
in, or how much they'll change when they do.

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

this is indeed an issue - but there is nothing that says that pgadminIII
has to support all the new features of a backend the they it get
released. pgadmin is decoupled from the min development cycle anyway so
adding support for a new features a few months later sounds not too much
an issue.

Sooner or later with things working the way they are now, we *will*
reach a breaking point at which pgAdmin simply won't be ready when
PostgreSQL is released.

well if it still works but does not yet support $wizzbangnewfeature that
sounds ok too me ?

[...]

2) Introduce a new patch management system. I suggest a web interface
through which patches be submitted. This would assign them an ID number,
and forward them to the patches list. The system would track any
responses to the initial email, logging the thread automatically and
making it available through the web interface. Posts from
trusted/experienced developers might be highlighted so that committers
can see at a glance if any of the more experienced guys have commented
on the patch yet. A status flag could easily include a status flag to
mark them as "won't accept", "committed", "under review" or "under
revision". If left at "under review" for too long, the patch might be
highlighted, and if at "under revision" for too long, the patch author
might be automatically requested to send a status report.

this sounds like trying to reinvent a real bugtracking system with an
email interface ...

[...]

Well, I'll stop there as this is getting long winded - I'm sure others
will have their own ideas about how we can improve our processes for
future releases; one thing I'm certain of though, is that we absolutely
must strive to improve them somehow as whilst they has served us well in
the past, the current process is starting to show that it just won't
scale as the project grows.

not sure I fully agree here - I think we could do a bit better on the
"bug tracking front" but it is a bit unclear if we cn honestly sy that
"the current process" does not work or is not going to scale in the
future. Complex patches need time - sometimes much more time than a
release or even two release cycles - it's unclear if trying to get those
in more agressively (by having more commiters or applying them earlier)
might not actually cause more harm due to -HEAD getting less stable and
causing developers to spend time hunting regressions.

Stefan

Stefan Kaltenbrunner wrote:

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

this is indeed an issue - but there is nothing that says that pgadminIII
has to support all the new features of a backend the they it get
released. pgadmin is decoupled from the min development cycle anyway so
adding support for a new features a few months later sounds not too much
an issue.

No it's not decoupled; it runs almost exactly in sync. Our most popular
platform ships with pgAdmin as standard because thats what the users of
that platform expect. Not shipping with pgAdmin (or a functionally
complete one) would be like shipping SQL Server without Enterprise Manager.

Sooner or later with things working the way they are now, we *will*
reach a breaking point at which pgAdmin simply won't be ready when
PostgreSQL is released.

well if it still works but does not yet support $wizzbangnewfeature that
sounds ok too me ?

Who's to say it will? Changes to pg_database have required a new release
in the past.

this sounds like trying to reinvent a real bugtracking system with an
email interface ...

More or less - but one that's simple by design, and specifically for
tracking what happens with our patches with the absolute minimum amount
of change required to our existing communication methods.

not sure I fully agree here - I think we could do a bit better on the
"bug tracking front" but it is a bit unclear if we cn honestly sy that
"the current process" does not work or is not going to scale in the
future. Complex patches need time - sometimes much more time than a
release or even two release cycles -

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

it's unclear if trying to get those

in more agressively (by having more commiters or applying them earlier)
might not actually cause more harm due to -HEAD getting less stable and
causing developers to spend time hunting regressions.

I'm not advocating committing patches that might destabilize the code,
I'm suggesting making it easier for individual committers to make use of
the knowledge and experience of everyone else in the community, whilst
at the same time reducing the reliance on their own experience. Even now
we occasionally see patches getting committed that (for example) Tom has
rejected months earlier. At the very least a tracker should help prevent
that happening, at best it will help committers work faster and more
effectively because they have all the relevant discussion in front of them.

Regards, Dave.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Sunday, April 29, 2007 21:30:36 +0200 Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:

not sure I fully agree here - I think we could do a bit better on the
"bug tracking front" but it is a bit unclear if we cn honestly sy that
"the current process" does not work or is not going to scale in the
future. Complex patches need time - sometimes much more time than a
release or even two release cycles - it's unclear if trying to get those
in more agressively (by having more commiters or applying them earlier)
might not actually cause more harm due to -HEAD getting less stable and
causing developers to spend time hunting regressions.

re: -HEAD getting less stable ... we wouldn't be adding just anyone as a
committer, only those that are trusted to know what they are doing ... even
'slapping in a patch', I would expect it to have some sort of preliminary
review by *someone* with a bit of knowledge in that aspect of the code ...

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFGNQrf4QvfyHIvDvMRAoUqAKDKZY8rThsQi1vTKeQgq/c4HPIyzQCfe9o2
2fHy763eMw/hxGrMnDwXnLM=
=g6E8
-----END PGP SIGNATURE-----

Stefan Kaltenbrunner wrote:

2) Introduce a new patch management system. I suggest a web interface
through which patches be submitted. This would assign them an ID number,
and forward them to the patches list. The system would track any
responses to the initial email, logging the thread automatically and
making it available through the web interface. Posts from
trusted/experienced developers might be highlighted so that committers
can see at a glance if any of the more experienced guys have commented
on the patch yet. A status flag could easily include a status flag to
mark them as "won't accept", "committed", "under review" or "under
revision". If left at "under review" for too long, the patch might be
highlighted, and if at "under revision" for too long, the patch author
might be automatically requested to send a status report.

this sounds like trying to reinvent a real bugtracking system with an
email interface ...

I think the angle of this suggestion is to approach things from the
perspective of trying to automate more according to how people are
currently working instead of shoehorning an existing solution. It seems
that most folks on -hackers prefer email based systems. The proposals
sounds like it would not change anything much for people who choose to
ignore the web interface and as such it has some appeal to it.

regards,
Lukas

Dave Page wrote:

But I don't like the idea of making a release out of it. Who would
use such a release? No one in production. Making a release comes with
a cost, even if it's just a dev release.

Agreed. That would have the opposite effect of what should happen.

I like the idea of having a sync point mid cycle, however, what I'd
like to see even more is an improved system in which we put less
pressure on the few committers we have, and give them more freedom to
commit patches they may not understand fully themselves by having an
improved community review and feedback process to give the patches the
approval they need. Doing so might allow us to keep the queue of a
more or less fixed, short length throughout the cycle. There would be
a few advantages to this:

I don't think we need a sync point. I think we need to get better at
setting expectations and at managing the patch queue so that it gets
drained better all the time. Nothing can be more frustrating for patch
authors than to have patches in the queue for a very long time. They
bitrot, and we sometime end up throwing curly questions at authors long
after the issues are hot in their minds. I'd like to see us set
ourselves some targets for handling patches. Something like: patches
held over from feature freeze from the previous release will be reviewed
within two months of the tree re-opening, and all other patches will be
reviewed within one month of being submitted. That implies that one
month after feature freeze the tree will only be open for bug fixes. Any
patches unapplied at that time would be held over. Maybe that would give
pgAdmin and friends enough head room to catch up.

cheers

andrew

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Sunday, April 29, 2007 19:38:21 -0400 Andrew Dunstan
<andrew@dunslane.net>
wrote:

patches held over from feature freeze from the previous
release will be reviewed within two months of the tree re-opening

a. why 2 months? shouldn't they be given priority, period?
b. what happens after 2 months? we delete them?

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFGNVQA4QvfyHIvDvMRAordAJ9SDrMHFumGNynXVjpdjlR5WoMfAgCgt1Xe
+7jMjrOUODmEK+glmGyrHYA=
=SgPr
-----END PGP SIGNATURE-----

Dave Page <dpage@postgresql.org> writes:

I like the idea of having a sync point mid cycle, however, what I'd like
to see even more is an improved system in which we put less pressure on
the few committers we have, and give them more freedom to commit patches
they may not understand fully themselves

That is a recipe for disaster :-(. The real problem I see with the big
patches that are currently in the queue is that I'm not sure even the
authors understand the patches (or more accurately, all their potential
consequences) completely. Telling committers they should apply such
patches without having understood them either is just going to lead to
an unfixably broken system.

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

regards, tom lane

Dave Page wrote:

Stefan Kaltenbrunner wrote:

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

this is indeed an issue - but there is nothing that says that pgadminIII
has to support all the new features of a backend the they it get
released. pgadmin is decoupled from the min development cycle anyway so
adding support for a new features a few months later sounds not too much
an issue.

No it's not decoupled; it runs almost exactly in sync. Our most popular
platform ships with pgAdmin as standard because thats what the users of
that platform expect. Not shipping with pgAdmin (or a functionally
complete one) would be like shipping SQL Server without Enterprise Manager.

Interesting ... so if you have a new feature (or a number of them) -
that is not directly depending on some sort of new backend feature - in
pgadmin you "delay" it until the next postgresql mjor release ?
To be honest I personally have not used pgadmin in years and I have no
idea what SQL-Server would be with or without Enterprise Manager so I
actually don't really know enough to further speculate on this ...

Sooner or later with things working the way they are now, we *will*
reach a breaking point at which pgAdmin simply won't be ready when
PostgreSQL is released.

well if it still works but does not yet support $wizzbangnewfeature that
sounds ok too me ?

Who's to say it will? Changes to pg_database have required a new release
in the past.

hmm true - but I imagine that a change to a catalog like pg_database is
not the kind of feature you need to rush lot's of code in (again
speculating here) ?

this sounds like trying to reinvent a real bugtracking system with an
email interface ...

More or less - but one that's simple by design, and specifically for
tracking what happens with our patches with the absolute minimum amount
of change required to our existing communication methods.

ah - ok

not sure I fully agree here - I think we could do a bit better on the
"bug tracking front" but it is a bit unclear if we cn honestly sy that
"the current process" does not work or is not going to scale in the
future. Complex patches need time - sometimes much more time than a
release or even two release cycles -

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

that one I agree with - heck even people very close to the project are
sometimes unclear about the status of this patch or that patch.
Part of that could probably be solved by the simple tracker you are
proposing - another way might be to promote more usage of the developer
wiki.

Stefan

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

I like the idea of having a sync point mid cycle, however, what I'd like
to see even more is an improved system in which we put less pressure on
the few committers we have, and give them more freedom to commit patches
they may not understand fully themselves

That is a recipe for disaster :-(. The real problem I see with the big
patches that are currently in the queue is that I'm not sure even the
authors understand the patches (or more accurately, all their potential
consequences) completely. Telling committers they should apply such
patches without having understood them either is just going to lead to
an unfixably broken system.

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

Err, I thought that was precisely what I was suggesting in my second
point :-). To reiterate:

- Expand the pool of committers (slowly, and as appropriate - not for
the sake of it). There inevitably is and will continue to be more work
for experienced committers. We should consider 'promoting' those
developers that become experienced and trusted.

- Use a tracking system to enable the committers to rely more on the
experience of the users. Ideas we have discussed here in the office
~(which I didn't mention earlier) included a scoring system, where
trusted developers (who aren't necessarily committers) can score patches
or veto them if there are real problems spotted and to highlight the
comments from those experienced people to make it easy to spot what they
say.

Regards, Dave.

Stefan Kaltenbrunner wrote:

Interesting ... so if you have a new feature (or a number of them) -
that is not directly depending on some sort of new backend feature - in
pgadmin you "delay" it until the next postgresql mjor release ?

It's not so much that we delay the new features, it's just that we
follow the same development schedule, just with a shorter beta/feature
freeze period. We try to release just before PostgreSQL to avoid our
respective advocacy efforts trampling each other - but that's usually a
bit of a guessing game in itself!

To be honest I personally have not used pgadmin in years and I have no
idea what SQL-Server would be with or without Enterprise Manager so I
actually don't really know enough to further speculate on this ...

I imagine the %age of SQL Server users that *don't* use Enterprise
Manager is close to zero. It's a platform on which everything is
expected to be a GUI.

Who's to say it will? Changes to pg_database have required a new release
in the past.

hmm true - but I imagine that a change to a catalog like pg_database is
not the kind of feature you need to rush lot's of code in (again
speculating here) ?

No, but it's exactly the reason why we release with/just before
PostgreSQL. If we were offset by six months, we might find ourselves
having to do compatibility releases mid-cycle for the latest PostgreSQL
release. A change in pg_database such as we had previously wouldn't be
an issue for the stable branch, but the changes to op classes etc. in
8.3 certainly would be of great concern.

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

that one I agree with - heck even people very close to the project are
sometimes unclear about the status of this patch or that patch.
Part of that could probably be solved by the simple tracker you are
proposing - another way might be to promote more usage of the developer
wiki.

Yep, true - though the reason I promote the use of the tracker is that
it could be implemented with minimal invasiveness into the existing
process, such that it automatically captures all discussion on the
topic, whereas I imagine some might object to repeatedly
visting/re-reading/editting a wiki to discuss a patch.

Regards, Dave

On Mon, Apr 30, 2007 at 08:01:04AM +0200, Stefan Kaltenbrunner wrote:

Dave Page wrote:

Stefan Kaltenbrunner wrote:

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

this is indeed an issue - but there is nothing that says that pgadminIII
has to support all the new features of a backend the they it get
released. pgadmin is decoupled from the min development cycle anyway so
adding support for a new features a few months later sounds not too much
an issue.

No it's not decoupled; it runs almost exactly in sync. Our most popular
platform ships with pgAdmin as standard because thats what the users of
that platform expect. Not shipping with pgAdmin (or a functionally
complete one) would be like shipping SQL Server without Enterprise Manager.

Interesting ... so if you have a new feature (or a number of them) -
that is not directly depending on some sort of new backend feature - in
pgadmin you "delay" it until the next postgresql mjor release ?

Yes.

<snip>

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

that one I agree with - heck even people very close to the project are
sometimes unclear about the status of this patch or that patch.
Part of that could probably be solved by the simple tracker you are
proposing - another way might be to promote more usage of the developer
wiki.

The advantage of a "proper system" would be that it'd be consistent. Using
the wiki could get very unstructured (unstructured being a *feature* of a
wiki). A specialised system will be able to enforce how things look and are
worked with, and makes the *use* of the system easier. Using the wiki makes
installing it easier, since it's already there, at the cost of usage.

//Magnus

Dave Page wrote:

Stefan Kaltenbrunner wrote:

Interesting ... so if you have a new feature (or a number of them) -
that is not directly depending on some sort of new backend feature - in
pgadmin you "delay" it until the next postgresql mjor release ?

It's not so much that we delay the new features, it's just that we
follow the same development schedule, just with a shorter beta/feature
freeze period. We try to release just before PostgreSQL to avoid our
respective advocacy efforts trampling each other - but that's usually a
bit of a guessing game in itself!

ah ok - makes sense

To be honest I personally have not used pgadmin in years and I have no
idea what SQL-Server would be with or without Enterprise Manager so I
actually don't really know enough to further speculate on this ...

I imagine the %age of SQL Server users that *don't* use Enterprise
Manager is close to zero. It's a platform on which everything is
expected to be a GUI.

I will take your word on that - Iäm neither a Windows nor a MSSQL Server
user :-)

Who's to say it will? Changes to pg_database have required a new release
in the past.

hmm true - but I imagine that a change to a catalog like pg_database is
not the kind of feature you need to rush lot's of code in (again
speculating here) ?

No, but it's exactly the reason why we release with/just before
PostgreSQL. If we were offset by six months, we might find ourselves
having to do compatibility releases mid-cycle for the latest PostgreSQL
release. A change in pg_database such as we had previously wouldn't be
an issue for the stable branch, but the changes to op classes etc. in
8.3 certainly would be of great concern.

hmm - understood. I guess I simply speculated that doing a pgadmin
release might be much more lightweight than doing a PostgreSQL release
(how many "backbranches" is pgadmin supporting?". I think I now
understand why you are doing it that way though ...

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

that one I agree with - heck even people very close to the project are
sometimes unclear about the status of this patch or that patch.
Part of that could probably be solved by the simple tracker you are
proposing - another way might be to promote more usage of the developer
wiki.

Yep, true - though the reason I promote the use of the tracker is that
it could be implemented with minimal invasiveness into the existing
process, such that it automatically captures all discussion on the
topic, whereas I imagine some might object to repeatedly
visting/re-reading/editting a wiki to discuss a patch.

you are probably right here too - though I can see some value in a wiki
too. Some things that come to mind are subproject specific todo
lists(like the XMLTodo) or the Wishlist (which is a rather abstracted
thing to point people to that ask "what can we expect in $release if we
are really really lucky" and don't want to parse the pgpatches list
bruce keeps)

Stefan

Stefan Kaltenbrunner wrote:

No, but it's exactly the reason why we release with/just before
PostgreSQL. If we were offset by six months, we might find ourselves
having to do compatibility releases mid-cycle for the latest PostgreSQL
release. A change in pg_database such as we had previously wouldn't be
an issue for the stable branch, but the changes to op classes etc. in
8.3 certainly would be of great concern.

hmm - understood. I guess I simply speculated that doing a pgadmin
release might be much more lightweight than doing a PostgreSQL release

Oh, it is - but then the work is done mainly by me, with Guillaume
handling the translations and other people producing some of the binary
builds. We don't have anything like to pool of developers that
PostgreSQL does.

(how many "backbranches" is pgadmin supporting?". I think I now
understand why you are doing it that way though ...

We don't support any back branches because each version of pgAdmin III
supports PostgreSQL >= 7.3. When a new major release comes out, we drop
support for the previous. We do have the advantage of not having to
worry about on-disk upgrading data formats :-)

Regards,Dave

Tom Lane wrote:

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

Agreed, committing a patch is not much work. If all the patches in the
queue were perfect and just waiting for committing, one person could
just commit them all in a day.

What takes time is reviewing. We have people capable of reviewing
patches, we should encourage them to do that (that includes myself).
Maybe we should have a more official protocol of "signing-off" patches.
And part of the review is refactoring and commenting and fixing tiny
bugs that the original author missed. I've refrained from doing that
myself because I'm afraid I'd step on the authors toes or joggle the
elbow of a committer.

One problem with the current patch queue is that it's really hard to get
a picture of where each patch stands. There's many different reasons why
a patch can get stalled in the patch queue. Looking at the patches there
now:

Design issues have been raised:
- index advisor (messy API)
- full page writes improvement, code update (increases WAL size)
- dead space map (uses a fixed size shared memory block)

Dependency on other patches:
- scan-resistant buffer cache
- group commit
- error correction for n_dead_tuples

Do we want this or not? -category
- POSIX shared mem support

Unfinished:
- bitmap indexes

Author is working on patch, based on comments
- heap page diagnostic functions
- load distributed checkpoint

Author is waiting for feedback on how to proceed:
- clustered indexes / grouped index tuples
- concurrent psql patch

(not exhaustive list, just examples)

The above list reflects my view of the status of the patches. Others
might not agree, and that's a problem in itself: it's not clear even to
a patch author why a patch isn't moving forward. Author might think a
patch is just about to be committed, while others are waiting for the
author to fix an issue raised earlier. Or an author might think there's
a problem with his patch, while it just hasn't been committed because of
a dependency to another patch.

If we had a 1-2 lines status blurp attached to each patch in the queue,
like "waiting for review", "author is fixing issue XX", etc., that might
help. Bruce would need to do that if we keep the current patch queue
system unmodified otherwise, or we'd need to switch to something else.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Stefan Kaltenbrunner wrote:

Simon Riggs wrote:

On Thu, 2007-04-26 at 23:13 -0400, Bruce Momjian wrote:

Our community could probably handle a few of these complex patches, but
the volume for this release is significantly higher than previous
releases. The community is doing a good job of giving patch writers
feedback and new patch versions are getting generated. However, this
amount of patch churn is not normal.

I think it is probably going to be normal from now on. We now a
significant number of reasonably prolific developers.

I think the community has to come up with ideas on how to accomplish this.

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

I'm not really convinced that this is good idea at all - it would lead

Agreed, a bad idea.

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

I would rather like to see patches we don't are confident enough in to
be dropped from 8.3 and moved to 8.4 - the goal should not be jamming as
much patches into a single release s we can (because they are proposed)
but rather putting those in that meet the quality bar and we trust in.

I don't want us postponing patches for a later release if the patch will
be no easier to apply later than it is right now, i.e. if it is "buckle
down" now or "buckle down" later, we might as well do it now, hence my
desire to move things along now rather than when our backs are against
the wall to get a release out.

Of course, patches were will learn more by waiting for 8.4 should
probably be kept for that release.

again - the bottleneck right now seems to be reviewer capacity coupled
with a large number of intrusive patches. So the most logical answer is
to drop those patches we are not fully confident in and try to get them
in during 8.4.

We are not going to magically have more time or be more confident for
8.4. If a patch needs more research or time, we should hold it, but not
having time to review it isn't a reason to hold it -- it just
bottlenecks the next release.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Heikki Linnakangas wrote:

Simon Riggs wrote:

My thinking is to move to a two stage release process: Do one
"production" release annually, and one "dev" release at the 6 month
mid-point. That way each new release contains a manageable number of new
features and we have a realistic chance of integrating them
successfully. Support companies would then have the option to support
both releases, or just the main production release. Leading edge users,
of which we have many, would then benefit from more frequent additional
features.

I like the idea of draining the patch queue mid-way through the release
cycle. That'll hopefully encourage people to submit patches earlier in
the release cycle, knowing they will be reviewed. It'll also give people
working on external projects, drivers and tools, a checkpoint to sync with.

Aside from a few complex patches all the patches in the queue are from
work completed just before feature freeze --- we have been draining it
during the entire release. I think for the few complex patches that
have been in there for a while, the problem is that reviewing them is
going to be so hard, no one has done it. Now that we are in feature
freeze, we have to do it --- but the idea that we have somehow just been
holding patches during the whole release isn't true.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:

Stefan Kaltenbrunner wrote:

Simon Riggs wrote:

I would also suggest that 8.3 be labelled a dev release. We have a
reasonable number of fairly invasive patches, so we need a mechanism to
integrate them with reduced risk.

I would rather like to see patches we don't are confident enough in to
be dropped from 8.3 and moved to 8.4 - the goal should not be jamming as
much patches into a single release s we can (because they are proposed)
but rather putting those in that meet the quality bar and we trust in.

Yeah; the agreement we had was that 8.3 would be a short release. So if
we're going to take too long to review and apply the outstanding patches
we have, we should rather push them to 8.4, get 8.3 released quickly and
then go on with the regular annual release. The postponed patches can
be reviewed and committed early in 8.4, instead of at the last minute in
8.3. Sounds like a smarter, safer move.

Because we are dealing with this now, and not later, we have time to
give all patches the appropriate review time --- we don't need to panic
yet and start throwing patches to 8.4, especially since we might have
even more patches for 8.4.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Gregory Stark wrote:

"Alvaro Herrera" <alvherre@commandprompt.com> writes:

The postponed patches can be reviewed and committed early in 8.4, instead of
at the last minute in 8.3.

Given that some of those patches have been in the queue since last September
is there any reason to think they'll get reviewed early in this cycle if they
weren't last cycle?

Agreed, though most patches are from late March, just before feature
freeze. I don't see any really old stuff except the index advisor,
which isn't ready for application because the user and backend API need
work.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Lukas Kahwe Smith wrote:

Alvaro Herrera wrote:

Yeah; the agreement we had was that 8.3 would be a short release. So if
we're going to take too long to review and apply the outstanding patches
we have, we should rather push them to 8.4, get 8.3 released quickly and
then go on with the regular annual release. The postponed patches can
be reviewed and committed early in 8.4, instead of at the last minute in
8.3. Sounds like a smarter, safer move.

Hmm, I do not have an overview on this, but like Alvaro mentions, the
shorter release cycles for 8.3 was done because we felt that a number of
patches that were originally slated for 8.2 were almost but not quite
ready for 8.2. So are all of those patches from back then ready to go
into 8.3? If not then it would indicate that fast tracking a release
cycle for patches there are not quite there yet is not paying off?

Otherwise, if all/most of the patches originally planned for 8.2 have
made it into 8.3, everything is good. If new additions are not yet ready
then they will just get bumped to 8.4, just like the changes that got
bumped to 8.3.

The patches _might_ be ready. Please re-read my earlier posting that
started this thread -- the major problem is new developers adding
complex features, and the difficulty of reviewing all of that.

Fortunately I have gotten approval from EnterpriseDB for Heikki to spend
full-time helping with the 8.3 patch queue, and he, Tom and I have
already been over many of the items via private email and will be moving
forward.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Dave Page wrote:

2) Introduce a new patch management system. I suggest a web interface
through which patches be submitted. This would assign them an ID number,
and forward them to the patches list. The system would track any
responses to the initial email, logging the thread automatically and
making it available through the web interface. Posts from
trusted/experienced developers might be highlighted so that committers
can see at a glance if any of the more experienced guys have commented
on the patch yet. A status flag could easily include a status flag to
mark them as "won't accept", "committed", "under review" or "under
revision". If left at "under review" for too long, the patch might be
highlighted, and if at "under revision" for too long, the patch author
might be automatically requested to send a status report.

It would be interesting if such a system could be made to work simply,
but I am afraid that isn't possible. What happens now is that as people
post email comments about the patches, I add the emails to the patches
queue. It would be interesting to put comments on the patches
themselves, but in many cases the opinions about patches are too candid
to put in public so committers email each other to give status reports.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Dave Page wrote:

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

Agreed. Remember that patches queue is just patches that no one has
dealt with. It was never designed to be a community thing, but Tom and
others do pull from it as necessary. If the community dealt with all
patches, I wouldn't have to add anything to the queue.

I'm not advocating committing patches that might destabilize the code,
I'm suggesting making it easier for individual committers to make use of
the knowledge and experience of everyone else in the community, whilst
at the same time reducing the reliance on their own experience. Even now
we occasionally see patches getting committed that (for example) Tom has
rejected months earlier. At the very least a tracker should help prevent
that happening, at best it will help committers work faster and more
effectively because they have all the relevant discussion in front of them.

This gets back to the same issue as a bug trackers --- the information
has to be managed or it just becomes a dumping ground, and who is going
to do that if the community can't even comment on some patches.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan wrote:

I don't think we need a sync point. I think we need to get better at
setting expectations and at managing the patch queue so that it gets
drained better all the time. Nothing can be more frustrating for patch
authors than to have patches in the queue for a very long time. They
bitrot, and we sometime end up throwing curly questions at authors long
after the issues are hot in their minds. I'd like to see us set
ourselves some targets for handling patches. Something like: patches
held over from feature freeze from the previous release will be reviewed
within two months of the tree re-opening, and all other patches will be
reviewed within one month of being submitted. That implies that one
month after feature freeze the tree will only be open for bug fixes. Any
patches unapplied at that time would be held over. Maybe that would give
pgAdmin and friends enough head room to catch up.

This is what already happens, and if it doesn't happen sometimes, it is
just because people are too busy. Making arbitrary deadlines isn't
going to help, partly because we have little control over our community.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

I like the idea of having a sync point mid cycle, however, what I'd like
to see even more is an improved system in which we put less pressure on
the few committers we have, and give them more freedom to commit patches
they may not understand fully themselves

That is a recipe for disaster :-(. The real problem I see with the big
patches that are currently in the queue is that I'm not sure even the
authors understand the patches (or more accurately, all their potential
consequences) completely. Telling committers they should apply such
patches without having understood them either is just going to lead to
an unfixably broken system.

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

I am still unclear how the patch would get into such a system, and how
we would add comments, apply, and later remove it, without causing us
even more work.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Every patch in the queue is ready for review. If we have bounced
something back for the author to fix, it gets removed from the queue.

As far as adding comments, again, this wasn't meant to be a community
resource, just my tracking system, and I have given feedback on the
status to any committers who wants to know about a patch.

---------------------------------------------------------------------------

Heikki Linnakangas wrote:

Tom Lane wrote:

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

Agreed, committing a patch is not much work. If all the patches in the
queue were perfect and just waiting for committing, one person could
just commit them all in a day.

What takes time is reviewing. We have people capable of reviewing
patches, we should encourage them to do that (that includes myself).
Maybe we should have a more official protocol of "signing-off" patches.
And part of the review is refactoring and commenting and fixing tiny
bugs that the original author missed. I've refrained from doing that
myself because I'm afraid I'd step on the authors toes or joggle the
elbow of a committer.

One problem with the current patch queue is that it's really hard to get
a picture of where each patch stands. There's many different reasons why
a patch can get stalled in the patch queue. Looking at the patches there
now:

Design issues have been raised:
- index advisor (messy API)
- full page writes improvement, code update (increases WAL size)
- dead space map (uses a fixed size shared memory block)

Dependency on other patches:
- scan-resistant buffer cache
- group commit
- error correction for n_dead_tuples

Do we want this or not? -category
- POSIX shared mem support

Unfinished:
- bitmap indexes

Author is working on patch, based on comments
- heap page diagnostic functions
- load distributed checkpoint

Author is waiting for feedback on how to proceed:
- clustered indexes / grouped index tuples
- concurrent psql patch

(not exhaustive list, just examples)

The above list reflects my view of the status of the patches. Others
might not agree, and that's a problem in itself: it's not clear even to
a patch author why a patch isn't moving forward. Author might think a
patch is just about to be committed, while others are waiting for the
author to fix an issue raised earlier. Or an author might think there's
a problem with his patch, while it just hasn't been committed because of
a dependency to another patch.

If we had a 1-2 lines status blurp attached to each patch in the queue,
like "waiting for review", "author is fixing issue XX", etc., that might
help. Bruce would need to do that if we keep the current patch queue
system unmodified otherwise, or we'd need to switch to something else.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Marc G. Fournier wrote

patches held over from feature freeze from the previous
release will be reviewed within two months of the tree re-opening

a. why 2 months? shouldn't they be given priority, period?

Yes, but they don't always seem to get it.

b. what happens after 2 months? we delete them?

Of course not. What I am suggesting is that we set ourselves review
targets. The if we start to get in danger of missing them the authors
and/or anyone else can legitimately start ringing alarm bells.

cheers

andrew

Bruce Momjian wrote:

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

I like the idea of having a sync point mid cycle, however, what I'd like
to see even more is an improved system in which we put less pressure on
the few committers we have, and give them more freedom to commit patches
they may not understand fully themselves

That is a recipe for disaster :-(. The real problem I see with the big
patches that are currently in the queue is that I'm not sure even the
authors understand the patches (or more accurately, all their potential
consequences) completely. Telling committers they should apply such
patches without having understood them either is just going to lead to
an unfixably broken system.

[ thinks for a bit... ] What we need to expand is not so much the pool
of committers as the pool of reviewers. If a patch had been signed off
on by X number of reasonably-qualified people then it'd be fair to
consider that it could be committed. The tracking system you suggest
could make that sort of approach manageable.

I am still unclear how the patch would get into such a system, and how
we would add comments, apply, and later remove it, without causing us
even more work.

In my original message I described my thinking:

- Developer submits patch, with additional info through a web interface.

- The web interface formats an email containing the patch description,
patch and any other info entered, assigns it a patch number, and
forwards it to the patches list.

- Discussion ensues on list as per normal. The tracking system monitors
the list and automatically attaches any posts to the patch that have the
appropriate reference in the subject line.

- Community members and committers can review the entire discussion
through the systems web interface. Updated patches could be submitted
online.

- Committers log into the system when necessary to alter the status
(committed, rejected, awaiting revision, under review etc), or the queue
name (8.3, 8.4 etc). This could also be done automagically via email
keywords from specific addresses.

You would no longer need to manually manage the queue, and the
committers would simply need to tweak the status flag as required.

Regards, Dave

Bruce Momjian wrote:

Dave Page wrote:

2) Introduce a new patch management system. I suggest a web interface
through which patches be submitted. This would assign them an ID number,
and forward them to the patches list. The system would track any
responses to the initial email, logging the thread automatically and
making it available through the web interface. Posts from
trusted/experienced developers might be highlighted so that committers
can see at a glance if any of the more experienced guys have commented
on the patch yet. A status flag could easily include a status flag to
mark them as "won't accept", "committed", "under review" or "under
revision". If left at "under review" for too long, the patch might be
highlighted, and if at "under revision" for too long, the patch author
might be automatically requested to send a status report.

It would be interesting if such a system could be made to work simply,
but I am afraid that isn't possible. What happens now is that as people
post email comments about the patches, I add the emails to the patches
queue.

This what I propose to automate.

It would be interesting to put comments on the patches
themselves, but in many cases the opinions about patches are too candid
to put in public so committers email each other to give status reports.

Any out of band discussion (between you and Tom for example) would
presumably be summarised on list anyway by one or other of you. There
would be no reason I could see to need to be any more candid than to
reject a patch outright, giving a list of reasons why - all of which can
and should be public. If you feel a need to vent about the author for
any other reason, well, thats why we have pubs in the UK :-).

Regards, Dave

Dave Page wrote:

In my original message I described my thinking:

- Developer submits patch, with additional info through a web interface.

- The web interface formats an email containing the patch description,
patch and any other info entered, assigns it a patch number, and
forwards it to the patches list.

- Discussion ensues on list as per normal. The tracking system monitors
the list and automatically attaches any posts to the patch that have the
appropriate reference in the subject line.

- Community members and committers can review the entire discussion
through the systems web interface. Updated patches could be submitted
online.

- Committers log into the system when necessary to alter the status
(committed, rejected, awaiting revision, under review etc), or the queue
name (8.3, 8.4 etc). This could also be done automagically via email
keywords from specific addresses.

You would no longer need to manually manage the queue, and the
committers would simply need to tweak the status flag as required.

You can look on Apache Derby project. I think everything what you
mentioned there Derby project is using.

See http://db.apache.org/derby/

Zdenek

Dave Page wrote:

Stefan Kaltenbrunner wrote:

This means that there is a huge rush of new code in pgAdmin's
development cycle, right at the time when we should be testing - making
the release process more and more rushed as each release of PostgreSQL
gets more efficient and adds more and more new features.

this is indeed an issue - but there is nothing that says that pgadminIII
has to support all the new features of a backend the they it get
released. pgadmin is decoupled from the min development cycle anyway so
adding support for a new features a few months later sounds not too much
an issue.

No it's not decoupled; it runs almost exactly in sync. Our most popular
platform ships with pgAdmin as standard because thats what the users of
that platform expect. Not shipping with pgAdmin (or a functionally
complete one) would be like shipping SQL Server without Enterprise Manager.

And also from another point of view Postgres and related version of
PgAdmin must fit in same release cycle windows of OS distributions. When
OS release is out new feature is not usually accepted and OS should be
shipped with old version pgAdmin.

And bigger problem then new feature in pgAdmin is
pg_upgrade/pg_migrator. Upgrade procedure must be finished at same time
as new release, but some upgrade functions are possible coding only
after feature freeze or when all affected patches are committed.

If we want to have this feature (upgrade) in postgres we would adjust
release cycle anyway.

Zdenek

On Mon, Apr 30, 2007 at 07:27:10AM -0400, Bruce Momjian wrote:

Dave Page wrote:

I'm not specifically talking about complex patches (nor am I talking at
all about bug tracking) - there are a variety of patches in the queue,
of varying complexity. Some have been there for months, and worse, some
of them recieved little or no feedback when submitted leaving the
authors completely in the dark about whether their work will be
included, whether further changes are required, or whether they should
continue with additional enhancements.

Agreed. Remember that patches queue is just patches that no one has
dealt with. It was never designed to be a community thing, but Tom and
others do pull from it as necessary. If the community dealt with all
patches, I wouldn't have to add anything to the queue.

I think that summarises the problem with it fairly well. It was never
designde to be a community thing. But we need a community thing now (we
didn't at the time, probably). Oh, and you are of course no less community
than anybody else ;-)

//Magnus

Heikki,

We're having a short 8.3 cycle because we wanted to shift our release
schedule from Autumn to Spring. That would get us back to releasing in
Autumn.

Er, no. We wanted to change the cycle to avoid having Feature Freeze occur at
midsummer (N. hemisphere) when many of our committers are unavailable due to
conferences or vacation.

-----------

Question #559: Would changing version control systems help us on this at all?
I'm specifically thinking of preventing bitrot; would using a decentralized
VCS allow patch authors to easily prevent bitrot on their own?

-----------

I do like the idea of a web management interface for patches. It has a number
of additional advantages:

-- Advocacy volunteers would know what's under development and thus what they
can talk about at user groups

-- Advanced users who are interested in a specific patch could download that
patch early, test it for their own applications, and supply feedback to the
community even before feature freeze.

-- A more organized queue would make backporting by the backports project
easier.

-- We could save the patches by applied date and index them, and then have a
place to point users when they ask: "When was X fixed? Do I *have* to
upgrade to 8.1 or just 8.0?"

-- It would make it easier to manage Google Summer of Code students and their
work.

-- The status of a partially complete patch abandoned by its author would be
much clearer and thus more likely to get picked up by someone else.

-- The patch manager could eventually be integrated with the Buildfarm to do
automated patch testing.

Overall, I think this would be an excellent direction to move for 8.4. As web
applications go, it doesn't even sound hard; I think I could write it if I
weren't on airplanes all the time.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

On Mon, 2007-30-04 at 08:56 -0300, Heikki Linnakangaspgsql wrote:

Date: Mon, 30 Apr 2007 09:18:36 +0100
From: Heikki Linnakangas <heikki@enterprisedb.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Dave Page <dpage@postgresql.org>, Simon Riggs
<simon@2ndquadrant.com>,
Bruce Momjian <bruce@momjian.us>,
PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Feature freeze progress report
Message-ID: <4635A65C.70005@enterprisedb.com>

If we had a 1-2 lines status blurp attached to each patch in the
queue,
like "waiting for review", "author is fixing issue XX", etc., that
might
help. Bruce would need to do that if we keep the current patch queue
system unmodified otherwise, or we'd need to switch to something else.

Would it be possible to also automatically determine some sort of
bit-rot status? What I had in mind was an automated process that would
apply each patch to HEAD on a daily basis and report whether the patch
still applies cleanly and still allows all regression tests to pass on
at least one platform. If and when the result of these tests changes
from pass to fail, the patch submitter would be automatically
notified.

The patch status could then also show the last time at which the patch
applied cleanly, and the last time that regression tests ran
successfully.

__
Marc

Marc Munro wrote:

On Mon, 2007-30-04 at 08:56 -0300, Heikki Linnakangaspgsql wrote:

Date: Mon, 30 Apr 2007 09:18:36 +0100
From: Heikki Linnakangas <heikki@enterprisedb.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Dave Page <dpage@postgresql.org>, Simon Riggs
<simon@2ndquadrant.com>,
Bruce Momjian <bruce@momjian.us>,
PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Feature freeze progress report
Message-ID: <4635A65C.70005@enterprisedb.com>

If we had a 1-2 lines status blurp attached to each patch in the
queue,
like "waiting for review", "author is fixing issue XX", etc., that
might
help. Bruce would need to do that if we keep the current patch queue
system unmodified otherwise, or we'd need to switch to something else.

Would it be possible to also automatically determine some sort of
bit-rot status? What I had in mind was an automated process that would
apply each patch to HEAD on a daily basis and report whether the patch
still applies cleanly and still allows all regression tests to pass on
at least one platform. If and when the result of these tests changes
from pass to fail, the patch submitter would be automatically
notified.

The patch status could then also show the last time at which the patch
applied cleanly, and the last time that regression tests ran
successfully.

This or something similar has been discussed in the past w.r.t. the
buildfarm. One major problem is that most sane system owners won't want
to apply, compile and run an arbitrary patch. It could well have an
intended or unintended trojan horse, for example. So you'd need some
level of sanity checking to be done by some trusted person even to get
it to this stage.

cheers

andrew

marc@bloodnok.com (Marc Munro) writes:

On Mon, 2007-30-04 at 08:56 -0300, Heikki Linnakangaspgsql wrote:

Date: Mon, 30 Apr 2007 09:18:36 +0100
From: Heikki Linnakangas <heikki@enterprisedb.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Dave Page <dpage@postgresql.org>, Simon Riggs
<simon@2ndquadrant.com>,
Bruce Momjian <bruce@momjian.us>,
PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Feature freeze progress report
Message-ID: <4635A65C.70005@enterprisedb.com>

If we had a 1-2 lines status blurp attached to each patch in the
queue,
like "waiting for review", "author is fixing issue XX", etc., that
might
help. Bruce would need to do that if we keep the current patch queue
system unmodified otherwise, or we'd need to switch to something else.

Would it be possible to also automatically determine some sort of
bit-rot status? What I had in mind was an automated process that would
apply each patch to HEAD on a daily basis and report whether the patch
still applies cleanly and still allows all regression tests to pass on
at least one platform. If and when the result of these tests changes
from pass to fail, the patch submitter would be automatically
notified.

The patch status could then also show the last time at which the patch
applied cleanly, and the last time that regression tests ran
successfully.

Hmm. That would be an interesting extension to the build farm.

If only the timing were right for us to get a GSoC person or something
such to add such functionality...
--
let name="cbbrowne" and tld="acm.org" in String.concat "@" [name;tld];;
http://linuxdatabases.info/info/emacs.html
"Lisp is an eternal thought in the mind of God."
--Crassus, mutatis mutandi

Dave Page wrote:

In my original message I described my thinking:

- Developer submits patch, with additional info through a web interface.

- The web interface formats an email containing the patch description,
patch and any other info entered, assigns it a patch number, and
forwards it to the patches list.

- Discussion ensues on list as per normal. The tracking system monitors
the list and automatically attaches any posts to the patch that have the
appropriate reference in the subject line.

- Community members and committers can review the entire discussion
through the systems web interface. Updated patches could be submitted
online.

- Committers log into the system when necessary to alter the status
(committed, rejected, awaiting revision, under review etc), or the queue
name (8.3, 8.4 etc). This could also be done automagically via email
keywords from specific addresses.

You would no longer need to manually manage the queue, and the
committers would simply need to tweak the status flag as required.

Sounds interesting, but I am not sure how that is going to track
multiple versions of the patch, or changes in the email subject.

The bottom line is that there is a lot of thinking that the patch queue
is so large because no one knows what to do. "Oh, if we were better
communicators, more would be done". The patch queue is large because we
have lots of March 31 patches, and because we don't have enough people
to review them quickly.

The people who have expressed interest in reviewing patches already know
where we stand on the patches, and a status email of where we are each
patch will be posted shortly. I just can't do it this time because I am
traveling.

If you want to try a tracking system, go ahead, just pull from the
patches email list, and somehow try to grab discussion from
hackers/patches on this patches, and give a way to manually update the
patch status via a web site. If your system works, I will not need to
maintain a separate patches queue, but I will keep doing it until we
know the web site idea will work, just in case.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

I believe the suggestion was to have an automated process that only ran
on known, sane patches. I don't think he was suggesting a mechanism for
the great unwashed masses to dump arbitrary code into and have it
applied in the buildfarm. You'd have an inventory of patches (you could
use a hash to ensure they hadn't changed just before they ar
automatically applied) that were verified as good, and the system would
apply them to HEAD periodically.

Even if the patch inventory wasn't kept right up to date, this system
could potentially help many regression issues or bugs to surface sooner,
and as it would require zero work once set up besides system maintenance
(which should be low if it is implemented in a reasonably intelligent
manner) I feel that it is a great idea. Generally, I am all for
automating mundane tasks as much as possible.

Regards,
- Naz.

Andrew Dunstan wrote:

"Naz Gassiep" <naz@mira.net> writes:

Even if the patch inventory wasn't kept right up to date, this system
could potentially help many regression issues or bugs to surface sooner,

I just don't understand what this would accomplish. People run regressions
before submitting anyways. They can't run them on all architectures but bugs
that only affect some architectures are uncommon.

This seems to be merely institutionalizing having a large backlog of patches
which survive for long periods of time. But even in that situation I don't see
what it buys us. Detecting bitrot isn't terribly helpful and it doesn't help
us actually deal with the bitrot once it's happened.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Bruce Momjian wrote:

Sounds interesting, but I am not sure how that is going to track
multiple versions of the patch,

They could easily be submitted through the web interface as revisions to
the original version.

or changes in the email subject.

We'd need to keep the reference number in there, but other than that it
could change. We could also examine the in-reply-to header of every
email and attach based on that, but I'm not sure that's necessary.

The bottom line is that there is a lot of thinking that the patch queue
is so large because no one knows what to do. "Oh, if we were better
communicators, more would be done". The patch queue is large because we
have lots of March 31 patches, and because we don't have enough people
to review them quickly.

Thats is true, but there are also patches in there that have been there
for quite some time adn have had little or no feedback. Consider
Heikki's grouped index items patch
(http://momjian.us/mhonarc/patches/msg00032.html). That thread starts on
7th March when he posted an update because the old one had bitrotted.
There are six messages in the thread on the patch queue, four of which
say "message no available", and the last means little without the
preceeding messages.

When a committer comes to look at this, if he's not sure of something
he's going to be wasting time searching the archives to find all the
different thread fragments that make up the various discussions on the
topic, and those related to each individual revision of the patch. That
has got to be part of what makes reviewing patches both hard, and
uninteresting work. If we can make it easier and quicker, even with just
the current committers we might have found that one of you were able
(and keen) to review much more quickly.

The people who have expressed interest in reviewing patches already know
where we stand on the patches, and a status email of where we are each
patch will be posted shortly. I just can't do it this time because I am
traveling.

This is precisely the sort of trivial task that I believe we can make
much easier for you, if not eliminate altogether.

If you want to try a tracking system, go ahead, just pull from the
patches email list, and somehow try to grab discussion from
hackers/patches on this patches, and give a way to manually update the
patch status via a web site.

OK, I will give it some thought. Obviously I'm not going to do much more
before release though.

If your system works, I will not need to
maintain a separate patches queue, but I will keep doing it until we
know the web site idea will work, just in case.

I would expect nothing less :-)

Regards, Dave.

Two more ideas for the manager, now that we seem to have consensus to
build one.

On Apr 30, 2007, at 6:04 PM, Josh Berkus wrote:

-- We could save the patches by applied date and index them, and
then have a
place to point users when they ask: "When was X fixed? Do I *have* to
upgrade to 8.1 or just 8.0?"

This should also make doing the release notes substantially easier,
though since there's probably some stuff that wouldn't go through the
patch queue we'd want commits from the patch queue to be marked
differently.

That brings up another idea for the patch management webapp...
presumably it could handle most/all of the process of actually
committing a patch. Granted, that's not a huge amount of work, but
it's silly to have committers do by hand that hich could be automated.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Dave Page wrote:

The bottom line is that there is a lot of thinking that the patch queue
is so large because no one knows what to do. "Oh, if we were better
communicators, more would be done". The patch queue is large because we
have lots of March 31 patches, and because we don't have enough people
to review them quickly.

Thats is true, but there are also patches in there that have been there
for quite some time adn have had little or no feedback. Consider
Heikki's grouped index items patch
(http://momjian.us/mhonarc/patches/msg00032.html). That thread starts on
7th March when he posted an update because the old one had bitrotted.
There are six messages in the thread on the patch queue, four of which
say "message no available", and the last means little without the
preceeding messages.

When a committer comes to look at this, if he's not sure of something
he's going to be wasting time searching the archives to find all the
different thread fragments that make up the various discussions on the
topic, and those related to each individual revision of the patch. That
has got to be part of what makes reviewing patches both hard, and
uninteresting work. If we can make it easier and quicker, even with just
the current committers we might have found that one of you were able
(and keen) to review much more quickly.

The bottom line is if you had a system that was 100% perfect in
capturing all information about a patch, it only helps us 2% toward
reviewing the patch, and what is the cost of keeping 100% information?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Naz Gassiep wrote:

I believe the suggestion was to have an automated process that only ran
on known, sane patches.

How do we know in advance of reviewing them that they are sane?

What is more, we often run into situations where patch a will require
changes in patch b, so testing them individually against CVS is not
likely to be terribly useful.

Frankly, our problems are not primarily technological. They have to do
mainly with scarcity of available time from competent reviewers. No
amount of automation will fix that.

cheers

andrew

On Tue, 2007-01-05 at 02:54 +0100, Gregory Stark wrote:

"Naz Gassiep" <naz@mira.net> writes:

Even if the patch inventory wasn't kept right up to date, this system
could potentially help many regression issues or bugs to surface sooner,

I just don't understand what this would accomplish. People run regressions
before submitting anyways. They can't run them on all architectures but bugs
that only affect some architectures are uncommon.

The intention is to help keep patches, which may remain in the queue for
extended lengths of time, reasonably current. The mechanism aims to
help with these specific problems:

- patches accumulates bitrot and are consequently harder to apply and
understand
- the author, by the time review occurs, no longer has the details of
the patch uppermost in their mind

If the author can be automatically prodded when the patch no longer
cleanly applies, or when it suddenly breaks regression tests, they will
be able to keep the patch current, may discover bugs in it themselves
prior to review, and it will remain more fresh in their minds.

For sure, there will be classes of patch for which this mechanism
provides no benefit. For instance, where a patch contains code that is
for discussion only, or a patch that is dependant on another patch. In
these cases, the mechanism would simply note that they don't apply
cleanly, or don't pass tests, and would do nothing further. I can see
no harm here.

This seems to be merely institutionalizing having a large backlog of patches
which survive for long periods of time. But even in that situation I don't see
what it buys us. Detecting bitrot isn't terribly helpful and it doesn't help
us actually deal with the bitrot once it's happened.

I hope that it would not encourage reviewers to leave things in the
patch queue. I don't see why it would, so don't think this would
institutionalize a backlog.

I also disagree that detecting bitrot is not helpful. If I had eagerly
submitted a patch, I would definitely want to fix any bitrot that
occurred and would be thankful to be automatically informed.

And to clarify, I do not think the buildfarm should be involved in this
process.

__
Marc

Bruce Momjian wrote:

The bottom line is if you had a system that was 100% perfect in
capturing all information about a patch, it only helps us 2% toward
reviewing the patch, and what is the cost of keeping 100% information?

2% for you or Tom reviewing a recently discussed, run-of-the mill patch.
I suspect that %age will rise as the patch complexity increases and the
reviewers experience decreases - which is exactly the situation that it
would help to improve.

Also note that I'm not saying I can produce a system that's 100% correct
- just one that will capture the posts that keep the patch ID in their
subject line *automatically* - meaning you don't have to worry about
keeping threads for the existing queue or tracking the patch status.

Regards Dave

Bruce,

The bottom line is if you had a system that was 100% perfect in
capturing all information about a patch, it only helps us 2% toward
reviewing the patch, and what is the cost of keeping 100% information?

2% for you or Tom reviewing a recently discussed, run-of-the mill patch.
I suspect that %age will rise as the patch complexity increases and the
reviewers experience decreases - which is exactly the situation that it
would help to improve.

Moreover, what I'm looking for is tools which will:
1) allow existing reviewers to make better use of the time that they have, and
2) encourage/assist new reviewers in helping out, and
3) not bottleneck on the availability of a single project member

The current patch-queue process is failing to scale with the project: every
release it gets to be more work for you & Tom to integrate the patches. We
need to think of new approaches to make the review process scale. As a
pointed example, you're about to go on tour for 2 weeks and patch review will
stall while you're gone. That's not sustainable.

If you don't think that a web tool will help, then what *do* you think will
help? Just "soldiering on" isn't really an answer, and I notice that you're
very quick to come up with reasons why anything we might try will fail, but
extremely reluctant to make suggestions for improvement.

==============

Dave,

Also note that I'm not saying I can produce a system that's 100% correct
- just one that will capture the posts that keep the patch ID in their
subject line *automatically* - meaning you don't have to worry about
keeping threads for the existing queue or tracking the patch status.

Is there a reason why the system needs to be primarily based on e-mail? I was
thinking that the patch manager would be entirely a web tool, with people
submitting and modifying a patch directly through a web interface. This
would be lots easier to build than an e-mail based system, and also far more
useful from a monitoring standpoint. I've worked with e-mail based systems
like RT and OTRS, and frankly they're extremely high-maintenance and suffer a
large amount of "lost" information.

We could also build a number of other things into the web tool, like a "You
are submitting this patch under BSD" disclaimer and pointers to the Developer
FAQ and other relevant documents.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh,

Josh Berkus wrote:

Is there a reason why the system needs to be primarily based on e-mail? I was
thinking that the patch manager would be entirely a web tool, with people
submitting and modifying a patch directly through a web interface. This
would be lots easier to build than an e-mail based system, and also far more
useful from a monitoring standpoint. I've worked with e-mail based systems
like RT and OTRS, and frankly they're extremely high-maintenance and suffer a
large amount of "lost" information.

The reason for basing the system on email is simply that it minimises
the changes required in the community process. If it were entirely web
based, we'd have to change the way we all work to discuss patches in a
forum style, rather than a list style. I have a sneaking suspicion that
at least one of our most valued contributors might object to that.

As long as the patch were initially submitted through the web interface
so that it got assigned an ID, we could automatically track the initial,
and followup threads on any of the lists as long as the ID is retained
in the subject line.

We could also build a number of other things into the web tool, like a "You
are submitting this patch under BSD" disclaimer and pointers to the Developer
FAQ and other relevant documents.

Oh for sure. We could even do silly stuff like try to automatically
determine if the patch is in diff -c format ;-)

Regards, Dave

Dave,

The reason for basing the system on email is simply that it minimises
the changes required in the community process. If it were entirely web
based, we'd have to change the way we all work to discuss patches in a
forum style, rather than a list style. I have a sneaking suspicion that
at least one of our most valued contributors might object to that.

Hmmm, yeah, I can see. However, I think it's possible to separate the patch
discussion from patch submission; that is, to submit/edit/update patch code
through the interface, but to do the discussion either through e-mail or the
interface.

As long as the patch were initially submitted through the web interface
so that it got assigned an ID, we could automatically track the initial,
and followup threads on any of the lists as long as the ID is retained
in the subject line.

Yes, and any changes to the patch code itself, as well. My concern with
making the tool e-mail centric is that, based on e-mail based tools I've
worked with, I'm afraid that the tool will be clunky/buggy enough not to be
an improvement over the current process -- too much of e-mail format is
optional varies by MUA. If e-mail is limited to commentary, though, I don't
see that as a problem. And, of course, it would be easy for the tool to send
e-mail to pgsql-patches.

If many people are going to block on using a web tool for submitting new
versions of a patch, claiming responsibility for review, etc., though, then
we should probably abandon this discussion right here. No new tool is going
to work if we have people who won't make any changes at all in their work
habits.

Oh for sure. We could even do silly stuff like try to automatically
determine if the patch is in diff -c format ;-)

No! Dave, you're a real radical sometimes.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

On Tue, 2007-05-01 at 09:43 -0700, Josh Berkus wrote:

The current patch-queue process is failing to scale with the project: every
release it gets to be more work for you & Tom to integrate the patches. We
need to think of new approaches to make the review process scale. As a
pointed example, you're about to go on tour for 2 weeks and patch review will
stall while you're gone. That's not sustainable.

This seems a reasonable observation on events.

I'll come up with ideas to help, if asked, but I'd like to follow a
proposal from Core on this issue.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

Josh Berkus wrote:

If many people are going to block on using a web tool for submitting new
versions of a patch, claiming responsibility for review, etc., though, then
we should probably abandon this discussion right here. No new tool is going
to work if we have people who won't make any changes at all in their work
habits.

We could do most of what people want using a tracker, but that
discussion always seems to end up nowhere. And, as I have noted before,
use of a tracker will become a total mess unless there are adequate
resources to keep it clean and up to date (e.g. to put links in items to
mailing list discussions, update state etc.) So if the commercial
backers of PostgreSQL want better management of the project, maybe they
need to find some resources to help out.

cheers

andrew

Josh Berkus wrote:

Dave,

The reason for basing the system on email is simply that it minimises
the changes required in the community process. If it were entirely web
based, we'd have to change the way we all work to discuss patches in a
forum style, rather than a list style. I have a sneaking suspicion that
at least one of our most valued contributors might object to that.

Hmmm, yeah, I can see. However, I think it's possible to separate the patch
discussion from patch submission; that is, to submit/edit/update patch code
through the interface, but to do the discussion either through e-mail or the
interface.

I'd just keep all discussion in email, no need to do that off the web.
As long as you can *read* all the references on the web.

As long as the patch were initially submitted through the web interface
so that it got assigned an ID, we could automatically track the initial,
and followup threads on any of the lists as long as the ID is retained
in the subject line.

Yes, and any changes to the patch code itself, as well. My concern with
making the tool e-mail centric is that, based on e-mail based tools I've
worked with, I'm afraid that the tool will be clunky/buggy enough not to be
an improvement over the current process -- too much of e-mail format is
optional varies by MUA. If e-mail is limited to commentary, though, I don't
see that as a problem. And, of course, it would be easy for the tool to send
e-mail to pgsql-patches.

I believe that's what Dave is proposing. If we want to implement
something like "reviewer signoff" (which we'd at least eventually want
to do, IMHO), doing that through the web interface primarily (for a nice
"tally" table) is probably a good start - it's a lot easier than parsing
emails.

//Magnus

Andrew,

So if the commercial
backers of PostgreSQL want better management of the project, maybe they
need to find some resources to help out.

I don't think they really care, or we'd have heard something by now. I
think this is up to us PG developers.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus wrote:

Andrew,

So if the commercial
backers of PostgreSQL want better management of the project, maybe they
need to find some resources to help out.

I don't think they really care, or we'd have heard something by now. I
think this is up to us PG developers.

Well, I have no confidence that any formal system will succeed without
someone trusted by core and committers stepping up to the plate to do
the required ongoing legwork.

As for voting on patches, that seems a most un-postgres-like way of
doing things. What is more, it assumes that multiple people will be
reviewing patches. Our trouble right now is finding even one qualified
reviewer with enough time for some patches.

cheers

andrew

------- Original Message -------
From: Andrew Dunstan <andrew@dunslane.net>
To: Josh Berkus <josh@agliodbs.com>
Sent: 01/05/07, 21:10:07
Subject: Re: [HACKERS] Feature freeze progress report

So if the commercial
backers of PostgreSQL want better management of the project, maybe they
need to find some resources to help out.

I'm not proposing this as an EDB employee, but as a community member who has heard the same concerns from various PostgreSQL developers both in and out of EDB. And yes, I am expecting to put some resource into making this work.

Regards, Dave

Josh Berkus wrote:

Bruce,

The bottom line is if you had a system that was 100% perfect in
capturing all information about a patch, it only helps us 2% toward
reviewing the patch, and what is the cost of keeping 100% information?

2% for you or Tom reviewing a recently discussed, run-of-the mill patch.
I suspect that %age will rise as the patch complexity increases and the
reviewers experience decreases - which is exactly the situation that it
would help to improve.

Moreover, what I'm looking for is tools which will:
1) allow existing reviewers to make better use of the time that they have, and
2) encourage/assist new reviewers in helping out, and
3) not bottleneck on the availability of a single project member

The current patch-queue process is failing to scale with the project: every
release it gets to be more work for you & Tom to integrate the patches. We
need to think of new approaches to make the review process scale. As a
pointed example, you're about to go on tour for 2 weeks and patch review will
stall while you're gone. That's not sustainable.

I am traveling --- I left on Friday. I am in Sydney now.

As far as scaling, patch information isn't our problem right now. If
someone wants to help we can give them up-to-date information on exactly
how to deal with the patch. It is people doing the work that is the
bottleneck.

If you don't think that a web tool will help, then what *do* you think will
help? Just "soldiering on" isn't really an answer, and I notice that you're
very quick to come up with reasons why anything we might try will fail, but
extremely reluctant to make suggestions for improvement.

Well, if I had something better, I would be doing it already.

There isn't an improved solution to every problem.

We have this discussion regularly --- things aren't going well, so there
must be a better way. I know things aren't going well because I created
this thread, but I don't see collecting patch information as solving
this issue. What we actually need are more people doing things,
"soldiering on".

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

FYI, Tom, Heikki, I need one of you to post the list of patches and
where we think we are on each one, even if the list is imperfect.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan wrote:

Josh Berkus wrote:

Andrew,

So if the commercial
backers of PostgreSQL want better management of the project, maybe they
need to find some resources to help out.

I don't think they really care, or we'd have heard something by now. I
think this is up to us PG developers.

Well, I have no confidence that any formal system will succeed without
someone trusted by core and committers stepping up to the plate to do
the required ongoing legwork.

As for voting on patches, that seems a most un-postgres-like way of
doing things. What is more, it assumes that multiple people will be
reviewing patches. Our trouble right now is finding even one qualified
reviewer with enough time for some patches.

The typical use-case is that someone is going to like the patch, but
what X changed in it, so a simple vote isn't going to work, and neither
is automatic patch application. Rarely is a patch applied unmodified by
the applier.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

In article <7821E6AA-E46D-46E1-95FC-33BD16085532@decibel.org>,
decibel@decibel.org (Jim Nasby) wrote:

Two more ideas for the manager, now that we seem to have consensus to
build one.

One other thing a webapp would allow that would help grow the community.
If the patches are all in a public place then reviewer wannabees can get
their feet wet relatively easily.

Some may argue this is already possible but I, personally, don't even
know where to look for patches.

-arturo

Andrew Dunstan wrote:

Naz Gassiep wrote:

I believe the suggestion was to have an automated process that only ran
on known, sane patches.

How do we know in advance of reviewing them that they are sane?

Same way as happens now. I would assume this mechanism would only be
applied to patches that had already been approved to contrib, or some
other measure that can be used to isolate only those patches that we
*expect* to already be working. The intention of this mechanism, in my
head, is to just help us make sure that regression issues on patches get
detected sooner.

What is more, we often run into situations where patch a will require
changes in patch b, so testing them individually against CVS is not
likely to be terribly useful.

Yeap, given that this proposition is for an automated system, perhaps it
could be designed to apply combinations of patches together to look for
conflicts.

Frankly, our problems are not primarily technological. They have to do
mainly with scarcity of available time from competent reviewers. No
amount of automation will fix that.

I fully understand that. However I find the idea of an automated process
checking for big issues while we're all sleeping to be... sexy. I'm not
sure how difficult a system like this would be to set up but it doesn't
seem to me to be the sort of thing that requires more than a few simple
scripts. If it's not too had to set up, even if it only yields small and
rare benefits, it will have been a worthwhile exercise.

My 2c (adjusted for inflation).

Regards,
- Naz

Bruce,

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

The idea is to provide ways for other people to help where they can and to
provide better feedback to patch submitters so that they fix their own issues
faster. Also, lesser PostgreSQL hackers than you could take on reviewing the
"small" patches, leaving you to devote all of your attention to the "big"
patches.

Actually, that can happen with the current system. The real blocker there is
that some people, particularly Tom, work so fast that there's no chance for a
new reviewer to tackle the easy stuff. Maybe the real solution is to
encourage some of our other contributors to get their feet wet with easy
patches so that they can help with the big ones later on?

That is, if the problem is people and not tools, then what are we doing to
train up the people we need?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Naz Gassiep <naz@mira.net> writes:

Andrew Dunstan wrote:

How do we know in advance of reviewing them that they are sane?

Same way as happens now. I would assume this mechanism would only be
applied to patches that had already been approved to contrib, or some
other measure that can be used to isolate only those patches that we
*expect* to already be working.

What is "approved to contrib"?

The problem here is that having reasonable certainty that a patch is
not malicious requires having gone over it in some detail; at which
point you might as well apply the thing. Or if you didn't apply it,
you bounced it for reasons that are unlikely to have anything to do
with needing more automated testing.

ISTM this idea can only work if we have a "second tier" of reviewers
who are considered good enough to vet patches as safe, but not quite
good enough to certify them as commitable. I'm not seeing a large pool
of people volunteering to hold that position --- at best it'd be a
transitory state before attaining committerdom. If you are relying
on a constant large influx of new people, you are doomed to failure
(see "Ponzi scheme" for a counterexample).

regards, tom lane

Josh Berkus <josh@agliodbs.com> writes:

Actually, that can happen with the current system. The real blocker there is
that some people, particularly Tom, work so fast that there's no chance for a
new reviewer to tackle the easy stuff. Maybe the real solution is to
encourage some of our other contributors to get their feet wet with easy
patches so that they can help with the big ones later on?

Yeah, I hear what you say. This is particularly a problem for small bug
fixes: I tend to zing small bugs quickly, first because I enjoy finding/
fixing them and second because I worry that they'll fall off the radar
screen if not fixed. But I am well aware that fixing those sorts of
issues is a great way to learn your way around the code (I think that's
largely how I learned whatever I know about Postgres). I'd be more
willing to stand aside and let someone else do it if I had confidence
that issues wouldn't get forgotten. So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

regards, tom lane

What is "approved to contrib"?

The problem here is that having reasonable certainty that a patch is
not malicious requires having gone over it in some detail; at which
point you might as well apply the thing. Or if you didn't apply it,
you bounced it for reasons that are unlikely to have anything to do
with needing more automated testing.

ISTM this idea can only work if we have a "second tier" of reviewers
who are considered good enough to vet patches as safe, but not quite
good enough to certify them as commitable. I'm not seeing a large pool
of people volunteering to hold that position --- at best it'd be a
transitory state before attaining committerdom. If you are relying
on a constant large influx of new people, you are doomed to failure
(see "Ponzi scheme" for a counterexample).

Yep. For the record, Ponzi died in poverty, so it's not a counter
example, just proves that any gains that are had will be short lived and
increase the size of the crash when crunch time comes. :)
- Naz.

On Tue, May 01, 2007 at 07:09:24PM -0400, Bruce Momjian wrote:

The current patch-queue process is failing to scale with the project: every
release it gets to be more work for you & Tom to integrate the patches. We
need to think of new approaches to make the review process scale. As a
pointed example, you're about to go on tour for 2 weeks and patch review will
stall while you're gone. That's not sustainable.

I am traveling --- I left on Friday. I am in Sydney now.

As far as scaling, patch information isn't our problem right now. If
someone wants to help we can give them up-to-date information on exactly
how to deal with the patch. It is people doing the work that is the
bottleneck.

<snip>

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

FYI, Tom, Heikki, I need one of you to post the list of patches and
where we think we are on each one, even if the list is imperfect.

I think you just contradicted yourself. Information isn ot the problem, but
you need more information...

I think this is a fairly clear indication that we do need a better way to
track this information.

//Magnus

Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Actually, that can happen with the current system. The real blocker there is
that some people, particularly Tom, work so fast that there's no chance for a
new reviewer to tackle the easy stuff. Maybe the real solution is to
encourage some of our other contributors to get their feet wet with easy
patches so that they can help with the big ones later on?

Yeah, I hear what you say. This is particularly a problem for small bug
fixes: I tend to zing small bugs quickly, first because I enjoy finding/
fixing them and second because I worry that they'll fall off the radar
screen if not fixed. But I am well aware that fixing those sorts of
issues is a great way to learn your way around the code (I think that's
largely how I learned whatever I know about Postgres). I'd be more
willing to stand aside and let someone else do it if I had confidence
that issues wouldn't get forgotten. So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

A great way to learn would be to look at the patches in the queue, and
find bugs in them. There's a lot more bugs to be found in submitted
patches than in PostgreSQL itself. A patch tracker would help with that.

I'm in favor of some kind of a patch tracker. It doesn't need to be too
fancy, but if for each patch we had:

Patch name: Kitchen sink addition to planner
Latest patch: kitchen-sink-v123.patch, click to download
Summary: Adds a kitchen-sink node type to the planner to enable liquid
queries.
Status: Will be rejected unless race conditions are fixed. Needs
performance testing.
Discussions: <links to mail threads, like in the current patch queue>

That wouldn't necessarily help committers directly, but it'd give more
visibility to the patches. That would encourage more people to review
and test patches. And it'd make it clear what the status of all the
patches are to anyone who's interested.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Magnus Hagander wrote:

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

FYI, Tom, Heikki, I need one of you to post the list of patches and
where we think we are on each one, even if the list is imperfect.

I think you just contradicted yourself. Information isn ot the problem, but
you need more information...

I think this is a fairly clear indication that we do need a better way to
track this information.

No, my point is that 100% information is already available by looking at
email archives. What we need is a short description of where we are on
each patch --- that is a manual process, not something that can be
automated.

Tom has posted it --- tell me how we will get such a list in an
automated manner.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Josh Berkus wrote:

Bruce,

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

The idea is to provide ways for other people to help where they can and to
provide better feedback to patch submitters so that they fix their own issues
faster. Also, lesser PostgreSQL hackers than you could take on reviewing the
"small" patches, leaving you to devote all of your attention to the "big"
patches.

Actually, that can happen with the current system. The real blocker there is
that some people, particularly Tom, work so fast that there's no chance for a
new reviewer to tackle the easy stuff. Maybe the real solution is to
encourage some of our other contributors to get their feet wet with easy
patches so that they can help with the big ones later on?

That is, if the problem is people and not tools, then what are we doing to
train up the people we need?

We seem to handle trivial patches just fine. The current problem is
that the remaining patches require domain or subsystem-specific
knowledge to apply, e.g. XML or WAL, and those skills are available in a
limited number of people. If I had the expertise in those areas, I
would have applied the patches already.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce@momjian.us> writes:

We seem to handle trivial patches just fine.

You keep saying that but I think it's wrong. There are trivial patches that
were submitted last year that are still sitting in the queue.

In fact I claim we handle complex patches better than trivial ones. HOT, LDC,
DSM etc receive tons of feedback and acquire a momentum of their own.
Admittedly GII is a counter-example though.

On the other hand trivial patches tend to interest relatively few people and
have little urgency.

The current problem is that the remaining patches require domain or
subsystem-specific knowledge to apply, e.g. XML or WAL, and those skills are
available in a limited number of people. If I had the expertise in those
areas, I would have applied the patches already.

Well, I claim it's often the trivial patches that require the domain-specific
knowledge you describe. If they were major patches they would touch more parts
of the system. But that means they should be easy to commit if you could just
fill in the missing knowledge.

Could you pick a non-committer with the domain-specific knowledge you think a
patch needs and ask for their analysis of the patch then commit it yourself?
You can still review it for general code quality and trust the non-committer's
review of whether the domain-specific change is correct.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Gregory Stark wrote:

"Bruce Momjian" <bruce@momjian.us> writes:

We seem to handle trivial patches just fine.

You keep saying that but I think it's wrong. There are trivial patches that
were submitted last year that are still sitting in the queue.

You seem to be looking at something different than me. Which patches?

In fact I claim we handle complex patches better than trivial ones. HOT, LDC,
DSM etc receive tons of feedback and acquire a momentum of their own.
Admittedly GII is a counter-example though.

Well, I claim it's often the trivial patches that require the domain-specific
knowledge you describe. If they were major patches they would touch more parts
of the system. But that means they should be easy to commit if you could just
fill in the missing knowledge.

Could you pick a non-committer with the domain-specific knowledge you think a
patch needs and ask for their analysis of the patch then commit it yourself?
You can still review it for general code quality and trust the non-committer's
review of whether the domain-specific change is correct.

We are already pushing out patches to people with domain-specific
knowledge. Tom posted that summary today.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Naz Gassiep wrote:

Andrew Dunstan wrote:

Naz Gassiep wrote:

I believe the suggestion was to have an automated process that only ran
on known, sane patches.

How do we know in advance of reviewing them that they are sane?

Same way as happens now.

The question was rhetorical ... there is no list of "certified sane but
unapplied" patches. You are proceeding on the basis of a faulty
understanding of how our processes work.

cheers

andrew

Tom Lane wrote:

So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

Hallelujah Brother!

BTW, a bug tracker can be used as a patch tracker, although the reverse
isn't true. For example, the BZ people use BZ that way, in fact - most
patches arrive as attachments to bugs. And trackers can be used just as
well for tracking features as well as bugs.

cheers

andrew

andrew@dunslane.net (Andrew Dunstan) writes:

Tom Lane wrote:

So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

Hallelujah Brother!

BTW, a bug tracker can be used as a patch tracker, although the
reverse isn't true. For example, the BZ people use BZ that way, in
fact - most patches arrive as attachments to bugs. And trackers can be
used just as well for tracking features as well as bugs.

Well, Command Prompt set up a Bugzilla instance specifically so people
could track PG bugs. If only someone took interest and started using
it...
--
"cbbrowne","@","linuxfinances.info"
http://cbbrowne.com/info/lisp.html
Do Roman paramedics refer to IV's as "4's"?

Andrew Dunstan wrote:

Tom Lane wrote:

So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

Hallelujah Brother!

Amen

BTW, a bug tracker can be used as a patch tracker, although the reverse
isn't true. For example, the BZ people use BZ that way, in fact - most
patches arrive as attachments to bugs. And trackers can be used just as
well for tracking features as well as bugs.

The pidgin (previously known as Gaim) guys also use it that way. They
add a bug for each thing they want to change, even new features, and
track the patches in there. Then they have a list of issues that should
be solved for each release, so it's easy to see which ones are still
missing using their Trac interface.

http://developer.pidgin.im/roadmap

So the status email that Tom sent yesterday would be a very simple thing
to generate, just looking at the "bugs to fix" page.

I'm not saying we should use Trac, mainly because I hate how it
(doesn't) interact with email. But it does say that a bug tracker can
be useful to us.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Wed, 2007-02-05 at 08:27 -0400, Andrew Dunstan wrote:

Naz Gassiep wrote:

Andrew Dunstan wrote:

Naz Gassiep wrote:

I believe the suggestion was to have an automated process that only ran
on known, sane patches.

How do we know in advance of reviewing them that they are sane?

Same way as happens now.

The question was rhetorical ... there is no list of "certified sane but
unapplied" patches. You are proceeding on the basis of a faulty
understanding of how our processes work.

Why do we need to know the patch is sane? If it does not apply cleanly
or causes regression tests to fail, the process would figure that out
quickly and cheaply. There is little cost in attempting to apply a
non-sane patch.

I am not sure that I have explained exactly what I was suggesting. Some
people seem to grok it, others seem to be talking something slightly
different. To clarify, here it is in pseudo-code:

for each patch in the queue
regression_success := false
patch_success := attempt to apply patch to head
if patch_success
regression_success := attempt to run regression tests
-- (On one machine only, not on the buildfarm)
end if
if this is a new patch
maybe mail the author and tell them patch_success and
regression_success
else
if status is different from last time
mail the author and tell them their patch has changed status
end
end
record the status for this patch
end loop

__
Marc

On Wed, May 02, 2007 at 06:44:12AM -0400, Bruce Momjian wrote:

Magnus Hagander wrote:

As an example, how is patch information going to help us review HOT or
group-item-index? There is frankly more information about these in the
archives than someone could reasonable read. What someone needs is a
summary of where we are now on the patches, and lots of time.

FYI, Tom, Heikki, I need one of you to post the list of patches and
where we think we are on each one, even if the list is imperfect.

I think you just contradicted yourself. Information isn ot the problem, but
you need more information...

I think this is a fairly clear indication that we do need a better way to
track this information.

No, my point is that 100% information is already available by looking at
email archives.

Yes, and hard to find.

What we need is a short description of where we are on
each patch --- that is a manual process, not something that can be
automated.

Right. But it can be presented in a central way and incrementally updated.

Tom has posted it --- tell me how we will get such a list in an
automated manner.

There's no way to get it automated, but you can get it incrementally
updated.

//Magnus

Bruce, all,

No, my point is that 100% information is already available by looking at
email archives. What we need is a short description of where we are on
each patch --- that is a manual process, not something that can be
automated.

Tom has posted it --- tell me how we will get such a list in an
automated manner.

Several of us have already suggested a method. If we want the information to
be up-to-date, then the patch manager, or bug tracker, needs to be a required
part of the approval & application process, NOT an optional accessory. That
is, if patches & bug fixes can come in, get modified, get approved & applied
entirely on pgsql-patches or pgsql-bugs without ever touching the tracker
tool, then the tracker tool will be permanently out of date and useless.

It's going to require the people who are doing the majority of the bug hunting
& patch review to change the way they work, with the idea that any extra time
associated with the new tool will be offset by being able to spread the work
more and having information easy to find later, for you as well as others.
Tom seems to be willing; are you?

====================

Status: Will be rejected unless race conditions are fixed. Needs
performance testing.
Discussions: <links to mail threads, like in the current patch queue>

... this brings up another reason we could use a tracker. I now have access
to a performance testing lab and staff. However, these people are NOT going
to follow 3 different high-traffic mailing lists in order to keep up with
which patches to test. As a result, they haven't done much testing of 8.3
patches; they're depenant on me to keep them updated on new patch versions
and known issues and I'm on the road a lot.

If I had a web tool I could point them to where they could simply download the
current version of the patch, test, and comment a report, we'd get a LOT more
useful performance feedback from Sun. I suspect the same is true of Unisys.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus wrote:

Bruce, all,

No, my point is that 100% information is already available by looking at
email archives. What we need is a short description of where we are on
each patch --- that is a manual process, not something that can be
automated.

Tom has posted it --- tell me how we will get such a list in an
automated manner.

Several of us have already suggested a method. If we want the information to
be up-to-date, then the patch manager, or bug tracker, needs to be a required
part of the approval & application process, NOT an optional accessory. That
is, if patches & bug fixes can come in, get modified, get approved & applied
entirely on pgsql-patches or pgsql-bugs without ever touching the tracker
tool, then the tracker tool will be permanently out of date and useless.

It's going to require the people who are doing the majority of the bug hunting
& patch review to change the way they work, with the idea that any extra time
associated with the new tool will be offset by being able to spread the work
more and having information easy to find later, for you as well as others.
Tom seems to be willing; are you?

Hello,

Well according to himself the last time this came up:

http://archives.postgresql.org/pgsql-hackers/2006-08/msg01253.php

No, he isn't.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Marc Munro wrote:

On Wed, 2007-02-05 at 08:27 -0400, Andrew Dunstan wrote:

Naz Gassiep wrote:

Andrew Dunstan wrote:

Naz Gassiep wrote:

I believe the suggestion was to have an automated process that only

ran

on known, sane patches.

How do we know in advance of reviewing them that they are sane?

Same way as happens now.

The question was rhetorical ... there is no list of "certified sane but
unapplied" patches. You are proceeding on the basis of a faulty
understanding of how our processes work.

Why do we need to know the patch is sane?

Because not doing so is dangerous and a major security hole. I won't run
arbitrary code on my machine and I won't create infrastructure (e.g.
buildfarm) to get others to do it either.

You are also conveniently ignoring all the other reasons why this won't
help anyone much (e.g. see Bruce's comments upthread).

cheers

andrew

Chris Browne wrote:

andrew@dunslane.net (Andrew Dunstan) writes:

Tom Lane wrote:

So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

Hallelujah Brother!

BTW, a bug tracker can be used as a patch tracker, although the
reverse isn't true. For example, the BZ people use BZ that way, in
fact - most patches arrive as attachments to bugs. And trackers can be
used just as well for tracking features as well as bugs.

Well, Command Prompt set up a Bugzilla instance specifically so people
could track PG bugs. If only someone took interest and started using
it...

I lost interest last time around when it seemed clear to me that there was
not enough traction.

Maybe there is this time around.

The effort Tom mentions above is crucial to success.

cheers

andrew

Marc Munro <marc@bloodnok.com> writes:

On Wed, 2007-02-05 at 08:27 -0400, Andrew Dunstan wrote:

The question was rhetorical ... there is no list of "certified sane but
unapplied" patches. You are proceeding on the basis of a faulty
understanding of how our processes work.

Why do we need to know the patch is sane? If it does not apply cleanly
or causes regression tests to fail, the process would figure that out
quickly and cheaply. There is little cost in attempting to apply a
non-sane patch.

Unless it contains a trojan horse. I don't think many buildfarm owners
are running the tests inside a sandbox so tight that they don't care
how nasty the code that runs there might be.

regards, tom lane

Joshua D. Drake wrote:

Well according to himself the last time this came up:

http://archives.postgresql.org/pgsql-hackers/2006-08/msg01253.php

No, he isn't.

The last paragraph of
http://archives.postgresql.org/pgsql-hackers/2007-04/msg01258.php is
somewhat more positive regarding a patch tracker.

Regards, Dave

Gregory Stark <stark@enterprisedb.com> writes:

You keep saying that but I think it's wrong. There are trivial patches that
were submitted last year that are still sitting in the queue.

Um ... which ones exactly? I don't see *anything* in the current queue
that is utterly without issues, other than Heikki's ReadOrZeroBuffer
patch which certainly doesn't date from last year (and besides, has
now been applied).

I'm a bit more worried about stuff that may have slid through the
cracks, like the Darwin SysV-vs-Posix-semaphores patch that I complained
of in my triage listing. But the stuff that is listed on Bruce's patch
queue page is not "trivial". It's either large or has got problems.

regards, tom lane

"Tom Lane" <tgl@sss.pgh.pa.us> writes:

Gregory Stark <stark@enterprisedb.com> writes:

You keep saying that but I think it's wrong. There are trivial patches that
were submitted last year that are still sitting in the queue.

Um ... which ones exactly? I don't see *anything* in the current queue
that is utterly without issues

I didn't mean they were without issues. Only that they weren't complex patches
requiring broad knowledge of many parts of the system.

Anyways, I think we're well on our way to improving the situation so I don't
want to drag out my negativity any longer.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Josh Berkus wrote:

Bruce, all,

No, my point is that 100% information is already available by looking at
email archives. What we need is a short description of where we are on
each patch --- that is a manual process, not something that can be
automated.

Tom has posted it --- tell me how we will get such a list in an
automated manner.

Several of us have already suggested a method. If we want the information to
be up-to-date, then the patch manager, or bug tracker, needs to be a required
part of the approval & application process, NOT an optional accessory. That
is, if patches & bug fixes can come in, get modified, get approved & applied
entirely on pgsql-patches or pgsql-bugs without ever touching the tracker
tool, then the tracker tool will be permanently out of date and useless.

It's going to require the people who are doing the majority of the bug hunting
& patch review to change the way they work, with the idea that any extra time
associated with the new tool will be offset by being able to spread the work
more and having information easy to find later, for you as well as others.
Tom seems to be willing; are you?

No, I think it will be more work than benefit, and Tom and I will still
be doing the bulk of it, so we will have something pretty but more work
for people who are already too busy.

====================

Status: Will be rejected unless race conditions are fixed. Needs
performance testing.
Discussions: <links to mail threads, like in the current patch queue>

... this brings up another reason we could use a tracker. I now have access
to a performance testing lab and staff. However, these people are NOT going
to follow 3 different high-traffic mailing lists in order to keep up with
which patches to test. As a result, they haven't done much testing of 8.3
patches; they're depenant on me to keep them updated on new patch versions
and known issues and I'm on the road a lot.

If I had a web tool I could point them to where they could simply download the
current version of the patch, test, and comment a report, we'd get a LOT more
useful performance feedback from Sun. I suspect the same is true of Unisys.

This is so funny, I don't know how to respond, only to ask if Dtrace
will help us here. ;-) I admit having such a system would improve the
chances of commercial help by a miniscule percentage, but the cost to
patch submitters/managers is so high in proportion to be humorous.

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

To be more concrete, during normal development, we seem to have no
problem keeping track of patches, so it is only during feature freeze
that it is a problem. Now, everyone admits that having patches tracked
on a web interface is going to be more work for patch managers and
importantly also patch submitters. So do we do the web work just so we
can have a more orderly feature freeze, or do we just accept that Tom is
going to have to post a summary of where we are on patches periodically
during feature freeze and not do the web work for every patch?

I have added a link on the patches queue so you can download the emails
in mbox format. If someone wants to take that and make a nice web site
out of it and keep that up-to-date during our feature freeze period,
that would probably help. (I can even point the patch queue to a new
URL.) If no one is willing to do it, I question how much help we would
get with a web patch system.

Heck, if I was around, I would throw up a txt file on the web (using
txt2html) with the status of each patch and keep it current. I think I
have done that for some past releases. Why doesn't someone do that and
see how it goes? Thanks to Tom, you know have a current status of each
patch and who is supposed to work on it.

I am not anti-big-solution, but I don't want a big solution if it isn't
going to help. If no one is doing even a trivial solution, I don't want
to try a big one.

Get rid of gborg and let's talk.

Why am I having to spend hours in Syndey saying the same thing? Why
don't you guys go ahead and change things, and when they fail, I will
still be around.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

Bruce, I guess the analogy fails on the fact that you're not looking for
a key, but for people, and I thought "better light" will attract people
to find you instead of you to need to search...

I'm an outsider regarding postgres development, so my opinion does not
count, but the gut feeling is that more light attracts better people.
Not to mention it can help people get better...

Cheers,
Csaba.

Csaba Nagy wrote:

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

Bruce, I guess the analogy fails on the fact that you're not looking for
a key, but for people, and I thought "better light" will attract people
to find you instead of you to need to search...

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Thu, 2007-05-03 at 13:51, Bruce Momjian wrote:

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

Yes Bruce, but you're failing to see that a more structured information
infrastructure will attract more people to do the work which could
eventually solve the problem you're having in the first place
(contradicting your argument that it won't help), at the cost of some
possible additional work (which I actually think you're overestimating
the amount of - it's probably more like a learning curve than actual
additional work).

The fact that there is enough information is not relevant, it must be in
the right place too - too much information or hard to find information
is sometimes worst than no information.

Cheers,
Csaba.

Bruce Momjian wrote:

Csaba Nagy wrote:

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

Bruce, I guess the analogy fails on the fact that you're not looking for
a key, but for people, and I thought "better light" will attract people
to find you instead of you to need to search...

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

OK, so how do we attract more people if not by making the job easier for
them? It's not like there aren't plenty of very clever people here - we
just need to encourage them to chip in - and making the task less
daunting by making *all* the information they need readily available
seems a good start too me.

And heck, we're database people - storing and retrieving the data to do
a job is exactly what we do!

Regards, Dave.

Bruce Momjian wrote:

Csaba Nagy wrote:

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

Bruce, I guess the analogy fails on the fact that you're not looking for
a key, but for people, and I thought "better light" will attract people
to find you instead of you to need to search...

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

There are multiple problems.

Of course no amount of technology will provide us with a bigger pool of
qualified reviewers. That doesn't mean our present management methods
are good - you seem to be just about the only person left who thinks
they are. Moving from using a quill to using a fountain pen or even a
ballpoint doesn't mean we have more writers or better writers. But it
does make writing easier and is therefore a good thing to do.

cheers

andrew

Bruce Momjian wrote:

Csaba Nagy wrote:

We have _ample_ evidence that the problem is lack of people able to
review patches, and yet there is this discussion to track patches
better. It reminds me of someone who has lost their keys in an alley,
but is looking for them in the street because the light is better there.

Bruce, I guess the analogy fails on the fact that you're not looking for
a key, but for people, and I thought "better light" will attract people
to find you instead of you to need to search...

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

*sigh* Bruce, why is this so hard with you? Let's try this a different way.

In the beginning there was Archie. Archie was good but only people that
were intimately familiar with its use got much benefit.

Then there was Gopher. He was cool, got Bill Murray to do really bad
things to a golf course but he provided more structured information if
in a strictly hierarchical fashion. Again it was used, but only by those
in the know.

The came the World Wide Web. A fascinated World Wide Portal, with no
map... It brought, Yahoo!, Alta Vista and Lycos.

All of a sudden, anyone could find what they were looking for and in the
process more people joined the community.

The rest is history (and I think my timeline between archie and gopher
might be off) but the point is, everything grows up and reach a point
where they need a more structured, easy to find, easy to collaborate
method of processing data. Any data, not just patches or bugs.

You are stuck in Gopher land, I don't know why but it is at is is.
Perhaps it is type to at least jump to using Yahoo!?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Bruce,

Get rid of gborg and let's talk.

Touche'.

Actually, AFAICT, the only active thing left on GBorg is WWW. If we move
that, we can shut it down. Any objections?

Why am I having to spend hours in Syndey saying the same thing?  Why
don't you guys go ahead and change things, and when they fail, I will
still be around.

You're acting as a majority of one, here, Bruce. The reason why any solution
anyone else tries *will* fail is because you will refuse to participate in
it. That's why people are trying to persuade you instead of just going ahead
and doing it; you have the power to effectively sandbag anything anyone else
does, so that's why nobody wants to put effort into it if you're opposed.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus wrote:

Bruce,

Get rid of gborg and let's talk.

Touche'.

Actually, AFAICT, the only active thing left on GBorg is WWW. If we move
that, we can shut it down. Any objections?

This should be a different thread *but* to my knowledge there is more
than WWW active on Gborg. Or at least there is some data that people
still wished moved.

Gborg needs to be shutdown in a scheduled manner. I proposed a schedule
last year but it was largely overlooked:

http://archives.postgresql.org/pgsql-general/2006-08/msg01167.php

My rather bold attempt to reach all people associated with Gborg was not
met with positive responses either (granted due to a mistake on my end).

http://people.planetpostgresql.org/joshua/index.php?/archives/15-Blacklisting-PostgreSQL.Org,-Spam-and-hordes-of-geeks-Oh-My!.html

Why am I having to spend hours in Syndey saying the same thing? Why
don't you guys go ahead and change things, and when they fail, I will
still be around.

You're acting as a majority of one, here, Bruce. The reason why any solution
anyone else tries *will* fail is because you will refuse to participate in
it. That's why people are trying to persuade you instead of just going ahead
and doing it; you have the power to effectively sandbag anything anyone else
does, so that's why nobody wants to put effort into it if you're opposed.

That pretty much sums it up. Bruce you can either come to the party with
a smile or a grin. It is your party. If you come with a smile everyone
will be a lot happier.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Why not just send a notice out stated that Gborg will be shutdown as of June
1st ... give a finite deadline to move things over to pgfoundry ... just
because we 'shut down' the site on June 1st, it doesn't mean we are going to
wipe it all out, we can just put a Redirect on the web server on gborg over to
pgfoundry so that ppl can't go *to* gborg's web site ... we can also make the
CVS 'read-only', so that developers can't update the CVS there, but ppl can
still download the code ...

- --On Thursday, May 03, 2007 10:47:48 -0700 "Joshua D. Drake"
<jd@commandprompt.com> wrote:

Josh Berkus wrote:

Bruce,

Get rid of gborg and let's talk.

Touche'.

Actually, AFAICT, the only active thing left on GBorg is WWW. If we move
that, we can shut it down. Any objections?

This should be a different thread *but* to my knowledge there is more than
WWW active on Gborg. Or at least there is some data that people still wished
moved.

Gborg needs to be shutdown in a scheduled manner. I proposed a schedule last
year but it was largely overlooked:

http://archives.postgresql.org/pgsql-general/2006-08/msg01167.php

My rather bold attempt to reach all people associated with Gborg was not met
with positive responses either (granted due to a mistake on my end).

http://people.planetpostgresql.org/joshua/index.php?/archives/15-Blacklisting
-PostgreSQL.Org,-Spam-and-hordes-of-geeks-Oh-My!.html

Why am I having to spend hours in Syndey saying the same thing? Why
don't you guys go ahead and change things, and when they fail, I will
still be around.

You're acting as a majority of one, here, Bruce. The reason why any
solution anyone else tries *will* fail is because you will refuse to
participate in it. That's why people are trying to persuade you instead of
just going ahead and doing it; you have the power to effectively sandbag
anything anyone else does, so that's why nobody wants to put effort into it
if you're opposed.

That pretty much sums it up. Bruce you can either come to the party with a
smile or a grin. It is your party. If you come with a smile everyone will be
a lot happier.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFGOifh4QvfyHIvDvMRAvKSAJ9l87k9s8wp4nnw5W4xyCQJIgwNOQCeMLXt
bOqtGrCOZ8WnbrhAdYKOGKY=
=vJmp
-----END PGP SIGNATURE-----

I was just getting ready to suggest such an approach. We could
email all the project admins for the reamaining projects with the
dead-line. Backup the information and tell people who to contact in
order to claim whatever information they want. Once the dead-line is
past you can simply shutdown all the gborg services. Those who don't
claim their information either have already moved someplace else or
don't care about what is on gborg anymore.

Additionally if it were desired we could place tarballs of the
cvsrepositories and whatever download files where uploaded for each
project on some ftp server if anyone was interested in preserving that
ifnormation for posterity.

It would not be difficult for me to get a list of email addresses
and project names of those admins on gborg.

Chris Ryan

--- "Marc G. Fournier" <scrappy@hub.org> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Why not just send a notice out stated that Gborg will be shutdown as
of June
1st ... give a finite deadline to move things over to pgfoundry ...
just
because we 'shut down' the site on June 1st, it doesn't mean we are
going to
wipe it all out, we can just put a Redirect on the web server on
gborg over to
pgfoundry so that ppl can't go *to* gborg's web site ... we can also
make the
CVS 'read-only', so that developers can't update the CVS there, but
ppl can
still download the code ...

- --On Thursday, May 03, 2007 10:47:48 -0700 "Joshua D. Drake"
<jd@commandprompt.com> wrote:

Josh Berkus wrote:

Bruce,

Get rid of gborg and let's talk.

Touche'.

Actually, AFAICT, the only active thing left on GBorg is WWW. If

we move

that, we can shut it down. Any objections?

This should be a different thread *but* to my knowledge there is

more than

WWW active on Gborg. Or at least there is some data that people

still wished

moved.

Gborg needs to be shutdown in a scheduled manner. I proposed a

schedule last

year but it was largely overlooked:

http://archives.postgresql.org/pgsql-general/2006-08/msg01167.php

My rather bold attempt to reach all people associated with Gborg

was not met

with positive responses either (granted due to a mistake on my

end).

http://people.planetpostgresql.org/joshua/index.php?/archives/15-Blacklisting

-PostgreSQL.Org,-Spam-and-hordes-of-geeks-Oh-My!.html

Why am I having to spend hours in Syndey saying the same thing?

Why

don't you guys go ahead and change things, and when they fail, I

will

still be around.

You're acting as a majority of one, here, Bruce. The reason why

any

solution anyone else tries *will* fail is because you will refuse

to

participate in it. That's why people are trying to persuade you

instead of

just going ahead and doing it; you have the power to effectively

sandbag

anything anyone else does, so that's why nobody wants to put

effort into it

if you're opposed.

That pretty much sums it up. Bruce you can either come to the party

with a

smile or a grin. It is your party. If you come with a smile

everyone will be

a lot happier.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project:

http://www.postgresql.org/about/donate

PostgreSQL Replication: http://www.commandprompt.com/products/

---------------------------(end of

broadcast)---------------------------

TIP 9: In versions below 8.0, the planner will ignore your desire

to

choose an index scan if your joining column's datatypes do

not

match

- ----
Marc G. Fournier Hub.Org Networking Services
(http://www.hub.org)
Email . scrappy@hub.org MSN .
scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFGOifh4QvfyHIvDvMRAvKSAJ9l87k9s8wp4nnw5W4xyCQJIgwNOQCeMLXt
bOqtGrCOZ8WnbrhAdYKOGKY=
=vJmp
-----END PGP SIGNATURE-----

---------------------------(end of
broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

On Wednesday 02 May 2007 01:19, Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Actually, that can happen with the current system. The real blocker there
is that some people, particularly Tom, work so fast that there's no
chance for a new reviewer to tackle the easy stuff. Maybe the real
solution is to encourage some of our other contributors to get their feet
wet with easy patches so that they can help with the big ones later on?

Yeah, I hear what you say. This is particularly a problem for small bug
fixes: I tend to zing small bugs quickly, first because I enjoy finding/
fixing them and second because I worry that they'll fall off the radar
screen if not fixed. But I am well aware that fixing those sorts of
issues is a great way to learn your way around the code (I think that's
largely how I learned whatever I know about Postgres). I'd be more
willing to stand aside and let someone else do it if I had confidence
that issues wouldn't get forgotten. So in a roundabout way we come back
to the idea that we need a bug tracker (NOT a patch tracker), plus
people putting in the effort to make sure it stays a valid source
of up-to-date info. Without the latter it won't really be useful.

Maybe you just need to have a 1 week clock skew when reading pgsql-bugs?

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

I think this is the apprach joshua tried the first time and it backfired... I
think we need a more personal approach. I'm willing to put time into this if
people want a new point man (I don't think Joshua will mind, lmk if you do)
but it will have to wait untill after pgcon.

On Thursday 03 May 2007 15:12, Chris Ryan wrote:

I was just getting ready to suggest such an approach. We could
email all the project admins for the reamaining projects with the
dead-line. Backup the information and tell people who to contact in
order to claim whatever information they want. Once the dead-line is
past you can simply shutdown all the gborg services. Those who don't
claim their information either have already moved someplace else or
don't care about what is on gborg anymore.

Additionally if it were desired we could place tarballs of the
cvsrepositories and whatever download files where uploaded for each
project on some ftp server if anyone was interested in preserving that
ifnormation for posterity.

It would not be difficult for me to get a list of email addresses
and project names of those admins on gborg.

Chris Ryan

--- "Marc G. Fournier" <scrappy@hub.org> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Why not just send a notice out stated that Gborg will be shutdown as
of June
1st ... give a finite deadline to move things over to pgfoundry ...
just
because we 'shut down' the site on June 1st, it doesn't mean we are
going to
wipe it all out, we can just put a Redirect on the web server on
gborg over to
pgfoundry so that ppl can't go *to* gborg's web site ... we can also
make the
CVS 'read-only', so that developers can't update the CVS there, but
ppl can
still download the code ...

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

Csaba Nagy wrote:

On Thu, 2007-05-03 at 13:51, Bruce Momjian wrote:

I believe the problem is not that there isn't enough information, but
not enough people able to do the work. Seeking solutions in areas that
aren't helping was the illustration.

Yes Bruce, but you're failing to see that a more structured information
infrastructure will attract more people to do the work which could
eventually solve the problem you're having in the first place
(contradicting your argument that it won't help), at the cost of some
possible additional work (which I actually think you're overestimating
the amount of - it's probably more like a learning curve than actual
additional work).

The fact that there is enough information is not relevant, it must be in
the right place too - too much information or hard to find information
is sometimes worst than no information.

If I can't get help on a simple request of maintaining an 8.3 status web
page, I think more help is unlikely and I am not interested in doing
more work to find out.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Josh Berkus wrote:

Bruce,

Get rid of gborg and let's talk.

Touche'.

Actually, AFAICT, the only active thing left on GBorg is WWW. If we move
that, we can shut it down. Any objections?

Why am I having to spend hours in Syndey saying the same thing? ?Why
don't you guys go ahead and change things, and when they fail, I will
still be around.

You're acting as a majority of one, here, Bruce. The reason why any solution
anyone else tries *will* fail is because you will refuse to participate in
it. That's why people are trying to persuade you instead of just going ahead
and doing it; you have the power to effectively sandbag anything anyone else
does, so that's why nobody wants to put effort into it if you're opposed.

People aren't willing to hel pme in even a simple task of maintaining an
8.3 patches status page, so why would they want to help with something
larger. I am not going to make my job harder only to find out no one
wants to help.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

I have already responded to all the email comments. Here is my idea of
moving forward. There are basically three interrelated issues:

1) bug tracking
2) getting more people to review complex patches
3) patch tracking

I am not going to go into #1, except to say that the problem has always
been the effort needed to track the bugs vs. the value.

I am not going to go into #2, except to say that this is going to require
a personal approach to assisting developers. One thing I am going to do
is to add an item to the developer's FAQ outlining how patches are analyzed
for application, which should help both patch submitters and committers
(sample attached).

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

What I think we can do simply is to have our email software automatically
number emails submitted to the patches list that already don't have a
number. This way, all followups, even if moved to the hackers list, would
maintain that patch number, and if an updated version is posted, the user
would keep the same number in the email subject.

Once that is done, it should be trivial to write a web application that
will track the patches based on the subject line, something like
"PATCH#555". Committers can include that patch string in the commit
message, so committed patches can be automatically removed from the open
patch queue. The only tricky part is to handle patches that are rejected
or kept for a later release. That would probably require web access to
adjust.

One thing that has always bothered me about tracking patches is that when
an email to the patches list is bounced for discussion to the hackers
list, there isn't any good way for outside people to track the full patch
discussion because we don't track threads that move to different mailing
lists. The special patch number would make such tracking easier.

The good thing about such a simple system is that it has a very light
burden on patch submitters and committers. The major work is done by the
web application, but that can all be programmed.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

I have already responded to all the email comments. Here is my idea of
moving forward. There are basically three interrelated issues:

1) bug tracking
2) getting more people to review complex patches
3) patch tracking

I am not going to go into #1, except to say that the problem has always
been the effort needed to track the bugs vs. the value.

I am not going to go into #2, except to say that this is going to require
a personal approach to assisting developers. One thing I am going to do
is to add an item to the developer's FAQ outlining how patches are analyzed
for application, which should help both patch submitters and committers
(sample attached).

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

What I think we can do simply is to have our email software automatically
number emails submitted to the patches list that already don't have a
number. This way, all followups, even if moved to the hackers list, would
maintain that patch number, and if an updated version is posted, the user
would keep the same number in the email subject.

Once that is done, it should be trivial to write a web application that
will track the patches based on the subject line, something like
"PATCH#555". Committers can include that patch string in the commit
message, so committed patches can be automatically removed from the open
patch queue. The only tricky part is to handle patches that are rejected
or kept for a later release. That would probably require web access to
adjust.

One thing that has always bothered me about tracking patches is that when
an email to the patches list is bounced for discussion to the hackers
list, there isn't any good way for outside people to track the full patch
discussion because we don't track threads that move to different mailing
lists. The special patch number would make such tracking easier.

The good thing about such a simple system is that it has a very light
burden on patch submitters and committers. The major work is done by the
web application, but that can all be programmed.

Bruce,

I will say publicly what I have said to others privately. Forgive me if
I'm a bit blunter than usual. I do not see any value in this at all.
What we need to track are problems to be solved, be they bugs or
features, not particular patches. Tracking patches simply comes too late
in the process.

I think that your attitude to the use of bug/feature trackers is quite
unreasonable, and certainly in opposition to what seems to be the
majority opinion among developers. It's a great pity that you are so
utterly resistant to use of tracking software. The only reason that this
system, at best a half measure in almost everyone's eyes, is being
proposed, as far as I can see, is that you will not agree to use
anything else.

So if this goes ahead and proves to be of little value, I hope that you
will relent and agree the use of proper tracking software like almost
every other open source project uses. It really is time that PostgreSQL
managed to advance beyond thinking that email lists are the greatest
management tool since sliced bread. It's just indefensible in 2007.

cheers

a disappointed andrew

Andrew Dunstan wrote:

I will say publicly what I have said to others privately. Forgive me if
I'm a bit blunter than usual. I do not see any value in this at all.
What we need to track are problems to be solved, be they bugs or
features, not particular patches. Tracking patches simply comes too late
in the process.

I think that your attitude to the use of bug/feature trackers is quite
unreasonable, and certainly in opposition to what seems to be the
majority opinion among developers. It's a great pity that you are so
utterly resistant to use of tracking software. The only reason that this
system, at best a half measure in almost everyone's eyes, is being
proposed, as far as I can see, is that you will not agree to use
anything else.

So if this goes ahead and proves to be of little value, I hope that you
will relent and agree the use of proper tracking software like almost
every other open source project uses. It really is time that PostgreSQL
managed to advance beyond thinking that email lists are the greatest
management tool since sliced bread. It's just indefensible in 2007.

As I said before, I am involved in patches only when a patch isn't
addressed. If a new system works, I will have nothing to do, which is
good.

If you want me to believe it will work better than what we do now, I
can't. Prove me wrong. Forget about what I think. Do something and
stop talking about it.

What I am not going to do is to do 2x more work and get 2% more help,
which is what I fear.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

What I think we can do simply is to have our email software automatically
number emails submitted to the patches list that already don't have a
number. This way, all followups, even if moved to the hackers list, would
maintain that patch number, and if an updated version is posted, the user
would keep the same number in the email subject.

<snip tracker outline>

Barring a few trivial details, that sounds almost identical to what I proposed.

Regards, Dave

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

What I think we can do simply is to have our email software automatically
number emails submitted to the patches list that already don't have a
number. This way, all followups, even if moved to the hackers list, would
maintain that patch number, and if an updated version is posted, the user
would keep the same number in the email subject.

<snip tracker outline>

Barring a few trivial details, that sounds almost identical to what I
proposed.

Well, Andrew says everyone he talks to doesn't want it. They want a
more comprehensive solution that goes from bug to patch.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Dave Page wrote:

Barring a few trivial details, that sounds almost identical to what I

proposed.

Well, Andrew says everyone he talks to doesn't want it. They want a

more comprehensive solution that goes from bug to patch.

Dave can speak for his own views, but I think you're misquoting me somewhat.

I said that a majority of developers wanted to move to use of a tracking
system, not "everyone".

I did say that this patch tracker would be "at best a half measure in
almost everyone's eyes". Note the "almost". That doesn't mean nobody wants
it. Possibly some see significant benefit where I see little or none.
Clearly Dave does. But it does mean that it's not what most people really
want.

I would be prepared to put considerable effort (say, comparable to what I
have put into the buildfarm) into establishing and maintaining a
feature/bug tracker system, if I thought there was enough buyin. I have
not done so in the past because others (principally you) have been against
it, and so it seemed doomed to failure. Unlike the buildfarm, which can
stand on its own, a tracker requires cooperation from the developers in
order to be effective.

Our present change management methods strike me as being analogous to
keeping track of a banking system in a spreadsheet (don't get me started).
It's quite ironic (not to mention sad) given that we are producing a
sophisticated database ...

cheers

andrew

Bruce Momjian wrote:

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

are you thinking about something like
http://developer.postgresql.org/index.php/Todo:WishlistFor83 on steriods
(ie with more references to actual patches and discussion and
explaination of functionality) or something completely different ?
I'm a bit unsure on how this webpage would differ from a typical
bugtracker ...
Maybe you could give a concrete example for a particular patch in the
queue so that everybody can follow ?

Stefan

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

are you thinking about something like
http://developer.postgresql.org/index.php/Todo:WishlistFor83 on steriods
(ie with more references to actual patches and discussion and
explaination of functionality) or something completely different ?
I'm a bit unsure on how this webpage would differ from a typical
bugtracker ...
Maybe you could give a concrete example for a particular patch in the
queue so that everybody can follow ?

At this point, just one line for each patch, and who is working on it:

Patch, Author Committer
HOT Pavan ?
XML misc Peter
etc.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

are you thinking about something like
http://developer.postgresql.org/index.php/Todo:WishlistFor83 on steriods
(ie with more references to actual patches and discussion and
explaination of functionality) or something completely different ?
I'm a bit unsure on how this webpage would differ from a typical
bugtracker ...
Maybe you could give a concrete example for a particular patch in the
queue so that everybody can follow ?

At this point, just one line for each patch, and who is working on it:

Patch, Author Committer
HOT Pavan ?
XML misc Peter
etc.

that would be easy to do on either the wishlist or a seperate wiki page
and I would volunteer to do that if you think it is useful.

Stefan

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

are you thinking about something like
http://developer.postgresql.org/index.php/Todo:WishlistFor83 on steriods
(ie with more references to actual patches and discussion and
explaination of functionality) or something completely different ?
I'm a bit unsure on how this webpage would differ from a typical
bugtracker ...
Maybe you could give a concrete example for a particular patch in the
queue so that everybody can follow ?

At this point, just one line for each patch, and who is working on it:

Patch, Author Committer
HOT Pavan ?
XML misc Peter
etc.

that would be easy to do on either the wishlist or a seperate wiki page
and I would volunteer to do that if you think it is useful.

OK, you have to go back to Tom's email stating where we are on each
patch, then look over the patch application and find out which ones have
been applied. Also you have to read the replies to find out who has
taken ownership of patches.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Stefan Kaltenbrunner wrote:

Bruce Momjian wrote:

Dave Page wrote:

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Sent: 05/05/07, 03:00:25
Subject: [HACKERS] New idea for patch tracking

As for #3, again, I don't want us to take on a burdensome patch tracking
process that is more effort than it is worth, and the lack of people
jumping to even manage a simple web page for current 8.3 patches has me
questioning what kind of support a burdensome tracking system would
receive.

I don't recall hearing you ask for people to help with a web page.

I want create and maintain a web page that tracks where we are on each
8.3 patch, but have had not takers.

are you thinking about something like
http://developer.postgresql.org/index.php/Todo:WishlistFor83 on steriods
(ie with more references to actual patches and discussion and
explaination of functionality) or something completely different ?
I'm a bit unsure on how this webpage would differ from a typical
bugtracker ...
Maybe you could give a concrete example for a particular patch in the
queue so that everybody can follow ?

At this point, just one line for each patch, and who is working on it:

Patch, Author Committer
HOT Pavan ?
XML misc Peter
etc.

that would be easy to do on either the wishlist or a seperate wiki page
and I would volunteer to do that if you think it is useful.

OK, you have to go back to Tom's email stating where we are on each
patch, then look over the patch application and find out which ones have
been applied. Also you have to read the replies to find out who has
taken ownership of patches.

ok I did a rough sketch of how I interpreted your proposal on
http://developer.postgresql.org/index.php/Todo:PatchStatus.
This table is by far not complete yet(more of a PoC) but I wanted to get
some feedback if I'm on the right track before I put more time into this.

Stefan

People aren't willing to hel pme in even a simple task of maintaining

an

8.3 patches status page, so why would they want to help with something
larger. I am not going to make my job harder only to find out no one
wants to help.

I thought about volunteering to do this, but:

1. I am a little warry of inserting myself (as an outsider) into a major
controversy as my first contribution to the project.

2. It seems like it would be difficult or impossible for an outsider to
do this well. Essentially, I'd have to read every message on -hackers,
-patches, and -committers, and try to figure out which of those messages
amounted to a change in status for which patches, and then update the
status of the patches.

Example: Tom says "what about XYZ? ISTM this will have to wait for
8.4". The person who wrote the patch replies with "I think XYZ is not
an issue because of ABC." It's not clear (at least to me) whether the
patch is now in play for 8.3 again or whether it's still on hold.

In addition, if some discussion is happening via private email (which it
sounds like it is), then this wouldn't be complete even if it were done
perfectly.

I write web-based workflow applications for a living, so in theory I'm
more amenable to the idea of helping out in that way. But it seems to
me that right now there's no consensus on whether we need this at all,
and if so what it should do.

I don't really want to get involved in the central argument about what
the "right" way of doing this is, but I think Bruce's proposal to put a
patch number in every email that hasn't got one can't possibly be any
worse than what we're doing now, and it might be better, so why not?
I'm even willing help with this if there is consensus on it.

Thanks,

...Robert

I would like to add one point:

Bruce Momjian wrote:

Patch committers check several things before applying a patch:

1) Follows the SQL standard or community agreed-upon behavior
2) Style merges seamlessly into the surrounding code
3) Written as simply and efficiently as possible
4) Uses the available PostgreSQL subsystems properly
5) Contains sufficient comments
6) Contains code that works on all supported operating systems
7) Has proper documentation
8) Passes all regression tests

8.5) Contains regression test(s) which covered performed changes

9) Behaves as expected, even under unusual cirumstances
10) Contains no reliability risks
11) Does not overly complicate the source code
12) If performance-related, it should have a measureable performance benefit
13) Is of sufficient usefulness to the average PostgreSQL user
14) Follows existing PostgreSQL coding standards

Zdenek

OK, item modified to:

<li>Passes all regression tests, and if needed, adds new
ones</li>

---------------------------------------------------------------------------

Zdenek Kotala wrote:

I would like to add one point:

Bruce Momjian wrote:

Patch committers check several things before applying a patch:

1) Follows the SQL standard or community agreed-upon behavior
2) Style merges seamlessly into the surrounding code
3) Written as simply and efficiently as possible
4) Uses the available PostgreSQL subsystems properly
5) Contains sufficient comments
6) Contains code that works on all supported operating systems
7) Has proper documentation
8) Passes all regression tests

8.5) Contains regression test(s) which covered performed changes

9) Behaves as expected, even under unusual cirumstances
10) Contains no reliability risks
11) Does not overly complicate the source code
12) If performance-related, it should have a measureable performance benefit
13) Is of sufficient usefulness to the average PostgreSQL user
14) Follows existing PostgreSQL coding standards

Zdenek

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

------- Original Message -------
From: Bruce Momjian <bruce@momjian.us>
To: Dave Page <dpage@postgresql.org>
Sent: 05/05/07, 11:06:37
Subject: Re: [HACKERS] New idea for patch tracking

<snip tracker outline>

Barring a few trivial details, that sounds almost identical to what I
proposed.

Well, Andrew says everyone he talks to doesn't want it. They want a
more comprehensive solution that goes from bug to patch.

I don't recall him saying that, though I do know that's /his/ opinion. It's certainly *not* the opinion of most of the people I've spoken with.

I don't disagree with the idea in principle though, but I don't believe it will work for us because it's so fundamentally different from the way we currently work and still wouldn't solve the problem of capturing all the relevant discussion regarding a given patch (or bug) without a reasonable amount of manual work, or grafting a large part of what I'm proposing on the side.

Regards, Dave

On May 5, 2007, at 11:40 AM, Dave Page wrote:

<snip tracker outline>

Barring a few trivial details, that sounds almost identical to
what I
proposed.

Well, Andrew says everyone he talks to doesn't want it. They want a
more comprehensive solution that goes from bug to patch.

I don't recall him saying that, though I do know that's /his/
opinion. It's certainly *not* the opinion of most of the people
I've spoken with.

I don't disagree with the idea in principle though, but I don't
believe it will work for us because it's so fundamentally different
from the way we currently work and still wouldn't solve the problem
of capturing all the relevant discussion regarding a given patch
(or bug) without a reasonable amount of manual work, or grafting a
large part of what I'm proposing on the side.

IIRC, every recent debate about going to a bug/issue (and now patch)
tracker has ultimately boiled down to the impact it would have on our
current work processes: it has to work 100% painlessly off of the
mailing lists. It's got to do more than just pipe changes to the
mailing list; it's got to be able to be driven by the list as well.
That's the real challenging part.

People have suggested different trackers that have varying amounts of
email capability, but I don't think any of them have had the full
capability that we'd need. At best they might accept comments on a
bug/issue via email, but to work for the community they'd need to go
beyond that. You'd have to be able to resolve via email (preferably
tied to -commiters). You'd need to be able to make a bug as invalid.
You'd need to be able to open a new issue via email. And change
status. And assign it to someone. And it would have to actually
thread discussion to be useful. Probably some other things as well.

Since a system like that doesn't exist I think it's going to be up to
us to create one. When it comes to the full set of features you'd
expect out of an issue tracker, it would probably make sense to start
with an existing project and try and add this functionality. But
right now I don't think such an effort would work well, because we
don't know well enough how all these new features should work.

But writing a patch tracker would be simpler than a full issue
tracker. It's also something we could more easily do in a piece-meal
fashion, since the only users will be developers. Building such a
tool would provide a wealth of experience that could then be applied
to tackling a full-blown issue tracking system.

The system Bruce and Dave have outlined shouldn't be terribly hard to
implement. Let's start with that and see what we learn (as I've
already told Dave, this is something I'll help with). Otherwise we'll
once again have spent another chunk of community effort on a tracker
discussion that results in nothing being done (I guess it has been 6
months since it was last brought up, so we were due again anyway...)
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Jim Nasby wrote:

People have suggested different trackers that have varying amounts of
email capability, but I don't think any of them have had the full
capability that we'd need. At best they might accept comments on a
bug/issue via email, but to work for the community they'd need to go
beyond that. You'd have to be able to resolve via email (preferably tied
to -commiters). You'd need to be able to make a bug as invalid. You'd
need to be able to open a new issue via email. And change status. And
assign it to someone. And it would have to actually thread discussion to
be useful. Probably some other things as well.

As I wrote few days ago. You can see how and what use e.g. Apache Derby
community. I guess more of mentioned issues they have solved and we can
take some of their ideas. However I still miss list of tracker
requirements - what tracker MUST have and what is nice to have.

And you describe current processes based on email communication. But if
we setup some tracker some process will be changed. I think first step
is determine what we really want and after we can discuss how to reach it.

Since a system like that doesn't exist I think it's going to be up to us
to create one. When it comes to the full set of features you'd expect
out of an issue tracker, it would probably make sense to start with an
existing project and try and add this functionality. But right now I
don't think such an effort would work well, because we don't know well
enough how all these new features should work.

Create own tracker is reinvent a wheel and waste a time. There are a lot
of trackers and I believe that one of them fit postgres requirements. I
agree with your idea to try one and if it will be necessary we can add
some functionality. But I think that there are not clear requirements
and I also afraid that there is not unified view of core team on this.

I suggest following process:

1) create list of requirements - MUST HAVE/NICE TO HAVE
2) create list of tracker
3) reject trackers which does not fit "MUST HAVE"
4) each member of core team create own order
5) results will be put together and one tracker will be select for testing.

Zdenek

On May 7, 2007, at 7:47 AM, Zdenek Kotala wrote:

Jim Nasby wrote:
And you describe current processes based on email communication.
But if we setup some tracker some process will be changed. I think
first step is determine what we really want and after we can
discuss how to reach it.

If we lived in an ideal world I'd agree with you 100%. But we live in
PostgreSQL-community-world. :) There is a *lot* of resistance in the
development community to going to any kind of a tracker, even if it
would mean essentially zero change to how the development has to
work. If you don't believe me go look in the archives; I believe this
debate happens about twice a year, and every time the result is the
same: lots of emails, zero change.

Create own tracker is reinvent a wheel and waste a time. There are
a lot of trackers and I believe that one of them fit postgres
requirements. I agree with your idea to try one and if it will be
necessary we can add some functionality. But I think that there are
not clear requirements and I also afraid that there is not unified
view of core team on this.

Yes, when it comes to doing a full-blown tracker it would be a huge
amount of wheel reinvention. But that's not the case with a simple
patch tracker.

Let's take the baby step of a patch tracker first and see what we
learn from it.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)