Patch committers
Robert Haas wrote:
The next CommitFest is scheduled to start in a week. So far, it
doesn't look too bad, though a lot could change between now and then.I would personally prefer not to be involved in the management of the
next CommitFest. Having done all of the July CommitFest and a good
chunk of the September CommitFest, I am feeling a bit burned out.
This brings up a concern I have --- that the number of patch
committers/managers is decreasing while our patch volume is increasing.
Consider that Heikki is working mostly on Hot Standby and Streaming
Replication, Alvaro isn't as involved in applying patches, Neil Conway
isn't involved with Postgres anymore, I am in a 42-day travel period,
and Robert Haas is feeling burnt-out --- that is not a pretty sight.
Much of the patch burden is falling on Tom. Now, Tom isn't complaining,
but I am concerned about placing too much of a burden on him. I know we
are growing new patch reviewers who will eventually be able to review
_and_ apply patches on their own, but getting them to that point is
going to take time.
I have no real answers, but I am concerned. I have talked to many of
you privately about this to get your input.
On a brighter note, I was able to read three weeks of back-logged emails
(3k emails) in the past two days because our community is now very good
at resolving the greatly increased email volume of bug reports and
questions.
I can definitely say that the quality and volume of our email support to
users had advanced significantly in the past year. It seems that a
similar increase in patch application support is going to take longer to
materialize.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
bruce wrote:
This brings up a concern I have --- that the number of patch
committers/managers is decreasing while our patch volume is increasing.
Consider that Heikki is working mostly on Hot Standby and Streaming
Replication, Alvaro isn't as involved in applying patches, Neil Conway
isn't involved with Postgres anymore, I am in a 42-day travel period,
and Robert Haas is feeling burnt-out --- that is not a pretty sight.Much of the patch burden is falling on Tom. Now, Tom isn't complaining,
but I am concerned about placing too much of a burden on him. I know we
are growing new patch reviewers who will eventually be able to review
_and_ apply patches on their own, but getting them to that point is
going to take time.I have no real answers, but I am concerned. I have talked to many of
you privately about this to get your input.
There is a more worrisome/sinister possibility that I didn't want to
mention in my first email --- that companies are hiring our most
experienced developers and having them work almost exclusively on
company-related or closed-source projects.
Unfortunately I can think of at least half-a-dozen cases of this
happening. Now, this was expected, but the hope was that this kind of
skill siphoning would be offset by additional people being paid to be
involved in Postgres development, and that clearly is happening. What I
am worried about is that this is not happening as much among our most
experienced people --- that in fact they are the most likely to be hired
and perhaps placed into roles where they are less involved in the
community than they were before. (Of course, there are counter-examples
where experience developers are hired to work on community Postgres
full-time.)
There is not much we as a community can do to prevent skill siphoning,
except perhaps publicly complaining about companies that do this.
If this is indeed a pattern, it has serious long-term consequences
because it means we will always have an unnaturally small pool of very
skilled people.
Up to this point we have been able to maintain a happy group of
developers. If things become unbalanced and people are regularly
required to do things they don't like doing, it will lead to Postgres no
longer being fun for them, which leads to burn-out and them leaving the
project. That happens frequently in other open source projects, but it
has been a very rare occurance for us, and I hope it stays that way.
Our ability to retain people for many years has benefitted us in
countless ways.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Import Notes
Reply to msg id not found: | Resolved by subject fallback
On ons, 2009-11-11 at 10:25 -0500, Bruce Momjian wrote:
There is a more worrisome/sinister possibility that I didn't want to
mention in my first email --- that companies are hiring our most
experienced developers and having them work almost exclusively on
company-related or closed-source projects.
I can't claim to know everyone's employment terms, but I think it's a
bit of an illusion to think that the above hasn't always happened.
Except for a handful of people who have very special job arrangements,
*everyone* is working almost exclusively on company-related or
closed-source projects. The patches that get sent in are almost always
either fallout from a customer/company project, or stuff that one of the
closed-sourced forks has developed that they don't want to maintain, or
stuff people do "at night" to make their lives easier in the distant
future. All of those things are already special arrangements that
people need to make with their employers and their lives, but they have
discernible benefits. But you can't expect a lot of people or employers
to reserve time on top of that for handholding other people's patches
and for other "community" stuff that has no easy to measure benefits.
Peter Eisentraut wrote:
On ons, 2009-11-11 at 10:25 -0500, Bruce Momjian wrote:
There is a more worrisome/sinister possibility that I didn't want to
mention in my first email --- that companies are hiring our most
experienced developers and having them work almost exclusively on
company-related or closed-source projects.I can't claim to know everyone's employment terms, but I think it's a
bit of an illusion to think that the above hasn't always happened.
Except for a handful of people who have very special job arrangements,
Yes, it has always happened. I think the big difference is that those
"special employments" are currently a fixed size, and the volume of
patches has increased as our user-base has increased.
I also think the bad economy is making it harder for people/companies to
devote time to community stuff when paid work is available.
I always had an assumption that those special employments would grow as
the community grew, but I am not sure I am seeing that. Combine that
with skill siphoning, and I get concerned.
*everyone* is working almost exclusively on company-related or
closed-source projects. The patches that get sent in are almost always
either fallout from a customer/company project, or stuff that one of the
closed-sourced forks has developed that they don't want to maintain, or
stuff people do "at night" to make their lives easier in the distant
future. All of those things are already special arrangements that
people need to make with their employers and their lives, but they have
discernible benefits. But you can't expect a lot of people or employers
to reserve time on top of that for handholding other people's patches
and for other "community" stuff that has no easy to measure benefits.
Totally agree. It is that zero-return work that is hard to justify for
people and companies. It is clearly something that requires
self-sacrifice, and personally I think a culture of self-sacrifice is
what has given us such great success and such a nuturing community
environment.
Historically, there was a "golden era" when Tom, Heikki, Neil Conway,
Alvaro, and others all handled patches and things ran with much less of
an individual burden than we have now, but that might have been an
aberration.
Personally, I don't think we have a major problem now, but I do think
there is the chance of this getting worse in the future, and making
people unhappy. I am brining it up now in case there is something we
can do to avoid such unhappiness.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Bruce,
Well, I think the answer is to promote some new committers. When was
the last time we added a committer?
We have added a bunch of new reviewers and major contributors over the
last 2 years. It's time to review their work and see who can be
promoted to more responsibility for other people's patches.
--Josh Berkus
Josh Berkus wrote:
Bruce,
Well, I think the answer is to promote some new committers. When was
the last time we added a committer?We have added a bunch of new reviewers and major contributors over the
last 2 years. It's time to review their work and see who can be
promoted to more responsibility for other people's patches.
Yes, I did consider that, and it might be a solution. The larger
problem is that with many of the patches, it isn't commit-rights that is
preventing people from being involved, but rather the expert knowledge
necessary to make sure the patch works properly.
For example, probably only Simon and Heikki are knowledge enough to
apply the HS and SR patchs --- but those patch is handled. The harder
part is the other patches where there are only a small pool of people
knowledgeable enough to apply the patch, but many of those knowledgeable
people are tied up with non-community things.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
For example, probably only Simon and Heikki are knowledge enough to
apply the HS and SR patchs --- but those patch is handled. The harder
part is the other patches where there are only a small pool of people
knowledgeable enough to apply the patch, but many of those knowledgeable
people are tied up with non-community things.
People learn to do this by doing it. That's how you learned, and Tom
and Heikki. Ultimately, it's a source code control system; if someone
screws up, we can always back it out.
--Josh Berkus
Josh Berkus wrote:
For example, probably only Simon and Heikki are knowledge enough to
apply the HS and SR patchs --- but those patch is handled. The harder
part is the other patches where there are only a small pool of people
knowledgeable enough to apply the patch, but many of those knowledgeable
people are tied up with non-community things.People learn to do this by doing it. That's how you learned, and Tom
and Heikki. Ultimately, it's a source code control system; if someone
screws up, we can always back it out.
True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.
Right, but being responsible for applying the patch is the only real
incentive anyone has to learn enough to understand its effects. If a
contributor is not responsible, they can always think "oh, Tom will get
it, I'll learn that later."
--Josh Berkus
Couldn't you have a policy that every patch is reviewed first by someone who wants to be an expert in that area, and then by someone who is currently an expert. Then you have the best of both worlds. The patch is reviewed by someone will make sure it won't cause problems, and the want to be expert gets training and will eventually be able to be the expert.
On Nov 11, 2009, at 12:58 PM, Josh Berkus wrote:
Show quoted text
True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.Right, but being responsible for applying the patch is the only real
incentive anyone has to learn enough to understand its effects. If a
contributor is not responsible, they can always think "oh, Tom will get
it, I'll learn that later."--Josh Berkus
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Rick Gigger wrote:
Couldn't you have a policy that every patch is reviewed first by someone who wants to be an expert in that area, and then by someone who is currently an expert. Then you have the best of both worlds. The patch is reviewed by someone will make sure it won't cause problems, and the want to be expert gets training and will eventually be able to be the expert.
The wannabe-experts can do all the reviewing they want today (and they
do). But we cannot make this a hard requirement, or we risk stalling
the project for patches that are not reviewed timely by non-experts.
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Wed, Nov 11, 2009 at 12:45 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
The patches that get sent in are almost always
either fallout from a customer/company project, or stuff that one of the
closed-sourced forks has developed that they don't want to maintain, or
stuff people do "at night" to make their lives easier in the distant
future. All of those things are already special arrangements that
people need to make with their employers and their lives, but they have
discernible benefits. But you can't expect a lot of people or employers
to reserve time on top of that for handholding other people's patches
and for other "community" stuff that has no easy to measure benefits.
Well, I'm perfectly willing to handhold other people's patches and do
community stuff that has no easy to measure benefits. I'm willing to
do it precisely because I believe it will make it easier to get my own
patches accepted. AIUI, the whole point of CommitFests is that I
agree to review other people's patches and they in turn agree to
review mine, and we all get to scratch our respective itches. In
fact, because I enjoy community work, I'm even willing to spend MORE
time helping other people get their patches in than I do writing my
own patches - but I'm only willing to spend 2x or 3x the time on other
people's stuff that I spend on my own, not 10x.
In the last two CommitFests it has become evident that we have a
problem with freeloading. During the September CommitFest, the set of
reviewers and the set of patch submitters were almost disjoint. Now,
fortunately, we still had enough reviewers; the real shortage was of
committers. But I think that some of the reviews were not as good as
they could have been had they been reviewed by more experienced
contributors, or even just by multiple people. A fair amount of easy
stuff slipped through the review process. Tom cleaned it up, but he
shouldn't have to be responsible for things that a read-through of the
patch by an experienced C programmer should have caught. I tried to
help, but I was fairly tied up with overall CommitFest management and
did not have time for a full read-through of every patch.
...Robert
On Wed, Nov 11, 2009 at 2:51 PM, Bruce Momjian <bruce@momjian.us> wrote:
Josh Berkus wrote:
For example, probably only Simon and Heikki are knowledge enough to
apply the HS and SR patchs --- but those patch is handled. The harder
part is the other patches where there are only a small pool of people
knowledgeable enough to apply the patch, but many of those knowledgeable
people are tied up with non-community things.People learn to do this by doing it. That's how you learned, and Tom
and Heikki. Ultimately, it's a source code control system; if someone
screws up, we can always back it out.True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.
Well, if we're going to add committers (and I'm generally in favor of
that) we need to pick people who won't apply patches that they don't
understand. In a way, that's really the only HARD requirement for a
committer, I think. If someone doesn't understand any aspect of the
system well enough to commit patches, but they know that, then having
them as a committer will be useless, but not actively harmful (beyond
being a waste of time). On the other hand, someone who actually does
know some parts of the system well enough to commit patches but
doesn't have the discretion to avoid things that are more than they
can tackle will create chaos.
In other words, we should be looking for people who (1) know enough to
be able to commit at least some patches without breaking the world and
(2) are responsible enough to know which things they can commit and
which they should leave alone.
Of course, anyone you pick will probably make some mistakes, but
that's the general idea...
...Robert
Bruce Momjian wrote:
True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.
The usual type of practice here should come from applying trivial
patches, or ones that don't impact code quality. Docs patches come to
mind as a good way someone could get used to the commit process without
introducing much potential mayhem along the way. As far as keeping new
people away from complicated patches, ultimately you just have to trust
that anyone who can commit has a reasonable idea of their own
capabilities. I seriously doubt you're going to find a new committer
jumping right in by committing hot standby out of the gate just because
they could do so.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
Robert Haas wrote:
I tried to help, but I was fairly tied up with overall CommitFest management and
did not have time for a full read-through of every patch.
I think it's completely unreasonable to expect the CF manager to do any
patch review themselves. It's a hard enough job to keep going without
actually getting your hands into the details.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
On Wed, Nov 11, 2009 at 3:57 PM, Greg Smith <greg@2ndquadrant.com> wrote:
Robert Haas wrote:
I tried to help, but I was fairly tied up with overall CommitFest
management and
did not have time for a full read-through of every patch.I think it's completely unreasonable to expect the CF manager to do any
patch review themselves. It's a hard enough job to keep going without
actually getting your hands into the details.
Well, that's another reason I'd kind of like to take a break from CF
management - it would free up more of my time for actual patch review.
I have been doing some review anyhow, but given as how I don't get
paid for any of this, there's only a few hours a day that are
potentially available for this, and it has to compete with the
dishwasher breaking and my wife wanting to play Pandemic. By the way,
if anyone wants to buy me a new dishwasher, I will put reviewing your
patch at the top of my "to do" list. :-)
...Robert
Bruce Momjian wrote:
I also think the bad economy is making it harder for people/companies to
devote time to community stuff when paid work is available.
I think this explains away more of the recent situation than you're
giving it credit for. When everybody's fat and happy and it's easy to
generate/raise money, it's also easy to throw money toward the
community. When times are tight, giving away work that you might charge
for (or have already charged for) is harder for a company to justify.
It's easy to plan to have someone do community work when you hire them,
only to realize down the line that business has dried up enough that
you're stuck with the choice between them doing that and a job that will
make or break and upcoming payroll. And that's where a lot more
businesses are at right now than at any time in a long while.
After looking for an example of the boom/bust cycle impacting this
community's work that's old enough to be clearer in hindsight, I would
suggest noting that Great Bridge was officially announced in May of 2000
and was gone by the end of 2001. Overlay those dates on top of
http://www.google.com/finance?q=INDEXNASDAQ:.IXIC after switching "Zoom"
to show 10 years.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
On 11/11/09 12:55 PM, Greg Smith wrote:
I seriously doubt you're going to find a new committer
jumping right in by committing hot standby out of the gate just because
they could do so.
We'd be lucky to get them to *read* the Hot Standby patch and comment on it.
--Josh Berkus
The following email expresses my personal opinion and does not reflect
the opinion of my employers.
Bruce Momjian wrote:
I also think the bad economy is making it harder for people/companies to
devote time to community stuff when paid work is available.
Actually the bad economy should be a booster for open source projects.
There should be more developers with time to acquire new skills on
projects that will get them a better job when the economy comes back.
I think the problems are more rooted in the developer community itself.
The pg-hackers mailing list is probably the less socially skilled
developer community I have ever seen in all the open source projects I
have been involved with. A very high standard is set for contributions,
which is good for the quality of the code, but the lack of development
process and clear decision chain turns every new contributor into
endless frustration. For a patch to be committed, a vague consensus has
to arise among the strong technical voice(s) (usually convincing Tom is
enough). If a more complex feature needs to be implemented, the lack of
decision process ends up in a first long round of emails until everybody
gets tired of it. Then sometimes later someone tries to re-activate the
debate for another round and so on (partitioning is a good example). You
lost potential committers at each of these rounds.
The way I see it, most companies try to push their agenda, contribute
their patches back to the community if it works and just go with their
own fork and closed implementation if this is too much work or burden.
Whatever the economy, very few people can commit to an indefinite amount
of time to get a feature integrated in Postgres.
Now you should probably ask yourself what you should do as a community
to get more committers? Like it was said at PGCon, to get a patch in, it
is going to be hard and painful. How do you make it less hard and less
painful?
On the other end, how do we, simple developers, become better to reach
the status of committers? How can we endure the constant bashing
especially in the early stages of our learning phase (especially if you
are not paid to do it)? How do we justify to our employers that they
should persist through this long process without visibility, so that
eventually their contribution will make it back to the community? How do
we make it easier for companies to contribute code?
The lightness of the development process (no project manager, no
steering committee, no voting, no product management, ...) is both a
strength and a weakness that makes Postgres what it is today. The
commitfest started to address some of the most obvious issues but there
is probably much more that can be done by looking at what is happening
in the other open source communities.
Even if the economy is hard, I found time to invest my own 2 cents in
this email ;-)
Emmanuel
Hi,
True, but even I avoid patches I don't understand, and "practicing" by
applying them could lead to a very undesirable outcome, e.g.
instability.
How about having a staging server to help around novice committers?
Basically the selected new band of people can take a patch, review it
and if they deem it fit for checkin, they could check it into the
staging server.
Then they should try out the regression runs (we do have very few test
cases) and other sanity checks to ensure that things are sane.
Probably the buildfarm infrastructure can also be used to run against
this staging server on all platforms.
That ways, we can avoid the initial instability upto some extent and
then noob committer can then muster courage to atleast tell the
core-committers what they think about the staged patch commit.
Regards,
Nikhils
--
http://www.enterprisedb.com
