Thought provoking piece on NetBSD

It argues that 'forking' a project is not the real solution of a project going
wrong but rather staging a palace revolution of the existing infrastructure.

On Thu, Aug 31, 2006 at 09:11:52AM -0700, Joshua D. Drake wrote:

I thought some people in this group may find this letter from one of
NetBSD's founders very interesting.

http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html

It is current, to the point and has some direct correlations with our
project that we may want to be aware of.

Nice post, though I don't think PostgreSQL really has many of the
faults he lists. The only obvious one to me is the strong leadership
part, but that's not quite as necessary (I think) because the project
has a clear goal (to a certain extent): SQL compliance.

I think operating systems are a particularly hard area because of the
amount of evolution going on and the amount of work needed just to keep
working on newer machines. The field of databases and SQL is nowhere
near that difficult.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Josh,

It is current, to the point and has some direct correlations with our
project that we may want to be aware of.

Well, we're not in any danger of the board of a foundation taking over
Postgres. ;-)

The only part of this that I see as relevant to us is setting of
development goals. And we've already discussed this ad nauseum on the
Hackers list and AFAIK have an initial plan (the enhanced TODO), lacking
only the resources to implement it this month.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

On 8/31/06, Josh Berkus <josh@agliodbs.com> wrote:

The only part of this that I see as relevant to us is setting of
development goals. And we've already discussed this ad nauseum on the
Hackers list and AFAIK have an initial plan (the enhanced TODO), lacking
only the resources to implement it this month.

As a proponent of PostgreSQL over MySQL and other database inside a
bunch of companies, one thing that's been problematic is the fact that
feature set is accidental, or appears that way.

I totally understand that people want to work on what they want to
work on, but there are obvious places that Postgres has issues that
make it less competitive. I can't go to anyone in the company and say
"8.3 will solve X," and in fact, until 8.2 hits the release, I'll not
really know what's in it. This makes planning difficult.

That's my main real complaint.

Chris
--
| Christopher Petrilli
| petrilli@gmail.com

The only part of this that I see as relevant to us is setting of
development goals. And we've already discussed this ad nauseum on the
Hackers list and AFAIK have an initial plan (the enhanced TODO), lacking
only the resources to implement it this month.

Almost the whole thing is relevant :). Keep in mind that I am not saying
that it is negative. For example the NetBSD core is obviously cranked,
where our Core tends to stay out of the way. That is a positive.

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

We do have portions of a meritocracy in place but we are by no means
mature in that arena. Likely because of our lock problem ;)

We are also better at having cross over between sub projects so that
many people who are the same people are part of many projects. This
allows communication to flow between sub projects.

Not perfect of course :) but better then many I see.

Another odd issue, which may or may not be a positive is that we don't
have a public leader. We have half a dozen people (less I think) that
are very, very public (I am not talking mailing list public).

Anyway, the post as I said was for provoking thought, not for
antagonistic measures. I saw good and bad and thought it would be good
for everyone to review as we are as a project dealing with some of our
own growth problems.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Joshua D. Drake wrote:

The only part of this that I see as relevant to us is setting of
development goals. And we've already discussed this ad nauseum on the
Hackers list and AFAIK have an initial plan (the enhanced TODO), lacking
only the resources to implement it this month.

Almost the whole thing is relevant :). Keep in mind that I am not saying

I totally agree!

that it is negative. For example the NetBSD core is obviously cranked,
where our Core tends to stay out of the way. That is a positive.

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Yep, but fortunately this problem doesn't happen to us often.

Anyway, the post as I said was for provoking thought, not for
antagonistic measures. I saw good and bad and thought it would be good
for everyone to review as we are as a project dealing with some of our
own growth problems.

Yes. There are lessons to be learned.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Martijn van Oosterhout wrote:
-- Start of PGP signed section.

On Thu, Aug 31, 2006 at 09:11:52AM -0700, Joshua D. Drake wrote:

I thought some people in this group may find this letter from one of
NetBSD's founders very interesting.

http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html

It is current, to the point and has some direct correlations with our
project that we may want to be aware of.

Nice post, though I don't think PostgreSQL really has many of the
faults he lists. The only obvious one to me is the strong leadership
part, but that's not quite as necessary (I think) because the project
has a clear goal (to a certain extent): SQL compliance.

I think the issue is complacent leadership on the one hand, vs. a single
forceful leader on the other. I think the best configuration somewhere
is in the middle, which is what we have. I don't see how it is related
to the OS problem domain.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Josh,

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Yep, and that was immediately recognized as a problem in need of a
solution. In fact, some of the arguments againts the issue/feature
tracker were that it would encourage the locked project issue. So the
NetBSD experience should inform our design of the future feature/bug
tracker: it should be used to encourage new developers (by providing clear
specs and status information) rather than locking in old ones.

We do have portions of a meritocracy in place but we are by no means
mature in that arena. Likely because of our lock problem ;)

What specific issues do you see? We're pretty strongly merit-based -- the
only reservation I see on that is a bias toward more eloquent writers
having disproprotionate influence. But I don't see any way to avoid that.

Another odd issue, which may or may not be a positive is that we don't
have a public leader. We have half a dozen people (less I think) that
are very, very public (I am not talking mailing list public).

Actually, this issue is a complete red herring. People like to point to
Linux as successful because of Linus's benevolent dictatorship, but Linus
is the exception rather than the rule. Most of the very successful
projects (Apache, Perl, MySQL, Debian, X.org, etc.) are led by councils or
companies without a dictator. I can name more than a few projects where
the "charismatic leader" was the main thing preventing the project's
success.

In general, I think that people who harp on PostgreSQL's lack of a
benevolent dictator as an inhibitor to progress are people who are not
comfortable with democracy and are looking for excuses why company X needs
to "take over the project for its own good."

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

In general, I think that people who harp on PostgreSQL's lack of a
benevolent dictator as an inhibitor to progress are people who are not
comfortable with democracy and are looking for excuses why company X needs
to "take over the project for its own good."

Well I definitely don't think we need a benevolent dictator... however
considering the relatively small number of people in the public eye, a
definition of goals that we all speak too might be good :)

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Thu, Aug 31, 2006 at 11:18:27AM -0700, Joshua D. Drake wrote:

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Maybe, but we don't have the extreme form. Patches have been submitted
by people other than the ones saying they'd do it, and no-one got their
head bitten off for it. Indeed, the original person was often grateful
that it wasn't their problem anymore.

One thing about the discussion about locking was where we wanted a more
formal locking strategy (keeping a list). I think this is the wrong
approach. If you want some feature that hasn't seen any recent
discussion, *do it*, don't wait around seeing if someone else will do
it. This was in the article also:

... there was no sense that anyone else "owned" a piece
of Linux (although de facto "ownership" has happened in some parts);
if you didn't produce, Linus would use someone else's code. If you
wanted people to use your stuff, you had to keep moving.

I really think that's a better idea than tracking who is doing what.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Yep, but fortunately this problem doesn't happen to us often.

I think this might happen more then you think. I ran into it with Alvaro
just a couple of days ago. I brought up 3/4 items I thought he might be
interested in working on for 8.3.

The immediate response was well that is such a person's or that a person's.

Now, all we have to do is actually communicate ;) to make sure that we
move forward to eliminate the lock and we will. However it does point to
the fact that not everyone is going to take that extra step, some are
going to assume that it is being worked on.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

In response to "Joshua D. Drake" <jd@commandprompt.com>:

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Yep, but fortunately this problem doesn't happen to us often.

I think this might happen more then you think. I ran into it with Alvaro
just a couple of days ago. I brought up 3/4 items I thought he might be
interested in working on for 8.3.

The immediate response was well that is such a person's or that a person's.

Now, all we have to do is actually communicate ;) to make sure that we
move forward to eliminate the lock and we will. However it does point to
the fact that not everyone is going to take that extra step, some are
going to assume that it is being worked on.

In my experience, some of this is culture. Some groups communicate more
easily than others. When people don't communicate well, stuff has to
be done to encourage it. At the extreme end, stuff has to be done to
enforce it.

I think it's best if it happens naturally, but you can't always count on
that.

--
Bill Moran
Collaborative Fusion Inc.

Josh Berkus <josh@agliodbs.com> writes:

In general, I think that people who harp on PostgreSQL's lack of a
benevolent dictator as an inhibitor to progress are people who are not
comfortable with democracy and are looking for excuses why company X needs
to "take over the project for its own good."

I don't recall having seen that idea being pushed for Postgres ... not
seriously anyway. However, it's certainly true that historically we've
had effectively *no* project leadership, in the sense of anyone setting
feature goals for releases or creating a long-term roadmap. Would we
be better off if we had done that? I'm not sure.

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who they're
paid by. So I tend to think that a project roadmap would be more of an
exercise in wishful thinking than a useful management tool. OTOH it
*could* be useful, if there are any developers out there wondering what
they should work on next. Are there any ... and would they listen to a
roadmap if they had one, rather than scratching their own itches?

regards, tom lane

bruce@momjian.us (Bruce Momjian) writes:

Joshua D. Drake wrote:

The only part of this that I see as relevant to us is setting of
development goals. And we've already discussed this ad nauseum on the
Hackers list and AFAIK have an initial plan (the enhanced TODO), lacking
only the resources to implement it this month.

Almost the whole thing is relevant :). Keep in mind that I am not saying

I totally agree!

that it is negative. For example the NetBSD core is obviously cranked,
where our Core tends to stay out of the way. That is a positive.

On the other hand, we do suffer from the locked project problem (the
recent recursive query debacle is a perfect example).

Yep, but fortunately this problem doesn't happen to us often.

It seems to me that PostgreSQL doesn't suffer in similar degree from
the "oh, dear, someone has that TODO item; now noone else can ever
look at it!" problem.

The recursive query situation is one where, while they may have missed
the 8.2 release, it's not as if this represents something that sat and
sat and which will see no action for 8.3 because "so and so has it on
their list..."

Anyway, the post as I said was for provoking thought, not for
antagonistic measures. I saw good and bad and thought it would be
good for everyone to review as we are as a project dealing with
some of our own growth problems.

Yes. There are lessons to be learned.

I'm not sure how much can be readily learned from the "Oh, dear, we
bundle whatever features people come in with" problem. Managing a
free software project is quite a bit like herding cats; what you get
is what their wanderings resulted in.

To the degree to which people volunteer for things that strike them as
individually interesting, I don't see how you change that.

The classic item for PostgreSQL that people wish it had which has a
pretty wide-open kind of scope is the in-place upgrade. There's the
suggestion that perhaps one of the Sun folk will look at it for 8.3;
that's definitely NOT the sort of thing that fits into the
'independent volunteer noodling around with a feature they find
interesting' category.
--
output = reverse("gro.gultn" "@" "enworbbc")
http://www.ntlug.org/~cbbrowne/linuxxian.html
Rules of the Evil Overlord #47. "If I learn that a callow youth has
begun a quest to destroy me, I will slay him while he is still a
callow youth instead of waiting for him to mature."
<http://www.eviloverlord.com/&gt;

tgl@sss.pgh.pa.us (Tom Lane) writes:

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who
they're paid by. So I tend to think that a project roadmap would be
more of an exercise in wishful thinking than a useful management
tool. OTOH it *could* be useful, if there are any developers out
there wondering what they should work on next. Are there any
... and would they listen to a roadmap if they had one, rather than
scratching their own itches?

It seems to me that there's a vital difference between "sponsored
developers" and "hobbyists," here.

If a sponsored developer's work isn't providing their own organization
with value, then some managers outside the scope of PGDG will
doubtless demand some answers. And if the organization wants some
particular features, that may help guide some developers without PGDG
having expended any "managerial effort."

Those "answerability mechanisms" don't apply to as material a degree
to hobbyists.
--
"cbbrowne","@","ntlug.org"
http://cbbrowne.com/info/wp.html
if (argc > 1 && strcmp(argv[1], "-advice") == 0) {
printf("Don't Panic!\n");
exit(42);
}
(Arnold Robbins in the LJ of February '95, describing RCS)

Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

In general, I think that people who harp on PostgreSQL's lack of a
benevolent dictator as an inhibitor to progress are people who are not
comfortable with democracy and are looking for excuses why company X needs
to "take over the project for its own good."

I don't recall having seen that idea being pushed for Postgres ... not
seriously anyway. However, it's certainly true that historically we've
had effectively *no* project leadership, in the sense of anyone setting
feature goals for releases or creating a long-term roadmap. Would we
be better off if we had done that? I'm not sure.

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who they're
paid by. So I tend to think that a project roadmap would be more of an
exercise in wishful thinking than a useful management tool. OTOH it
*could* be useful, if there are any developers out there wondering what
they should work on next. Are there any ... and would they listen to a
roadmap if they had one, rather than scratching their own itches?

I think the longer someone is with the project the more they start
working on what is good for the project, rather than what interests
them. I think we have seen many cases of that.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Tom Lane wrote:

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who they're
paid by. So I tend to think that a project roadmap would be more of an
exercise in wishful thinking than a useful management tool. OTOH it
*could* be useful, if there are any developers out there wondering what
they should work on next. Are there any ... and would they listen to a
roadmap if they had one, rather than scratching their own itches?

I would certainly listen to a roadmap if it talked to me ...

I think the longer someone is with the project the more they start
working on what is good for the project, rather than what interests
them. I think we have seen many cases of that.

On my particular case, I generally grab some problem that I perceive as
important and unhandled, and try to do something to remedy it. This is
how I got here in the first place, by fixing some problems in the
CLUSTER implementation. This is how I got to doing shared dependencies,
shared row locks and autovacuum -- neither of them were problems that
affected me in any way. Savepoints were a different matter. I chose to
work on them because Bruce and other people on this list suggested them
to me, back when I was looking for something to do my undergrad project
in.

So yes, I'd probably work on something "the community" considered
important.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

[ hijacking this thread over to where the developers hang out ]

Alvaro Herrera <alvherre@commandprompt.com> writes:

Tom Lane wrote:

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who they're
paid by. So I tend to think that a project roadmap would be more of an
exercise in wishful thinking than a useful management tool. OTOH it
*could* be useful, if there are any developers out there wondering what
they should work on next. Are there any ... and would they listen to a
roadmap if they had one, rather than scratching their own itches?

I would certainly listen to a roadmap if it talked to me ...

Well, this question keeps coming up, and we keep arguing about it, and
we still have no data to say whether it would work well for *this*
project. Maybe it's time to take the bull by the horns.

I propose a modest experiment: for the 8.3 development cycle, let's try
to agree (in the next month or so) on a roadmap of what major features
should be in 8.3 and who will make each one happen. A year from now,
we will know whether this is a great thing we should continue, or we
should stick to our traditional laissez-faire style of project
management. I figure that even if it really sucks, it wouldn't kill us
to try it for one release cycle --- at the very worst, we'd make up lost
time in future by no longer needing to waste bandwidth arguing about it.

regards, tom lane

On Aug 31, 2006, at 8:47 PM, Tom Lane wrote:

[ hijacking this thread over to where the developers hang out ]

Alvaro Herrera <alvherre@commandprompt.com> writes:

Tom Lane wrote:

It's pointless to suppose that individual developers would really be
answerable to any project-wide management, since that's not who
they're
paid by. So I tend to think that a project roadmap would be more
of an
exercise in wishful thinking than a useful management tool. OTOH it
*could* be useful, if there are any developers out there
wondering what
they should work on next. Are there any ... and would they
listen to a
roadmap if they had one, rather than scratching their own itches?

I would certainly listen to a roadmap if it talked to me ...

Well, this question keeps coming up, and we keep arguing about it, and
we still have no data to say whether it would work well for *this*
project. Maybe it's time to take the bull by the horns.

I propose a modest experiment: for the 8.3 development cycle, let's
try
to agree (in the next month or so) on a roadmap of what major features
should be in 8.3 and who will make each one happen. A year from now,
we will know whether this is a great thing we should continue, or we
should stick to our traditional laissez-faire style of project
management. I figure that even if it really sucks, it wouldn't
kill us
to try it for one release cycle --- at the very worst, we'd make up
lost
time in future by no longer needing to waste bandwidth arguing
about it.

Would this be a core postgresql code roadmap or something a bit
broader (contrib, custom types, GUI-ish stuff, utilities and what
have you)?

Cheers,
Steve