Kudos for Reviewers -- straw poll

On 25.06.2013 20:17, Josh Berkus wrote:

Hackers,

I'd like to take a straw poll here on how we should acknowledge
reviewers. Please answer the below with your thoughts, either on-list
or via private email.

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

a)

Sometimes a reviewer contributes greatly to the patch, revising it and
rewriting parts of it. At that point, it's not just a review anymore,
and he/she should be mentioned in the release notes as a co-author.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

This is one reason why I answered a) above. I don't want to set a criteria.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

a).

I don't think we should make any promises, though. Just arbitrarily send
a t-shirt when you feel that someone has done a good job reviewing other
people's patches. And I'm not sure it's really worth the trouble, to
arrange the logistics etc.

- Heikki

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, June 25, 2013 19:17, Josh Berkus wrote:

How should reviewers get credited in the release notes?

b) in a single block titled "Reviewers for this version" at the bottom.

Should there be a criteria for a "creditable" review?

b) yes, they have to do more than "it compiles"

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

b) no

Erik Rijkers

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2013-06-25 10:17:07 -0700, Josh Berkus wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

b).

If the review was substantial enough the reviewer gets bumped to a
secondary author, just as it already happens.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

b). Surely performance reviews should also count, they can be at least
as time consuming as a code review, so c) doesn't seem to make sense.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

Not sure. Seems like it might be a way to spend a lot of effort without
achieving all that much. But I can also imagine that it feels nice and
encourages a casual reviewer/contributor.

So it's either b) or c). Although I'd perhaps exclude regular
contributors to keep the list reasonable?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 10:46 AM, Andres Freund wrote:

Not sure. Seems like it might be a way to spend a lot of effort without
achieving all that much. But I can also imagine that it feels nice and
encourages a casual reviewer/contributor.

So it's either b) or c). Although I'd perhaps exclude regular
contributors to keep the list reasonable?

Well, one of the other "prizes" which occurred to me today would be a
pgCon lottery. That is, each review posted by a non-committer would go
in a hat, and in February we would draw one who would get a free
registration and airfare to pgCon.

Seems apropos and without the horrible logistics issues of mailing
tshirts to 15 countries.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 10:17 AM, Josh Berkus wrote:

Hackers,

I'd like to take a straw poll here on how we should acknowledge
reviewers. Please answer the below with your thoughts, either on-list
or via private email.

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

C. The idea that reviewers are somehow less than authors is rather
disheartening.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

B. I think it compiles, and I tested it via X should be the minimum.
Here is a case. I was considering taking a review of the new Gin Cache
patch. I can't really do a "code" review but I can certainly run
benchmarking tests before/after and apply the patch etc.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

Thanks for your feedback!

B. We already give them a multi-million dollar piece of software for free.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2013-06-25 11:04:38 -0700, Joshua D. Drake wrote:

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

C. The idea that reviewers are somehow less than authors is rather
disheartening.

It's not about the reviewers being less. It's a comparison of
effort. The effort for a casual review simply isn't comparable with the
effort spent on developing a nontrivial patch.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 01:17 PM, Josh Berkus wrote:

Hackers,

I'd like to take a straw poll here on how we should acknowledge
reviewers. Please answer the below with your thoughts, either on-list
or via private email.

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

b) seems about right.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

c). Compilation, functionality and performance tests are useful, but
what we desperately need are in depth code reviews of large patches. If
we debase the currency by rewarding things less than that then any
incentive effect of kudos in encouraging more reviews is lost.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

I'd like to see prizes each release for "best contribution" and "best
reviewer" - I've thought for years something like this would be worth
trying. Committers and core members should not be eligible - this is
about encouraging new people.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 11:26 AM, Andres Freund wrote:

On 2013-06-25 11:04:38 -0700, Joshua D. Drake wrote:

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

C. The idea that reviewers are somehow less than authors is rather
disheartening.

It's not about the reviewers being less. It's a comparison of
effort. The effort for a casual review simply isn't comparable with the
effort spent on developing a nontrivial patch.

I think this is a backwards way to look at it.

The effort may not be comparable but the drudgery certainly is.

Reviewing patches sucks. Writing patches (for the most part) is fun.

Should the patch submitter get first billing? Yes.
Should the reviewer that makes sure to a reasonable level that the patch
is sane also get billing? Absolutely.
Should the reviewer get billing that is about the patch they reviewed. Yes.

As I mentioned before in the release notes something like:

Author: Tom Lane
Reviewer(s): Greg Stark, Andrew Dunstan

I think that is perfectly reasonable.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc
For my dreams of your image that blossoms
a rose in the deeps of my heart. - W.B. Yeats

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Jun 25, 2013 at 2:17 PM, Josh Berkus <josh@agliodbs.com> wrote:

How should reviewers get credited in the release notes?
c) on the patch they reviewed, for each patch

This not only makes sense, it also lets people reading release notes
know there's been a review, and how thorough it was. I know, all
changes in PG get reviewed, but having it explicit on release notes I
imagine would be useful. Especially if I know the reviewers.

The co-author part also makes a lot of sense. When a reviewer
introduces changes directly, and they get committed, I think it should
be automatically considered co-authoring the patch.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile

It's not that they're all worthwhile, but arbitrary decisions lend
themselves to arbitrary criticism. Whatever criteria should be
straightforward, unambiguous and unbiased, and it's hard getting all
those three in any other criteria than: all are worthwhile.

b) yes, they have to do more than "it compiles"

This one's better than nothing, if you must have a minimum criteria.
But then people will just point out some trivial stuff and you'd be
tempted to say that trivialities don't count... and you get a snowball
going. IMHO, it's all or nothing.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

b) no

Yeah, while a fun idea, I don't think the logistics of it make it
worth the effort. Too much effort for too little benefit.

And I think recognition is a far better incentive than T-shirts
anyway. I know I'd be encouraged to review patches for the recognition
alone, a lot more than for a T-shirt I might not get. Contributing is
nice, but feeling appreciated while doing so is better.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 11:26 AM, Andres Freund wrote:

On 2013-06-25 11:04:38 -0700, Joshua D. Drake wrote:

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

C. The idea that reviewers are somehow less than authors is rather
disheartening.

It's not about the reviewers being less. It's a comparison of
effort. The effort for a casual review simply isn't comparable with the
effort spent on developing a nontrivial patch.

On the other hand, I will point out that we currently have a shortage of
reviewers, and we do NOT have a shortage of patch submitters. Seems
like we should apply incentives where we need help, no?

Mind you, my votes are (B), (A), and (A).

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 25 June 2013 18:17, Josh Berkus <josh@agliodbs.com> wrote:

Hackers,

I'd like to take a straw poll here on how we should acknowledge
reviewers. Please answer the below with your thoughts, either on-list
or via private email.

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

b) Unless they contribute enough to the patch to be considered a co-author.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

a) Sometimes even "it compiles" can be worthwhile, if there is doubt
over platform compatibility. All contributions should be acknowledged
and encouraged.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

b) Getting your name in the fine manual is reward enough ;-)

Regards,
Dean

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Jun 25, 2013 at 10:17:07AM -0700, Josh Berkus wrote:

Hackers,

I'd like to take a straw poll here on how we should acknowledge
reviewers. Please answer the below with your thoughts, either on-list
or via private email.

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

c) This keeps history better.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

a). If it turns out that people are gaming this, or appear to be
gaming this, we can revisit the policy.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

b). You want to be *extremely* careful when paying volunteers. The
chances of damaging their intrinsic motivations are high, especially
when it's not stuff like covering their expenses.

http://www.iew.uzh.ch/wp/iewwp007.pdf

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 26 June 2013 03:17, Josh Berkus <josh@agliodbs.com> wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

A weak preference for (c), with (b) running a close second. As others
have suggested, a review that leads to significant commitable changes
to the patch should bump the credit to co-authorship.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

(b), the review should at least look at usabililty, doc, and
regression test criteria even if there is no in-depth code analysis.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

Provisionally (b), if we first try giving proper credit, and that
still doesn't drum up enough reviewing, then look to further incentive
schemes. No need to jump the gun.

Cheers,
BJ

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Brendan Jurd wrote

On 26 June 2013 03:17, Josh Berkus &lt;

josh@

&gt; wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

I think some consideration toward a "commit and review summary" (outside the
release notes; and graphical/interactive in nature ideally) for each major
release is something worth considering. With regards to the release notes
I'd lean toward (b); significant contributions getting bumped to co-author
on specific patches covers (c) fairly well. I am unsure whether release
note mentions are significant enough motivation...see other thoughts below.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

Ideally (a) though (b) conceptually makes sense but it is too generic.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

One low-cost "prize" that I've pondered is, on an ongoing basis, the ability
to post a link and/or message to the PostgreSQL front page within a
significantly less stringent barrier to "acceptance" than is required for
current content. Basically except for topics or presentations deemed of
poor taste or detrimental to the project anything should be allowed. Some
kind of "this message was allowed because so-and-so has recently made the
following significant contributions to the project". There are probably
quite a few logistics to deal with down this path but a sponsor platform for
shameless self-promotion for people making the project successful -
something visible on an ongoing basis and not just once a year in a release
note - is likely a very valuable to the contributor while fairly inexpensive
to the project (i.e., some risk of reputation and some cost to setup the
infrastructure).

David J.

--
View this message in context: http://postgresql.1045698.n5.nabble.com/Kudos-for-Reviewers-straw-poll-tp5760952p5761031.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Dean Rasheed wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

b) Unless they contribute enough to the patch to be considered a co-author.

Should there be a criteria for a "creditable" review?

a) no, all reviews are worthwhile
b) yes, they have to do more than "it compiles"
c) yes, only code reviews should count

a) Sometimes even "it compiles" can be worthwhile, if there is doubt
over platform compatibility. All contributions should be acknowledged
and encouraged.

Should reviewers for 9.4 get a "prize", such as a t-shirt, as a
promotion to increase the number of non-submitter reviewers?

a) yes
b) no
c) yes, but submitters and committers should get it too

b) Getting your name in the fine manual is reward enough ;-)

+1, except that I like Josh's idea about a free ticket to pgCon.

Yours,
Laurenz Albe

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

For me, B,B and another B works.

Regards,

Atri

--
Regards,

Atri
l'apprenant

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jun 26, 2013 at 10:40:17AM +1000, Brendan Jurd wrote:

On 26 June 2013 03:17, Josh Berkus <josh@agliodbs.com> wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

A weak preference for (c), with (b) running a close second. As others
have suggested, a review that leads to significant commitable changes
to the patch should bump the credit to co-authorship.

As a reminder, I tried a variant of C for 9.2 beta release notes, and
got lots of complaints, particularly because the line describing the
feature now had many more names on it.

In my opinion, adding reviewer names to each feature item might result
in the removal of all names from features.

A poll is nice for gauging interest, but many people who vote don't
understand the ramifications of what they are voting on.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/26/2013 09:14 AM, Bruce Momjian wrote:

On Wed, Jun 26, 2013 at 10:40:17AM +1000, Brendan Jurd wrote:

On 26 June 2013 03:17, Josh Berkus <josh@agliodbs.com> wrote:

How should reviewers get credited in the release notes?

a) not at all
b) in a single block titled "Reviewers for this version" at the bottom.
c) on the patch they reviewed, for each patch

A weak preference for (c), with (b) running a close second. As others
have suggested, a review that leads to significant commitable changes
to the patch should bump the credit to co-authorship.

As a reminder, I tried a variant of C for 9.2 beta release notes, and
got lots of complaints, particularly because the line describing the
feature now had many more names on it.

In my opinion, adding reviewer names to each feature item might result
in the removal of all names from features.

A poll is nice for gauging interest, but many people who vote don't
understand the ramifications of what they are voting on.

That's why I voted for b :-)

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 06/25/2013 08:26 PM, Andres Freund wrote:

It's not about the reviewers being less. It's a comparison of
effort. The effort for a casual review simply isn't comparable with the
effort spent on developing a nontrivial patch.

Remember: "Debugging is twice as hard as writing the code in the first
place. ..." (Brian Kernighan)

IMO, the kind of reviews we need are of almost "debug level" difficulty.
(To the point where the reviewer becomes a co-author or even takes over
and submits a completely revamped patch instead.)

I agree that the casual review is several levels below that, so your
point holds. I doubt we need more reviews of that kind, though.

Thus, I'm in the AAB camp. The remaining question being: What's the
criterion for becoming a co-author (and thus getting mentioned in the
release notes)?

If at all, we should honor quality work with a "prize". Maybe a price
for the best reviewer per CF? Maybe even based on votes from the general
public on who's been the best, so reviews gain attention that way...
"Click here to vote for my review." ... Maybe a crazy idea.

Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers