MMAP Buffers

On 15.04.2011 13:32, Radosław Smogura wrote:

If I may, I want to share some concept to use mmap in PG. It's far, far
away from perfect, but it's keeps WAL before data. As well I crated
table, with index, inserted few values, and I done vacuum full on this
table. Db inits welcome from orginal sources.

Performance of read (if backend is loaded) is really good, query time
goes down from 450ms to about 410ms. Update may be slower - but work is
in progress (I will start with write, as I went to point when simple
updates may be performed). Even that I didn't covered all aspects off
updating, it's simple to do it, just to call PreopareBufferToUpdate
before modifing buffer, ofc some ideas of increasing this are still in
my head.

Any comments, suggestions welcome.

The patch is quite hard to read because of random whitespace changes and
other stylistic issues, but I have a couple of high-level questions on
the design:

* Does each process have its own mmappings, or are the mmapping done to
shared buffers?

* How do you handle locking? Do you still need to allocate a shared
buffer for each mmapped page?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, 15 Apr 2011 14:33:37 +0300, Heikki Linnakangas wrote:

On 15.04.2011 13:32, Radosław Smogura wrote:

If I may, I want to share some concept to use mmap in PG. It's far,
far
away from perfect, but it's keeps WAL before data. As well I crated
table, with index, inserted few values, and I done vacuum full on
this
table. Db inits welcome from orginal sources.

Performance of read (if backend is loaded) is really good, query
time
goes down from 450ms to about 410ms. Update may be slower - but work
is
in progress (I will start with write, as I went to point when simple
updates may be performed). Even that I didn't covered all aspects
off
updating, it's simple to do it, just to call PreopareBufferToUpdate
before modifing buffer, ofc some ideas of increasing this are still
in
my head.

Any comments, suggestions welcome.

The patch is quite hard to read because of random whitespace changes
and other stylistic issues, but I have a couple of high-level
questions on the design:

Yes, but, hmm... in Netbeans I had really long gaps (probably 8 spaces,
from tabs), so deeper "ifs", comments at the and of variables, went of
out my screen. I really wanted to not format this, but sometimes I
needed.

* Does each process have its own mmappings, or are the mmapping done
to shared buffers?

Those are MAP_SHARED mappings, but each process has it's own pointer to
this.

* How do you handle locking?

I do not do locking... I do different thing (worst)... When buffer
should be updated, it gets shared buffer, content is copied (so
situation almost same like fread), and depending on situation content is
used directly or pages between mmaping and shared (updatable) regions
are swapped - it keeps tuple pointers, etc. I really would be happy if
such method (lock flushing to file) could exists.

Do you still need to allocate a shared buffer for each mmapped page?

Currently each mmaped page has additional shared buffer, but it's
almost ready to use independent pool of shared buffers. This will be
good, as mmaped buffers could cover whole system cache, keeping maybe
10%-20% of this size for write in SHMEM.

I think about MAP_PRIVATE, but those has some pluses and minuses, e.g.
MAP_SHARED may be, for less critical systems, simplier equipped with GUC
mmap_direct_write=true.

Regards,
Radek

On 04/15/2011 08:12 AM, Radosław Smogura wrote:

The patch is quite hard to read because of random whitespace changes
and other stylistic issues, but I have a couple of high-level
questions on the design:

Yes, but, hmm... in Netbeans I had really long gaps (probably 8
spaces, from tabs), so deeper "ifs", comments at the and of variables,
went of out my screen. I really wanted to not format this, but
sometimes I needed.

Netbeans is possibly not very well suited to working on postgres code.
AFAIK emacs and/or vi(m) are used by almost all the major developers.

cheers

andrew

Radoslaw,

10% improvement isn't very impressive from a switch to mmap. What workload did you test with? What I'd really like to see is testing with databases which are 50%, 90% and 200% the size of RAM ... that's where I'd expect the greatest gain from limiting copying.

Netbeans is possibly not very well suited to working on postgres code.
AFAIK emacs and/or vi(m) are used by almost all the major developers.

Guys, can we *please* focus on the patch for now, rather than the formatting, which is fixable with sed?
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

Joshua Berkus <josh@agliodbs.com> Friday 15 April 2011 18:55:04

Radoslaw,

10% improvement isn't very impressive from a switch to mmap. What workload
did you test with? What I'd really like to see is testing with databases
which are 50%, 90% and 200% the size of RAM ... that's where I'd expect
the greatest gain from limiting copying.

I think 10% is quite good, as my stand-alone test of mmap vs. read shown that
speed up of copying 100MB data to mem may be from ~20ms to ~100ms (depends on
destination address). Of course deeper, system test simulating real usage will
say more. In any case after good deals with writes, I will speed up reads. I
think to bypass smgr/md much more and to expose shared id's (1,2,3...) for
each file segment.

Going to topic...

In attachment I sent test-scripts which I used to fill data, nothing complex
(left from 2nd level caches).

Query I've used to measure was SELECT count(substr(content, 1, 1)) FROM
testcase1 WHERE multi_id > 50000;

Timings ware taken from psql.

I didn't made load (I have about 2GB of free sapce at /home, and 4GB RAM) and
stress (I'm not quite ready to try concurrent updates of same page - may fail,
notice is and place to fix is in code) tests yet.

Netbeans is possibly not very well suited to working on postgres code.
AFAIK emacs and/or vi(m) are used by almost all the major developers.

Guys, can we *please* focus on the patch for now, rather than the
formatting, which is fixable with sed?

Netbeans is quite good, of course it depends who likes what. Just try 7.0 RC
2.

Regards,
Radek

Joshua Berkus wrote:

Guys, can we *please* focus on the patch for now, rather than the formatting, which is fixable with sed?

Never, and that's not true. Heikki was being nice; I wouldn't have even
slogged through it long enough to ask the questions he did before
kicking it back as unusable. A badly formatted patch makes it
impossible to evaluate whether the changes from a submission are
reasonable or not without the reviewer fixing it first. And you can't
automate correcting it, it takes a lot of tedious manual work. Start
doing a patch review every CommitFest cycle and you very quickly realize
it's not an ignorable problem. And lack of discipline in minimizing
one's diff is always a sign of other code quality issues.

Potential contributors to PostgreSQL should know that a badly formatted
patch faces an automatic rejection, because no reviewer can work with it
easily. This fact is not a mystery; in fact it's documented at
http://wiki.postgresql.org/wiki/Submitting_a_Patch : "The easiest way
to get your patch rejected is to make lots of unrelated changes, like
reformatting lines, correcting comments you felt were poorly worded etc.
Each patch should have the minimum set of changes required to fulfil the
single stated objective." I think I'll go improve that text
next--something like "Ways to get your patch rejected" should be its own
section.

The problem here isn't whether someone used an IDE or not, it's that
this proves they didn't read their own patch before submitting it.
Reading one's own diff and reflecting on what you've changed is one of
the extremely underappreciated practices of good open-source software
development. Minimizing the size of that diff is perhaps the most
important thing someone can do in order to make their changes to a piece
of software better. Not saying something that leads in that direction
would be a disservice to the submitter.

P.S. You know what else I feel should earn an automatic rejection
without any reviewer even looking at the code? Submitting a patch that
claims to improve performance and not attaching the test case you used,
along with detailed notes about before/after tests on your own
hardware. A hand wave "it's faster" is never good enough, and it's
extremely wasteful of our limited reviewer resources to try and
duplicate what the submitter claimed. Going to add something about that
to the submission guidelines too.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Apr 16, 2011, at 1:48 AM, Greg Smith <greg@2ndquadrant.com> wrote:

P.S. You know what else I feel should earn an automatic rejection without any reviewer even looking at the code?

Greg is absolutely right. And to the two he listed, let me add another of my own gripes: failing to provide submission notes that explain how the patch works, and how it addresses the conceptually difficult issues raised previously. The OP says that this patch maintains the WAL-before-data rule without any explanation of how it accomplishes that seemingly quite amazing feat. I assume I'm going to have to read this patch at some point to refute this assertion, and I think that sucks. I am pretty nearly 100% confident that this approach is utterly doomed, and I don't want to spend a lot of time on it unless someone can provide me with a compelling explanation of why my confidence is misplaced. But spending a lot of time on it is exactly what I'm going to have to do, because reading a undocumented patch full of spurious garbage to refute a hand-wavy claim of correctness is time-consuming, and if I give up on it without reading it, someone will yell "unfair, unfair!"

None of this is to say that I don't appreciate Radoslaw's interest in contributing, because I very much do. But I also think it's important to realize that we have a finite number of reviewers and they have finite time. Trying to minimize the amount of time that it takes someone to review or commit your patch is a service to the whole community, and we should acknowledge that it has value and appreciate the people who consistently do it.

...Robert

Greg Smith <greg@2ndquadrant.com> writes:

Reading one's own diff and reflecting on what you've changed is one of
the extremely underappreciated practices of good open-source software
development. Minimizing the size of that diff is perhaps the most
important thing someone can do in order to make their changes to a piece
of software better. Not saying something that leads in that direction
would be a disservice to the submitter.

A couple further comments on that thought:

* Another argument for avoiding unnecessary changes is that the larger
your patch's change footprint, the more likely it is to create merge
conflicts for people working on other patches. Now, if they're
necessary changes, that's the price of parallelism in development.
But gratuitous whitespace changes add nothing and they do have costs.

* On the other side of the coin, I have seen many a patch that was
written to minimize the length of the diff to the detriment of
readability or maintainability of the resulting code, and that's *not*
a good tradeoff. Always do what makes the most sense from a long-run
perspective.

I keep wanting to do a talk arguing that everything you need to know
about good patch style can be derived from the mantra "Make the patch
look like the code had always been there". If the functionality had
been designed in on day one, where would it be placed and how would it
be coded? You might be able to make the patch diff shorter with some
shortcut or other, but that's not the way to do it.

regards, tom lane

Tom Lane wrote:

* On the other side of the coin, I have seen many a patch that was
written to minimize the length of the diff to the detriment of
readability or maintainability of the resulting code, and that's *not*
a good tradeoff.

Sure. that's possible. But based on the reviews I've done, I'd say that
the fact someone is even aware that minimizing their diff is something
important to consider automatically puts them far ahead of the average
new submitter. There are a high percentage of patches where the
submitter generates a diff and sents it without even looking at it.
That a person would look at their diff and go too far without trying to
make it small doesn't happen nearly as much.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Robert Haas wrote:

The OP says that this patch maintains the WAL-before-data rule without any explanation of how it accomplishes that seemingly quite amazing feat. I assume I'm going to have to read this patch at some point to refute this assertion, and I think that sucks.

I don't think you have to read any patch that doesn't follow the
submission guidelines. The fact that you do is a great contribution to
the community. But if I were suggesting how your time would be best
spent improving PostgreSQL, "reviewing patches that don't meet coding
standards" would be at the bottom of the list. There's always something
better for the project you could be working on instead.

I just added
http://wiki.postgresql.org/wiki/Submitting_a_Patch#Reasons_your_patch_might_be_returned
, recycling some existing text, adding some new suggestions.

I hope I got the tone of that text right. The intention was to have a
polite but clear place to point submitters to when their suggestion
doesn't meet the normal standards here, such that they might even get
bounced before even entering normal CommitFest review. This MMAP patch
looks like it has all 5 of the problems mentioned on that now more
focused list.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Sat, Apr 16, 2011 at 7:24 AM, Robert Haas <robertmhaas@gmail.com> wrote:

The OP says that this patch maintains the WAL-before-data rule without any explanation of how it accomplishes that seemingly quite amazing feat.  I assume I'm going to have to read this patch at some point to refute this assertion, and I think that sucks. I am pretty nearly 100% confident that this approach is utterly doomed, and I don't want to spend a lot of time on it unless someone can provide me with a compelling explanation of why my confidence is misplaced.

Fwiw he did explain how he did that. Or at least I think he did --
it's possible I read what I expected because what he came up with is
something I've recently been thinking about.

What he did, I gather, is treat the mmapped buffers as a read-only
copy of the data. To actually make any modifications he copies it into
shared buffers and treats them like normal. When the buffers get
flushed from memory they get written and then the pointers get
repointed back at the mmapped copy. Effectively this means the shared
buffers get extended to include all of the filesystem cache instead of
having to evict buffers from shared buffers just because you want to
read another one that's already in filesystem cache.

It doesn't save the copying between filesystem cache and shared
buffers for buffers that are actually being written to. But it does
save some amount of other copies on read-only traffic and it can even
save some i/o. It does require a function call before each buffer
modification where the pattern is currently <lock buffer>, <mutate
buffer>, <mark buffer dirty>. From what he describes he needs to add a
<prepare buffer for mutation> between the lock and mutate.

I think it's an interesting experiment and it's good to know how to
solve some of the subproblems. Notably, how do you extend files or
drop them atomically across processes? And how do you deal with
getting the mappings to be the same across all the processes or deal
with them being different? But I don't think it's a great long-term
direction. It just seems clunky to have to copy things from mmapped
buffers to local buffers and back. Perhaps the performance testing
will show that clunkiness is well worth it but we'll need to see that
for a wide variety of workloads to judge that.

--
greg

Greg Stark <gsstark@mit.edu> Saturday 16 April 2011 13:00:19

On Sat, Apr 16, 2011 at 7:24 AM, Robert Haas <robertmhaas@gmail.com> wrote:

The OP says that this patch maintains the WAL-before-data rule without
any explanation of how it accomplishes that seemingly quite amazing
feat. I assume I'm going to have to read this patch at some point to
refute this assertion, and I think that sucks. I am pretty nearly 100%
confident that this approach is utterly doomed, and I don't want to
spend a lot of time on it unless someone can provide me with a
compelling explanation of why my confidence is misplaced.

Fwiw he did explain how he did that. Or at least I think he did --
it's possible I read what I expected because what he came up with is
something I've recently been thinking about.

What he did, I gather, is treat the mmapped buffers as a read-only
copy of the data. To actually make any modifications he copies it into
shared buffers and treats them like normal. When the buffers get
flushed from memory they get written and then the pointers get
repointed back at the mmapped copy. Effectively this means the shared
buffers get extended to include all of the filesystem cache instead of
having to evict buffers from shared buffers just because you want to
read another one that's already in filesystem cache.

It doesn't save the copying between filesystem cache and shared
buffers for buffers that are actually being written to. But it does
save some amount of other copies on read-only traffic and it can even
save some i/o. It does require a function call before each buffer
modification where the pattern is currently <lock buffer>, <mutate
buffer>, <mark buffer dirty>. From what he describes he needs to add a
<prepare buffer for mutation> between the lock and mutate.

I think it's an interesting experiment and it's good to know how to
solve some of the subproblems. Notably, how do you extend files or
drop them atomically across processes? And how do you deal with
getting the mappings to be the same across all the processes or deal
with them being different? But I don't think it's a great long-term
direction. It just seems clunky to have to copy things from mmapped
buffers to local buffers and back. Perhaps the performance testing
will show that clunkiness is well worth it but we'll need to see that
for a wide variety of workloads to judge that.

In short words, I swap, exchange (clash of terms) VM pages to prevent pointers
(only if needed). I tried to directly point to new memory area, but I saw that
some parts of code really depends on memory pointed by original pointers, e.g.
Vaccumm uses hint bits setted by previous scan (it depends on this if bit is
set or not! so for it it's not only hint). Just from this case I can't assume
there is no more such places, so VM pages swap does it for me.

Stand alone tests shows for me that this process (with copy from mmap) is
2x-3x time longer then previous. But until someone will not update whole
table, then benefit will be taken from pre-update scan, index scans, larger
availability of memory (you don't eat cache memory to keep copy of cache in
ShMem). Everything may be slower when database fits in ShMem, and similarly
(2nd level bufferes may increase performance slightly).

I reserve memory for whole segment even if file is smaller. Extending is by
wirte one byte at the end of block (here may come deal with Unfiorm Buffer
Caches, if I remember name well). For current processors, and current
implementation database size is limited to about 260TB (no dynamic segment
reservation is performed).

Truncation not implemented.

Each buffer descriptor has tagVersion to simple check if buffer tag has
changed. Descriptors (partially) are mirrored in local memory, and versions
are checked. Currently each re-read (is pointed to smgr/md), but introduce
shared segment id, and assuming each segment has constant maximum number of
blocks, will make it faster (this will be something like current buffer tag),
even version field will be unneeded.

I saw problems with vacuum, as it reopens relation and I got mappings of same
file twice (minor problem). Important will be about deletion, when pointers
must invalidated in "good way".

Regards,
Radek.

On Sat, Apr 16, 2011 at 8:48 AM, Greg Smith <greg@2ndquadrant.com> wrote:

Joshua Berkus wrote:

Guys, can we *please* focus on the patch for now, rather than the
formatting, which is fixable with sed?

Never, and that's not true.  Heikki was being nice; I wouldn't have even
slogged through it long enough to ask the questions he did before kicking it
back as unusable.  A badly formatted patch makes it impossible to evaluate
whether the changes from a submission are reasonable or not without the
reviewer fixing it first.  And you can't automate correcting it, it takes a
lot of tedious manual work.  Start doing a patch review every CommitFest
cycle and you very quickly realize it's not an ignorable problem.  And lack
of discipline in minimizing one's diff is always a sign of other code
quality issues.

Potential contributors to PostgreSQL should know that a badly formatted
patch faces an automatic rejection, because no reviewer can work with it
easily.  This fact is not a mystery; in fact it's documented at
http://wiki.postgresql.org/wiki/Submitting_a_Patch :  "The easiest way to
get your patch rejected is to make lots of unrelated changes, like
reformatting lines, correcting comments you felt were poorly worded etc.
Each patch should have the minimum set of changes required to fulfil the
single stated objective."  I think I'll go improve that text next--something
like "Ways to get your patch rejected" should be its own section.

The problem here isn't whether someone used an IDE or not, it's that this
proves they didn't read their own patch before submitting it.  Reading one's
own diff and reflecting on what you've changed is one of the extremely
underappreciated practices of good open-source software development.
 Minimizing the size of that diff is perhaps the most important thing
someone can do in order to make their changes to a piece of software better.
 Not saying something that leads in that direction would be a disservice to
the submitter.

P.S. You know what else I feel should earn an automatic rejection without
any reviewer even looking at the code?  Submitting a patch that claims to
improve performance and not attaching the test case you used, along with
detailed notes about before/after tests on your own hardware.  A hand wave
"it's faster" is never good enough, and it's extremely wasteful of our
limited reviewer resources to try and duplicate what the submitter claimed.
 Going to add something about that to the submission guidelines too.

Give the OP a break - he was not "re-styling", he was clearly trying to make
crappily indented Postgres code readable in his editor. Reading the patch
would not matter because the original code would still be crappily indented.

Yes, such patch is bad, but what should the proper response be in such
situation?

Hint: it can be both polite and short.

--
marko

Greg Stark <gsstark@mit.edu> writes:

What he did, I gather, is treat the mmapped buffers as a read-only
copy of the data. To actually make any modifications he copies it into
shared buffers and treats them like normal. When the buffers get
flushed from memory they get written and then the pointers get
repointed back at the mmapped copy.

That seems much too late --- won't other processes still be looking at
the stale mmap'ed version of the page until a write-out happens?

I'm pretty concerned about the memory efficiency of this too, since it
seems like it's making it *guaranteed*, not just somewhat probable,
that there are two copies in RAM of every database page that's been
modified since the last checkpoint (or so).

regards, tom lane

On Fri, 2011-04-15 at 12:32 +0200, Radosław Smogura wrote:

I didn't included this, as diff, because of ~150kb size (mainly
configure scripts, which are included in SVC). Due to this, You may
download it from
http://softperience.eu/downloads/pg_mmap_20110415.diff.bz2 (Legal:
Work
under PostgreSQL BSD Lincense). Patch is just GIT diff, later I will
try
to grab some git.

Btw., about 87% of this patch are diffs against configure and
pg_config.h.in, which are useless. If you strip those out, your patch
will be small enough to submit inline.

All,

Never, and that's not true. Heikki was being nice; I wouldn't have
even
slogged through it long enough to ask the questions he did before
kicking it back as unusable. A badly formatted patch makes it
impossible to evaluate whether the changes from a submission are
reasonable or not without the reviewer fixing it first.

Then you can say that politely and firmly with direct reference to the problem, rather than making the submitter feel bad.

"Thank you for taking on testing an idea we've talked about on this list for a long time and not had the energy to test. However, I'm having a hard time evaluating your patch for a few reasons ...(give reasons). Would it be possible for you to resolve these and resubmit so that I can give the patch a good evaluation?"

... and once *one* person on this list has made such a comment, there is no need for two other hackers to pile on the reformat-your-patch bandwagon.

Our project has an earned reputation for being rejection-happy curmudgeons. This is something I heard more than once at MySQLConf, including from one student who chose to work on Drizzle instead of PostgreSQL for that reason. I think that we could stand to go out of our way to be helpful to first-time submitters.

That doesn't mean that we have to accept patches mangled by using an IDE designed for Java, and which lack test cases. However, we can be nice about it.

--
Josh Berkus
Niceness Nazi

Radoslaw,

I think 10% is quite good, as my stand-alone test of mmap vs. read
shown that
speed up of copying 100MB data to mem may be from ~20ms to ~100ms
(depends on
destination address). Of course deeper, system test simulating real
usage will
say more. In any case after good deals with writes, I will speed up
reads. I
think to bypass smgr/md much more and to expose shared id's (1,2,3...)
for
each file segment.

Well, given the risks to durability and stability associated with using MMAP, I doubt anyone would even consider it for a 10% throughput improvement. However, I don't think the test you used demonstrates the best case for MMAP as a performance improvement.

In attachment I sent test-scripts which I used to fill data, nothing
complex
(left from 2nd level caches).

Query I've used to measure was SELECT count(substr(content, 1, 1))
FROM
testcase1 WHERE multi_id > 50000;

Timings ware taken from psql.

I didn't made load (I have about 2GB of free sapce at /home, and 4GB
RAM) and
stress (I'm not quite ready to try concurrent updates of same page -
may fail,
notice is and place to fix is in code) tests yet.

Yes, but this test case doesn't offer much advantage to MMAP. Where I expect it would shine would be cases where the database is almost as big as, or much bigger than RAM ... where the extra data copying by current code is both frequent and wastes buffer space we need to use. As well as concurrent reads from the same rows.

You can write a relatively simple custom script using pgBench to test this; you don't need a big complicated benchmark. Once we get over the patch cleanup issues, I might be able to help with this.

Netbeans is quite good, of course it depends who likes what. Just try
7.0 RC
2.

I don't know if you've followed the formatting discussion, but apparently there's an issue with Netbeans re-indenting lines you didn't even edit. It makes your patch hard to read or apply. I expect that Netbeans has some method to reconfigure indenting, etc.; do you think you could configure it to PostgresQL standards so that this doesn't get in the way of evaluation of your ideas?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

Joshua Berkus wrote:

Then you can say that politely and firmly with direct reference to the problem, rather than making the submitter feel bad.

That's exactly what happened. And then you responded that it was
possible to use a patch without fixing the formatting first. That's not
true, and those of us who do patch review are tired of even trying.

Our project has an earned reputation for being rejection-happy curmudgeons. This is something I heard more than once at MySQLConf, including from one student who chose to work on Drizzle instead of PostgreSQL for that reason. I think that we could stand to go out of our way to be helpful to first-time submitters.

I'll trade you anecdotes by pointing out that I heard from half a dozen
business people that the heavy emphasis on quality control and standards
was the reason they were looking into leaving MySQL derived
distributions for PostgreSQL.

I've spent days of time working on documentation to help new submitters
get their patches improve to where they meet this community's
standards. This thread just inspired another round of that. What
doesn't help is ever telling someone they can ignore those and still do
something useful we're interested in.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Fri, Apr 15, 2011 at 3:12 PM, Radosław Smogura
<rsmogura@softperience.eu> wrote:

On Fri, 15 Apr 2011 14:33:37 +0300, Heikki Linnakangas wrote:

The patch is quite hard to read because of random whitespace changes
and other stylistic issues, but I have a couple of high-level
questions on the design:

Yes, but, hmm... in Netbeans I had really long gaps (probably 8 spaces, from
tabs), so deeper "ifs", comments at the and of variables, went of out my
screen. I really wanted to not format this, but sometimes I needed.

Seems no one else has mentioned it yet -- Postgres uses non-standard
tab-width of 4, instead of 8. If you see ugly code in Postgres, thats why...

--
marko

Tom Lane <tgl@sss.pgh.pa.us> Saturday 16 April 2011 17:02:32

Greg Stark <gsstark@mit.edu> writes:

What he did, I gather, is treat the mmapped buffers as a read-only
copy of the data. To actually make any modifications he copies it into
shared buffers and treats them like normal. When the buffers get
flushed from memory they get written and then the pointers get
repointed back at the mmapped copy.

That seems much too late --- won't other processes still be looking at
the stale mmap'ed version of the page until a write-out happens?

No, no, no :) I wanted to do this, but from above reason I skipped it. I swap
VM pages, I do remap, in place where the shared buffer was I put mmaped page,
and in place where mmaped page was I put shared page (in certain cases, which
should be optimized by e. g. read for update, for initial read of page in
process I directly points to shared buffer), it can be imagined as I affects
TLB. This what I call "VM swap" is remapping, so I don't change pointers, I
change only where this pointers points in physical memory, preserving same
pointer in Virtual Memory.

if 0x1 is start of buffer 1 (at relation 1, block 1)
I have
0x1 - 0x1 + BLCKSZ -> mmaped area
0xfffff1000 - 0xfffff1000 + BLCKSZ -> Shmem

SWAP
0x1 - 0x1 + BLCKSZ -> Shmem
0xfffff1000 - 0xfffff1000 + BLCKSZ -> mmaped area

It's reason I putted in crash reports /proc/{pid}/maps. For e. g. maps after
swap looks like (from crash report):

[...]
#Data mappings
7fe69b7e3000-7fe69b7ef000 r--s 00000000 08:03 3196408
/home/radek/src/postgresql-2nd-level-cache/db/base/12822/12516
7fe69b7ef000-7fe69b7f1000 rw-s 00148000 00:04 8880132
/SYSV0052ea91 (deleted)
7fe69b7f1000-7fe6db7e3000 r--s 0000e000 08:03 3196408
/home/radek/src/postgresql-2nd-level-cache/db/base/12822/12516
[...]
#SysV shmem mappings

7fec60788000-7fec6078c000 rw-s 00144000 00:04 8880132
/SYSV0052ea91 (deleted)
7fec6078c000-7fec6078e000 r--s 0000c000 08:03 3196408
/home/radek/src/postgresql-2nd-level-cache/db/base/12822/12516
7fec6078e000-7fec6079c000 rw-s 0014a000 00:04 8880132
/SYSV0052ea91 (deleted)
[...]

Without swap 12516 should be mapped to one VM region of size equal to
BLCKSZ*BLOCKS_PER_SEGMENT (which is about 1GB).

When process reads buffer (or after taking lock), the shared buffer descriptor
is checked if page was modified (currently is it dirty) if yes do swap, if
page is currently in use, or use directly SysV shared areas if pages is just
pinned to process.

Regards,
Radek

Radosław Smogura wrote:

Yes, but, hmm... in Netbeans I had really long gaps (probably 8
spaces, from tabs), so deeper "ifs", comments at the and of variables,
went of out my screen. I really wanted to not format this, but
sometimes I needed.

The guide at
http://www.open-source-editor.com/editors/how-to-make-netbeans-use-tabs-for-indention.html
seems to cover how to fix this in Netbeans. You want it to look like
that screen shot: 4 spaces per indent with matching tab size of 4, and
"Expand Tabs to Spaces" unchecked.

Generally, if you look at the diff you've created, and your new code
doesn't line up right with what's already there, that means the
tab/space setup isn't quite right when you were editing. Reading the
diff is useful for catching all sorts of other issues, too, so it's just
generally a good practice. As Peter already mentioned, the big problem
here is that you checked in a modified configure file.

I also note that you use C++ style "//" comments, which aren't allowed
under the coding guidelines--even though they work fine on many common
platforms.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

No, no, no :) I wanted to do this, but from above reason I skipped it. I swap
VM pages, I do remap, in place where the shared buffer was I put mmaped page,
and in place where mmaped page was I put shared page (in certain cases, which
should be optimized by e. g. read for update, for initial read of page in
process I directly points to shared buffer), it can be imagined as I affects
TLB. This what I call "VM swap" is remapping, so I don't change pointers, I
change only where this pointers points in physical memory, preserving same
pointer in Virtual Memory.

... Huh? Are you saying that you ask the kernel to map each individual
shared buffer separately? I can't believe that's going to scale to
realistic applications.

regards, tom lane

On Sat, Apr 16, 2011 at 3:19 PM, Greg Smith <greg@2ndquadrant.com> wrote:

Joshua Berkus wrote:

Then you can say that politely and firmly with direct reference to the
problem, rather than making the submitter feel bad.

That's exactly what happened.  And then you responded that it was possible
to use a patch without fixing the formatting first.  That's not true, and
those of us who do patch review are tired of even trying.

It would be worth a lot if we could get it enough easier to use
pgindent, so that that could help *anyone* fix the formatting, as
opposed to being something that Bruce runs once in a long while.

If you can say, "here, run 'tools/frobozz/pg_indent' against each of
your files, then resubmit the patch," and have at least a fighting
chance of that being *nearly* right, that is a much nicer response to
give those folks.

Alternately, it would be nice if you could say, "I ran pgindent
against your files, here's the revised patch, please do that yourself
in future"

When application of formatting policy is near-nondeterministic, that's no fun!
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On Sat, Apr 16, 2011 at 9:12 PM, Christopher Browne <cbbrowne@gmail.com> wrote:

On Sat, Apr 16, 2011 at 3:19 PM, Greg Smith <greg@2ndquadrant.com> wrote:

Joshua Berkus wrote:

Then you can say that politely and firmly with direct reference to the
problem, rather than making the submitter feel bad.

That's exactly what happened.  And then you responded that it was possible
to use a patch without fixing the formatting first.  That's not true, and
those of us who do patch review are tired of even trying.

It would be worth a lot if we could get it enough easier to use
pgindent, so that that could help *anyone* fix the formatting, as
opposed to being something that Bruce runs once in a long while.

If you can say, "here, run 'tools/frobozz/pg_indent' against each of
your files, then resubmit the patch," and have at least a fighting
chance of that being *nearly* right, that is a much nicer response to
give those folks.

Alternately, it would be nice if you could say, "I ran pgindent
against your files, here's the revised patch, please do that yourself
in future"

I agree.

But it turns out that it doesn't really matter. Whitespace or no
whitespace, if you don't read the diff before you hit send, it's
likely to contain some irrelevant cruft, whether whitespace changes or
otherwise.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 04/16/2011 09:12 PM, Christopher Browne wrote:

On Sat, Apr 16, 2011 at 3:19 PM, Greg Smith<greg@2ndquadrant.com> wrote:

Joshua Berkus wrote:

Then you can say that politely and firmly with direct reference to the
problem, rather than making the submitter feel bad.

That's exactly what happened. And then you responded that it was possible
to use a patch without fixing the formatting first. That's not true, and
those of us who do patch review are tired of even trying.

It would be worth a lot if we could get it enough easier to use
pgindent, so that that could help *anyone* fix the formatting, as
opposed to being something that Bruce runs once in a long while.

If you can say, "here, run 'tools/frobozz/pg_indent' against each of
your files, then resubmit the patch," and have at least a fighting
chance of that being *nearly* right, that is a much nicer response to
give those folks.

Alternately, it would be nice if you could say, "I ran pgindent
against your files, here's the revised patch, please do that yourself
in future"

When application of formatting policy is near-nondeterministic, that's no fun!

What makes you think this isn't possible to run pgindent? There are no
secret incantations.

But it's probably overkill. emacs' indent-region gives you about a 90%
result or better if you're set up correctly.

cheers

andrew

On Sat, Apr 16, 2011 at 2:33 PM, Joshua Berkus <josh@agliodbs.com> wrote:

Well, given the risks to durability and stability associated with using MMAP, I doubt anyone would even consider it for a 10% throughput improvement.  However, I don't think the test you used demonstrates the best case for MMAP as a performance improvement.

Actually, I'd walk through fire for a 10% performance improvement if
it meant only a *risk* to stability. The problem is that this is
likely unfixably broken. In particular, I think the first sentence of
Tom's response hit it right on the nose, and mirrors my own thoughts
on the subject. To have any chance of working, you'd need to track
buffer pins and shared/exclusive content locks for the pages that were
being accessed outside of shared buffers; otherwise someone might be
looking at a stale copy of the page.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Apr 16, 2011 at 9:31 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Sat, Apr 16, 2011 at 2:33 PM, Joshua Berkus <josh@agliodbs.com> wrote:

Well, given the risks to durability and stability associated with using MMAP, I doubt anyone would even consider it for a 10% throughput improvement.  However, I don't think the test you used demonstrates the best case for MMAP as a performance improvement.

Actually, I'd walk through fire for a 10% performance improvement if
it meant only a *risk* to stability.  The problem is that this is
likely unfixably broken.  In particular, I think the first sentence of
Tom's response hit it right on the nose, and mirrors my own thoughts
on the subject.  To have any chance of working, you'd need to track
buffer pins and shared/exclusive content locks for the pages that were
being accessed outside of shared buffers; otherwise someone might be
looking at a stale copy of the page.

Of course, maybe the patch is doing that. Rereading the thread, I
grow increasingly confused about what this is actually supposed to do
and how it's supposed to work and why it's supposedly better than what
we do now. But please, everyone feel free to continue bashing me for
wanting a readable patch with some understandable submission notes.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Andrew Dunstan wrote:

What makes you think this isn't possible to run pgindent?

I have to say, I've been rather mystified by the difficulty
attributed to running pgindent. During work on the SSI patch, I ran
it about once every two weeks on files involved in the patch, just so
that it would be easier to review by people used to that format. I
also tried to keep src/tools/pgindent/typedefs.list up to date with
new structures, so that my runs were good. Granted, when the
official run was done there were a few adjustments to typedefs.list,
and some comments which were added after the commit of the main part
of the patch hadn't yet been wrapped to the right line length, but on
the whole I didn't find it a big deal to stay relatively close by
doing periodic runs. Maybe three minutes every two weeks.

When people talk like it's hugely difficult or hard to understand, I
wonder if they have actually made the attempt. When someone is eager
for feedback on a patch, it doesn't seem unreasonable to me to ask
them to read the README for pgindent and try to generate a patch with
conforming results.

Now, the other aspect to this whole discussion is that people often
have code they have developed for academic purposes or for their own
use which they want to offer to the community "FWIW", and I think we
sometimes miss an opportunity to take advantage of someone else's
work because of an assumption that they have some vested interest in
it's acceptance. The fact that someone doesn't care enough to try to
work with the community to get their patch accepted doesn't *always*
mean that we're better off for ignoring that patch. Maybe that's
true 90% of the time or better, but it seems to me that sometimes our
community is a bit provincial.

And I can't help but wonder why, in an off-list discussion with
Michael Cahill about the SSI technology he commented that he was
originally intending to implement the technique in PostgreSQL, but
later chose Oracle Berkeley DB and then latter InnoDB instead.
*Maybe* he was looking toward being hired by Oracle, and *maybe* it
was because the other databases already had predicate locking and
true serializable transaction isolation levels -- but was part of it
the reputation of the community? I keep wondering.

-Kevin

On 4/16/11, Andrew Dunstan <andrew@dunslane.net> wrote:

What makes you think this isn't possible to run pgindent? There are no
secret incantations.

A while ago I spent a few hours trying to run it and gave up. I think
it was something about needing some obscure BSD version of some tool
which conflicted with just about everything else on the system. I can
try again and report back if anyone cares.

On Apr 17, 2011 8:17 AM, "Jeff Janes" <jeff.janes@gmail.com> wrote:

On 4/16/11, Andrew Dunstan <andrew@dunslane.net> wrote:

What makes you think this isn't possible to run pgindent? There are no
secret incantations.

A while ago I spent a few hours trying to run it and gave up. I think
it was something about needing some obscure BSD version of some tool
which conflicted with just about everything else on the system. I can
try again and report back if anyone cares.

It does rely on BSD indent. For that very reason, we provide the source for
it, since most people have gnu indent. It's trivial to build, though, and
works just fine as a local build, and you can keep gnu indent as the main
one on your system - no conflicts.

It used to be a PITA due to the typedef list, but that has been fixed.
Perhaps we just need to document it a bit more...

/Magnus

Andrew Dunstan wrote:

What makes you think this isn't possible to run pgindent? There are no
secret incantations.

The first hit newbies find looking for info about pgident is
http://blog.hagander.net/archives/185-pgindent-vs-dash.html which sure
looks like secret incantations to me. The documentation
src/tools/pgindent/README reads like a magic spell too:

find . -name '*.[ch]' -type f -print | \
egrep -v -f src/tools/pgindent/exclude_file_patterns | \
xargs -n100 pgindent src/tools/pgindent/typedefs.list

And it doesn't actually work as written unless you've installed
pgindent, entab/detab, and the specially patched NetBSD indent into the
system PATH somewhere--unreasonable given that this may be executing on
a source only tree that has never been installed.. The fact that the
documention is only in the README and not with the rest of the code
conventions isn't helping either.

The last time I tried to do this a few years ago I failed miserably and
never came back. I know way more about building software now though,
and just got this to work for the first time. Attached is a WIP wrapper
script for running pgident that builds all the requirements into
temporary directories, rather than expecting you to install anything
system-wide or into a PostgreSQL destination directory. Drop this into
src/tools/pgindent, make it executable, and run it from that directory.
Should do the right thing on any system that has "make" as an alias for
"gmake" (TODO to be better about that in the file, with some other
nagging things).

When I just ran it against master I got a bunch of modified files, but
most of them look like things that have been touched recently so I think
it did the right thing. A test of my work here from someone who isn't
running this for the first time would be helpful. If this works well
enough, I think it would make a good helper script to include in the
distribution. The loose ends to fix I can take care of easily enough
once basic validation is finished.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 01:35:45

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

No, no, no :) I wanted to do this, but from above reason I skipped it. I
swap VM pages, I do remap, in place where the shared buffer was I put
mmaped page, and in place where mmaped page was I put shared page (in
certain cases, which should be optimized by e. g. read for update, for
initial read of page in process I directly points to shared buffer), it
can be imagined as I affects TLB. This what I call "VM swap" is
remapping, so I don't change pointers, I change only where this pointers
points in physical memory, preserving same pointer in Virtual Memory.

... Huh? Are you saying that you ask the kernel to map each individual
shared buffer separately? I can't believe that's going to scale to
realistic applications.

regards, tom lane

No, I do
mrempa(mmap_buff_A, MAP_FIXED, temp);
mremap(shared_buff_Y, MAP_FIXED, mmap_buff_A),
mrempa(tmp, MAP_FIXED, mmap_buff_A).

This is this additional overhead - and may have some disadvantages. All
regions SysV / Posix MMAP are mapped before.

I couldn't believe too, but as I done some work about read, I was in dead
corner:
1. Create Read Before Buffer (connect with XLOG) that will store each page
before modification (page should be flushed and synced to log)
2. Rewrite whole db to repoint pointers or similar stuff (I done few steps for
this).
3. Or find something different.

I couldn't believe too, it's way I still work on it. I saw it gains speed for
few simple updates. I'm not quite sure why it gets it. I only may think it
was from "pre update reads". But full checks will go after some good point of
updates.

Regards,
Radek

Robert Haas wrote:

But it turns out that it doesn't really matter. Whitespace or no
whitespace, if you don't read the diff before you hit send, it's
likely to contain some irrelevant cruft, whether whitespace changes or
otherwise.

Right. Presuming that pgident will actually solve anything leaps over
two normally incorrect assumptions:

-That the main tree was already formatted with pgident before you
started, so no stray diffs will result from it touching things the
submitter isn't even involved in.

-There is no larger code formatting or diff issues except for spacing.

This has been a nagging loose end for a while, so I'd like to see
pgindent's rough edges get sorted out so it's easier to use. But
whitespace errors because of bad editors are normally just a likely sign
of a patch with bigger problems, rather than something that can get
fixed and then submissions is good. There is no substitute for the
discipline of reading your own diff before submission. I'll easily
obsess over mine for an hour before I submit something major, and that
time is always well spent.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On 04/17/2011 02:16 AM, Jeff Janes wrote:

On 4/16/11, Andrew Dunstan<andrew@dunslane.net> wrote:

What makes you think this isn't possible to run pgindent? There are no
secret incantations.

A while ago I spent a few hours trying to run it and gave up. I think
it was something about needing some obscure BSD version of some tool
which conflicted with just about everything else on the system. I can
try again and report back if anyone cares.

A few hours? Seriously?

Here's what I just did, starting from scratch. It took me a few minutes.

* wget ftp://ftp.postgresql.org/pub/dev/indent.netbsd.patched.tgz
* mkdir bsdindent && cd bdsindent && tar -z -xf
.../indent.netbsd.patched.tgz
* make
* mv indent indent_for_pg
* sudo install -s -o bin -g bin indent_for_pg /usr/local/bin
* cd ../pg_head/src/tools/entab
* sudo make install
* cd ../pgindent
* sed -i 's/INDENT=indent/INDENT=indent_for_pg/' pgindent
* sudo install -s -o bin -g bin pgindent /usr/local/bin

Now we could certainly make this quite a bit slicker. Apart from
anything else, we should change the indent source code tarball so it
unpacks into its own directory. Having it unpack into the current
directory is ugly and unfriendly. And we should get rid of the "make
clean" line in the install target of entab's makefile, which just seems
totally ill-conceived.

It might also be worth setting it up so that instead of having to pass a
path to a typedefs file on the command line, we default to a file
sitting in, say, /usr/local/etc. Then you'd just be able to say
"pgindent my_file.c".

But it shouldn't take anyone hours to set up.

cheers

andrew

On 04/17/2011 04:08 AM, Greg Smith wrote:

Andrew Dunstan wrote:

What makes you think this isn't possible to run pgindent? There are
no secret incantations.

The first hit newbies find looking for info about pgident is
http://blog.hagander.net/archives/185-pgindent-vs-dash.html which sure
looks like secret incantations to me. The documentation
src/tools/pgindent/README reads like a magic spell too:

find . -name '*.[ch]' -type f -print | \
egrep -v -f src/tools/pgindent/exclude_file_patterns | \
xargs -n100 pgindent src/tools/pgindent/typedefs.list

That's the incantation for indenting the whole of the source code. But
very few people want to do that. Most people just want to indent a
single file, for which the incantation is "pgindent path_to_typedefs
my_file.c". See in another message my suggestion for defaulting the
typedefs arg, so you'd just be able to say "pg_indent my_file.c".

cheers

andrew

On 04/17/2011 04:26 AM, Andrew Dunstan wrote:

Now we could certainly make this quite a bit slicker. Apart from
anything else, we should change the indent source code tarball so it
unpacks into its own directory. Having it unpack into the current
directory is ugly and unfriendly. And we should get rid of the "make
clean" line in the install target of entab's makefile, which just
seems totally ill-conceived.

It might also be worth setting it up so that instead of having to pass
a path to a typedefs file on the command line, we default to a file
sitting in, say, /usr/local/etc. Then you'd just be able to say
"pgindent my_file.c".

OK, I have most of these bits.

A new tarball of indent is available at
<http://developer.postgresql.org/~andrew/indent.netbsd.patched.tgz&gt; and
if everyone agrees I'll push it out to the mirrors.

Attached are two patches, one to remove some infelicity in the entab
makefile, and the other to allow skipping specifying the typedefs file
location either by setting it in an environment variable or by putting
it in a hard coded location.

cheers

andrew

On 04/17/2011 09:51 AM, Andrew Dunstan wrote:

On 04/17/2011 04:26 AM, Andrew Dunstan wrote:

Now we could certainly make this quite a bit slicker. Apart from
anything else, we should change the indent source code tarball so it
unpacks into its own directory. Having it unpack into the current
directory is ugly and unfriendly. And we should get rid of the "make
clean" line in the install target of entab's makefile, which just
seems totally ill-conceived.

It might also be worth setting it up so that instead of having to
pass a path to a typedefs file on the command line, we default to a
file sitting in, say, /usr/local/etc. Then you'd just be able to say
"pgindent my_file.c".

OK, I have most of these bits.

A new tarball of indent is available at
<http://developer.postgresql.org/~andrew/indent.netbsd.patched.tgz&gt;
and if everyone agrees I'll push it out to the mirrors.

Attached are two patches, one to remove some infelicity in the entab
makefile, and the other to allow skipping specifying the typedefs file
location either by setting it in an environment variable or by putting
it in a hard coded location.

... and this one has a typo fixed.

cheers

andrew

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 01:35:45

... Huh? Are you saying that you ask the kernel to map each individual
shared buffer separately? I can't believe that's going to scale to
realistic applications.

No, I do
mrempa(mmap_buff_A, MAP_FIXED, temp);
mremap(shared_buff_Y, MAP_FIXED, mmap_buff_A),
mrempa(tmp, MAP_FIXED, mmap_buff_A).

There's no mremap() in the Single Unix Spec, nor on my ancient HPUX box,
nor on my quite-up-to-date OS X box. The Linux man page for it says
"This call is Linux-specific, and should not be used in programs
intended to be portable." So if the patch is dependent on that call,
it's dead on arrival from a portability standpoint.

But in any case, you didn't explain how use of mremap() avoids the
problem of the kernel having to maintain a separate page-mapping-table
entry for each individual buffer. (Per process, yet.) If that's what's
happening, it's going to be a significant performance penalty as well as
(I suspect) a serious constraint on how many buffers can be managed.

regards, tom lane

Robert Haas <robertmhaas@gmail.com> writes:

... But please, everyone feel free to continue bashing me for
wanting a readable patch with some understandable submission notes.

What he said. All this obsessing over whether the mmap patch could or
should have been run through pgindent is missing the big picture.
Namely, that no design documentation or theory-of-operation was offered,
and people are trying to extract that information from the code, and
it's just too messy for that to be feasible. (The patch isn't really
short of comments, but half of the comments seem to be TODOs or author's
questions to himself about whether something will work, and so they just
aren't particularly helpful to someone trying to understand what the
patch does or whether it will work.)

I think that rather than complaining about formatting, we should be
complaining about not following the overall patch submission process
and not providing adequate documentation. Most of the questions that
people are asking right now could have been answered on the strength of
a design sketch, before any code had been written at all. For a patch
as complicated and invasive as this, there should be a design sketch,
which perhaps gets fleshed out into a README file in the final patch.

The Submitting_a_Patch wiki page does touch on the point of getting some
early design feedback before you even try to write a patch, but I think
it could do with more emphasis on the issue.

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 17:48:56

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 01:35:45

... Huh? Are you saying that you ask the kernel to map each individual
shared buffer separately? I can't believe that's going to scale to
realistic applications.

No, I do
mrempa(mmap_buff_A, MAP_FIXED, temp);
mremap(shared_buff_Y, MAP_FIXED, mmap_buff_A),
mrempa(tmp, MAP_FIXED, mmap_buff_A).

There's no mremap() in the Single Unix Spec, nor on my ancient HPUX box,
nor on my quite-up-to-date OS X box. The Linux man page for it says
"This call is Linux-specific, and should not be used in programs
intended to be portable." So if the patch is dependent on that call,
it's dead on arrival from a portability standpoint.

Good point. This is from initial concept, and actually I done this to do not
leave gaps in VM in which library or something could be mmaped. Last time I
think about using mmap to replace just one VM page.

But in any case, you didn't explain how use of mremap() avoids the
problem of the kernel having to maintain a separate page-mapping-table
entry for each individual buffer. (Per process, yet.) If that's what's
happening, it's going to be a significant performance penalty as well as
(I suspect) a serious constraint on how many buffers can be managed.

regards, tom lane

Kernel merges vm_structs. So mappings are compacted. I'm not kernel
specialist, but skipping memory consumption, for not compacted mappings,
kernel uses btrees for dealing with TLB, so it should not matter if there is
100 vm_structs or 100000 vm_structs.

Swap isn't made everywhere. When buffer is initialy read (privaterefcount
==1), then any access to this buffer will directly point to latest valid area.
If it has assigned shmem area then this will be used. I plan to add
"readbuffer for update" to prevent swaps, when it's almost sure that buffer
will be used for update.

I measured performance of page modifications (with unpining, full process on
stand alone unit test) it's 2x-3x more time of normal page reads, but this
result may not be sure, as I saw memcpy to memory above 2GB is slower then
memcpy to first 2GB (this may be idea to try to put some shared structs <
2GB).

I know that this patch is big question. Sometimes I'm optimistic, and
sometimes I'm pessimistic about final result.

Regards,
Radek

On Sunday 17 April 2011 19:26:31 Radosław Smogura wrote:

Kernel merges vm_structs. So mappings are compacted. I'm not kernel
specialist, but skipping memory consumption, for not compacted mappings,
kernel uses btrees for dealing with TLB, so it should not matter if there
is 100 vm_structs or 100000 vm_structs.

But the CPUs TLB cache has maybe 16/256 (1lvl, 2nd) to 64/512 entries. That
will mean that there will be cachemisses all over.
Additionally your scheme requires flushing it regularly...

Andres

Robert,

Actually, I'd walk through fire for a 10% performance improvement if
it meant only a *risk* to stability.

Depends on the degree of risk. MMAP has the potential to introduce instability into areas of the code which have been completely reliable for years. Adding 20 new coredump cases with data loss for a 10% improvement seems like a poor bargain to me. It doesn't help that the only DB to rely heavily on MMAP (MongoDB) is OSSDB's paragon of data loss.

However, in the case where the database is larger than RAM ... or better, 90% of RAM ... MMAP has the theoretical potential to improve performance quite a bit more than 10% ... try up to 900% on some queries. However, I'd like to prove that in a test before we bother even debating the fundamental obstacles to using MMAP. It's possible that these theoretical performance benefits will not materialize, even without data safeguards.

The problem is that this is
likely unfixably broken. In particular, I think the first sentence of
Tom's response hit it right on the nose, and mirrors my own thoughts
on the subject. To have any chance of working, you'd need to track
buffer pins and shared/exclusive content locks for the pages that were
being accessed outside of shared buffers; otherwise someone might be
looking at a stale copy of the page.

Nothing is unfixable. The question is whether it's worth the cost. Let me see if I can build a tree with Radislaw's patch, and do some real performance tests.

I, for one, am glad he did this work. We've discussed MMAP in the code off and on for years, but nobody wanted to do the work to test it. Now someone has, and we can decide whether it's worth pursuing based on the numbers.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

Joshua Berkus <josh@agliodbs.com> writes:

I, for one, am glad he did this work. We've discussed MMAP in the code off and on for years, but nobody wanted to do the work to test it. Now someone has, and we can decide whether it's worth pursuing based on the numbers.

Well, the troubling issue is that it's not clear whether this patch is
realistic enough to think that performance measurements based on it
are representative of the whole idea of using mmap. The business of
remapping individual buffers in order to transition them to writable
state seems likely to me to be a huge performance penalty --- first
there's the direct cost of having to incur a kernel call each time we
do that, and second there's the distributed cost of asking the kernel
to manage thousands or millions of tiny mappings.

IOW, if this patch shows little or no performance improvement (as seems
likely to happen at scale), that doesn't prove that mmap in general
isn't potentially interesting, only that this isn't the right way to
approach it.

Still, if you do some tests and don't find a win, that might save time
compared to actually trying to understand and vet the patch ...

regards, tom lane

On Sun, Apr 17, 2011 at 11:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 01:35:45

... Huh?  Are you saying that you ask the kernel to map each individual
shared buffer separately?  I can't believe that's going to scale to
realistic applications.

No, I do
mrempa(mmap_buff_A, MAP_FIXED, temp);
mremap(shared_buff_Y, MAP_FIXED, mmap_buff_A),
mrempa(tmp, MAP_FIXED, mmap_buff_A).

There's no mremap() in the Single Unix Spec, nor on my ancient HPUX box,
nor on my quite-up-to-date OS X box.  The Linux man page for it says
"This call is Linux-specific, and should not be used in programs
intended to be portable."  So if the patch is dependent on that call,
it's dead on arrival from a portability standpoint.

But in any case, you didn't explain how use of mremap() avoids the
problem of the kernel having to maintain a separate page-mapping-table
entry for each individual buffer.  (Per process, yet.)  If that's what's
happening, it's going to be a significant performance penalty as well as
(I suspect) a serious constraint on how many buffers can be managed.

I share your suspicions, although no harm in measuring it.

But I don't understand is how this approach avoids the problem of
different processes seeing different buffer contents. If backend A
has the buffer mmap'd and backend B wants to modify it (and changes
the mapping), backend A is still looking at the old buffer contents,
isn't it? And then things go boom.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Andres Freund <andres@anarazel.de> Sunday 17 April 2011 20:02:11

On Sunday 17 April 2011 19:26:31 Radosław Smogura wrote:

Kernel merges vm_structs. So mappings are compacted. I'm not kernel
specialist, but skipping memory consumption, for not compacted mappings,
kernel uses btrees for dealing with TLB, so it should not matter if
there is 100 vm_structs or 100000 vm_structs.

But the CPUs TLB cache has maybe 16/256 (1lvl, 2nd) to 64/512 entries. That
will mean that there will be cachemisses all over.
Additionally your scheme requires flushing it regularly...

Andres

I only know Phenom has 4096 entries I think and this covers 16MB of memory.
But I was taking about memory usage of struct vm_struct in kernel. I tries as
well with huge pages, but I can't write really fast allocator for this, it's
slower then malloc, maybe from different reasons.

Regards,
Radek

On Sunday 17 April 2011 22:09:24 Radosław Smogura wrote:

I only know Phenom has 4096 entries I think and this covers 16MB of
memory.

The numbers I cited where intels before and after core2.

Andres

On Sun, Apr 17, 2011 at 12:26 AM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:

Now, the other aspect to this whole discussion is that people often
have code they have developed for academic purposes or for their own
use which they want to offer to the community "FWIW", and I think we
sometimes miss an opportunity to take advantage of someone else's
work because of an assumption that they have some vested interest in
it's acceptance.  The fact that someone doesn't care enough to try to
work with the community to get their patch accepted doesn't *always*
mean that we're better off for ignoring that patch.  Maybe that's
true 90% of the time or better, but it seems to me that sometimes our
community is a bit provincial.

We are.

On the other hand, cleaning up other people's not-ready-for-prime-time
patches isn't free. If I spend 4 hours cleaning up a patch in
preparation for a commit, then that's 4 hours I don't get to spend on
my own work. And since I *already* spend 3 or 4 times as much energy
on other people's work as I do on my own, I'm not willing to go much
further in that direction; if anything, I think I'd like to roll it
back a bit. On the other hand, I am emphatically in favor of other
people who are not me being willing to do that kind of work; I think
it benefits our whole community, much as the work of people who write
their own patches or review or volunteer in any other way benefits our
whole community.

Because I commit approximately 10 patches per CommitFest, and review
perhaps another 5-10 that I don't end up committing (either because
they get rejected or because someone else commits them), the amount of
time that I can afford to spend on each of those patches is limited.
Generally, if I can't commit a normal-size patch in half an hour of
looking at it, I send back a review and move on. For some patches
that I particularly care about, I have on occasion invested as much as
2-3 days (most recently, a big chunk of my Christmas vacation) to get
them beaten into shape for a commit. I'd be happy to devote more time
per patch, but it ain't gonna happen as long as the number that I have
to handle to get the CommitFest finished on time remains in the
two-digit range.

That having been said, the kind of fixing up that you're talking about
*does* happen, when someone cares enough to make it happen. We have
numerous examples in the archives where person A submits a patch, and
person B reviews it and, in lieu of a review, posts an updated patch,
sometimes when person A has meanwhile totally disappeared, or when
they haven't completely disappeared but don't have time to work on it.
This is actually quite commonplace; it just doesn't happen for every
patch. It tends to happen only for the things someone is really
excited about because, well, fixing up someone else's bad code is not
one of life's great pleasures. It'd be nice if we had even more of it
than we do, but this is an all-volunteer organization.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> Sunday 17 April 2011 22:01:55

On Sun, Apr 17, 2011 at 11:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

=?utf-8?q?Rados=C5=82aw_Smogura?= <rsmogura@softperience.eu> writes:

Tom Lane <tgl@sss.pgh.pa.us> Sunday 17 April 2011 01:35:45

... Huh? Are you saying that you ask the kernel to map each individual
shared buffer separately? I can't believe that's going to scale to
realistic applications.

No, I do
mrempa(mmap_buff_A, MAP_FIXED, temp);
mremap(shared_buff_Y, MAP_FIXED, mmap_buff_A),
mrempa(tmp, MAP_FIXED, mmap_buff_A).

There's no mremap() in the Single Unix Spec, nor on my ancient HPUX box,
nor on my quite-up-to-date OS X box. The Linux man page for it says
"This call is Linux-specific, and should not be used in programs
intended to be portable." So if the patch is dependent on that call,
it's dead on arrival from a portability standpoint.

But in any case, you didn't explain how use of mremap() avoids the
problem of the kernel having to maintain a separate page-mapping-table
entry for each individual buffer. (Per process, yet.) If that's what's
happening, it's going to be a significant performance penalty as well as
(I suspect) a serious constraint on how many buffers can be managed.

I share your suspicions, although no harm in measuring it.

But I don't understand is how this approach avoids the problem of
different processes seeing different buffer contents. If backend A
has the buffer mmap'd and backend B wants to modify it (and changes
the mapping), backend A is still looking at the old buffer contents,
isn't it? And then things go boom.

Each process has simple "mirror" of shared descriptors.

I "believe" that modifications to buffer content may be only done when holding
exclusive lock (with some simple exceptions) (+ MVCC), actually I saw only two
things that can change already loaded data and cause damage, you have
described (setting hint bits during scan, and vacuum - 1st may only cause, I
think, that two processes will ask for same transaction statuses <except
vacuum>, 2nd one is impossible as vacumm requires exclusive pin). When buffer
tag is changed the version of buffer is bumped up, and checked against local
version - this about reading buffer.

In other cases after obtaining lock check is done if buffer has associated
updatable buffer and if local "mirror" has it too, then swap should take
place.

Logic about updatable buffers is similar to "shared buffers", each updatable
buffer has pin count, and updatable buffer can't be free if someone uses it,
but in contrast to "normal buffers", updatable buffers doesn't have any
support for locking etc. Updatable buffers exists only on free list, or when
associated with buffer.

In future, I will change version to shared segment id, something like
relation's oid + block, but ids will have continuous numbering 1,2,3..., so I
will be able to bypass smgr/md during read, and tag version check - this looks
like faster solution.

Regards,
Radek

On Sat, Apr 16, 2011 at 11:26:34PM -0500, Kevin Grittner wrote:

I have to say, I've been rather mystified by the difficulty
attributed to running pgindent. During work on the SSI patch, I ran
it about once every two weeks on files involved in the patch

Well, as a counterpoint: during work on the SSI patch, I did *not* run
pgindent. I attempted to, at one point, but was discouraged when I
realized that it required BSD indent and my Linux machine only had GNU
indent. That meant I would need to find, build, and install a new
version of indent, and keep it separate from my existing GNU indent.
Hardly impossible, but it's a lot more of a hassle than simply running a
script, and it left me wondering if I was going to run into other
issues even if I did get the right indent installed.

Andrew's instructions upthread would certainly have been helpful to
have in the pgindent README.

(To be fair, I would probably have made much more of an effort to run
pgindent if I didn't already know Kevin was running it periodically on
the SSI code.)

And I can't help but wonder why, in an off-list discussion with
Michael Cahill about the SSI technology he commented that he was
originally intending to implement the technique in PostgreSQL, but
later chose Oracle Berkeley DB and then latter InnoDB instead.
*Maybe* he was looking toward being hired by Oracle, and *maybe* it
was because the other databases already had predicate locking and
true serializable transaction isolation levels -- but was part of it
the reputation of the community? I keep wondering.

I would discount the first explanation (being hired at Oracle)
entirely. I think the second explanation is the correct one: it's
simply much more difficult to implement SSI atop a database that does
not already have predicate locking (as we know!)

But I am aware of other cases in which people in the academic community
have done work that could well be of interest to the Postgres community
but didn't submit their work here. In part, that was because they did
not have the time/motivation to get the work into a polished,
acceptable state, and in part because of the reputation of the
community.

Dan

--
Dan R. K. Ports MIT CSAIL http://drkp.net/

Dan Ports <drkp@csail.mit.edu> writes:

... But I am aware of other cases in which people in the academic community
have done work that could well be of interest to the Postgres community
but didn't submit their work here. In part, that was because they did
not have the time/motivation to get the work into a polished,
acceptable state, and in part because of the reputation of the
community.

Well, if the author isn't interested in getting the work into a
committable state, it's not clear what's the point of submitting it.
It's not like people who are eager to do that kind of work on someone
else's patch are thick on the ground.

But I think the perception that we reject most patches is misplaced.
It's fairly easy to demonstrate that the default assumption around here
is that submitted patches will get committed. Looking at the past five
commitfests (covering a bit more than a year), we committed 201 out of
305 patches, and only 10 were actually marked "rejected". I'm too lazy
to try to determine just which of the 94 returned-with-feedback patches
got committed in later fests, but a quick scan suggests at least 20 did,
and there are more that might get committed in the next fest. That puts
the overall patch acceptance rate at perhaps 75%. At least since the CF
mechanism was instituted, it seems to me that the dynamic has been that
someone who doesn't like a patch has to show cause why it shouldn't get
committed, not the other way around. Robert's recent comment that he
was afraid he'd have to spend time digging into the mmap patch to prove
it was broken reflects exactly that feeling.

regards, tom lane

On Sun, Apr 17, 2011 at 7:13 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Dan Ports <drkp@csail.mit.edu> writes:

... But I am aware of other cases in which people in the academic community
have done work that could well be of interest to the Postgres community
but didn't submit their work here. In part, that was because they did
not have the time/motivation to get the work into a polished,
acceptable state, and in part because of the reputation of the
community.

Well, if the author isn't interested in getting the work into a
committable state, it's not clear what's the point of submitting it.
It's not like people who are eager to do that kind of work on someone
else's patch are thick on the ground.

But I think the perception that we reject most patches is misplaced.
It's fairly easy to demonstrate that the default assumption around here
is that submitted patches will get committed.  Looking at the past five
commitfests (covering a bit more than a year), we committed 201 out of
305 patches, and only 10 were actually marked "rejected".  I'm too lazy
to try to determine just which of the 94 returned-with-feedback patches
got committed in later fests, but a quick scan suggests at least 20 did,
and there are more that might get committed in the next fest.  That puts
the overall patch acceptance rate at perhaps 75%.

That someone overstates the acceptance rate, because it ignores the
patches that people post and immediately get flamed to a well-done
crisp before adding them to the CF app, but there are not very many of
those any more. (If someone thinks I'm wrong about this, they are
cheerfully invited to provide the evidence. It is certainly possible
that I'm guilty of selective memory; this is just how I remember it.)

At least since the CF
mechanism was instituted, it seems to me that the dynamic has been that
someone who doesn't like a patch has to show cause why it shouldn't get
committed, not the other way around. Robert's recent comment that he
was afraid he'd have to spend time digging into the mmap patch to prove
it was broken reflects exactly that feeling.

Yes, and I think it's also telling that the response to that was not
"oh, gee, if Robert thinks this patch is totally busted, we'd better
take that concern seriously" but rather "stop picking on the guy who
submitted the patch". Maybe someone out there is under the impression
that I get high off of rejecting patches; but the statistics you cite
from the CF app don't exactly support the contention that I'm going
around looking for reasons to reject things, or if I am, I'm doing a
pretty terrible job finding them.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 04/17/2011 07:41 PM, Robert Haas wrote:

That puts the overall patch acceptance rate at perhaps 75%.

That someone overstates the acceptance rate, because it ignores the
patches that people post and immediately get flamed to a well-done
crisp before adding them to the CF app, but there are not very many of
those any more.

I don't believe there were ever terribly many of them.

cheers

andrew

On Sun, Apr 17, 2011 at 5:32 PM, Radosław Smogura
<rsmogura@softperience.eu> wrote:

Each process has simple "mirror" of shared descriptors.

I "believe" that modifications to buffer content may be only done when holding
exclusive lock (with some simple exceptions) (+ MVCC), actually I saw only two
things that can change already loaded data and cause damage, you have
described (setting hint bits during scan, and vacuum - 1st may only cause, I
think, that two processes will ask for same transaction statuses <except
vacuum>, 2nd one is impossible as vacumm requires exclusive pin). When buffer
tag is changed the version of buffer is bumped up, and checked against local
version - this about reading buffer.

Yes, an exclusive lock is required for substantive content changes.
But if vacuum cleaning up the buffer is an issue for your patch, then
it's probably also a problem if someone grabs an exclusive content
lock and deletes the tuple (by setting XMAX) and some other backend
later sees the old buffer contents after having in the meanwhile taken
a new snapshot; or if likewise someone grabs an exclusive-lock, adds a
tuple, and then your backend takes a new snapshot and then sees the
old buffer contents. Basically, any time someone grabs an
exclusive-lock and releases it, it's necessary for all observers to
see the updated contents by the time the exclusive lock is released.

In other cases after obtaining lock check is done if buffer has associated
updatable buffer and if local "mirror" has it too, then swap should take
place.

I think this check would have to be done every time someone
share-locks the buffer, which seems rather expensive.

Logic about updatable buffers is similar to "shared buffers", each updatable
buffer has pin count, and updatable buffer can't be free if someone uses it,
but in contrast to "normal buffers", updatable buffers doesn't have any
support for locking etc. Updatable buffers exists only on free list, or when
associated with buffer.

I don't see how you're going to get away with removing buffer locks.
They exist for a reason, and adding mmap() to the mix is going to
require MORE locking, not less.

In future, I will change version to shared segment id, something like
relation's oid + block, but ids will have continuous numbering 1,2,3..., so I
will be able to bypass smgr/md during read, and tag version check - this looks
like faster solution.

I don't understand this part at all.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Andrew Dunstan wrote:

Now we could certainly make this quite a bit slicker. Apart from
anything else, we should change the indent source code tarball so it
unpacks into its own directory. Having it unpack into the current
directory is ugly and unfriendly. And we should get rid of the "make
clean" line in the install target of entab's makefile, which just
seems totally ill-conceived.

I think the script I submitted upthread has most of the additional
slickness needed here. Looks like we both were working on documenting a
reasonable way to do this at the same time the other day. The idea of
any program here relying on being able to write to /usr/local/bin as
your example did makes this harder for people to run; that's why I made
everything in the build tree and just pushed the appropriate directories
into the PATH.

Since I see providing a script to automate this whole thing as the
preferred way to make this easier, re-packaging the indent source
tarball to extract to a directory doesn't seem worth the backwards
compatibility trouble it will introduce. Improving the entab makefile I
don't have an opinion on.

It might also be worth setting it up so that instead of having to pass
a path to a typedefs file on the command line, we default to a file
sitting in, say, /usr/local/etc. Then you'd just be able to say
"pgindent my_file.c".

OK, so I need to update my script to handle either indenting a single
file, or doing all of them.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

Andrew Dunstan <andrew@dunslane.net> writes:

On 04/17/2011 07:41 PM, Robert Haas wrote:

That puts the overall patch acceptance rate at perhaps 75%.

That someone overstates the acceptance rate, because it ignores the
patches that people post and immediately get flamed to a well-done
crisp before adding them to the CF app, but there are not very many of
those any more.

I don't believe there were ever terribly many of them.

Well, that number also ignores patches that were *committed* without
ever making it to the CF list. There aren't terribly many of those
either I think, but it does happen, particularly for small patches.
If you want to argue about the acceptance rate for out-of-CF-process
patches you'd have to do some serious digging in the archives to say
anything about what it is.

But anyway this is quibbling. The point I was trying to make is that
our patch acceptance rate is fairly far north of 50%, not south of it.
So we might hold people's feet to the fire a bit in the process, but
it's hardly impossible to get a patch committed.

regards, tom lane

Robert Haas <robertmhaas@gmail.com> Monday 18 April 2011 03:06:17

On Sun, Apr 17, 2011 at 5:32 PM, Radosław Smogura

<rsmogura@softperience.eu> wrote:

Each process has simple "mirror" of shared descriptors.

I "believe" that modifications to buffer content may be only done when
holding exclusive lock (with some simple exceptions) (+ MVCC), actually
I saw only two things that can change already loaded data and cause
damage, you have described (setting hint bits during scan, and vacuum -
1st may only cause, I think, that two processes will ask for same
transaction statuses <except vacuum>, 2nd one is impossible as vacumm
requires exclusive pin). When buffer tag is changed the version of
buffer is bumped up, and checked against local version - this about
reading buffer.

Yes, an exclusive lock is required for substantive content changes.
But if vacuum cleaning up the buffer is an issue for your patch, then
it's probably also a problem if someone grabs an exclusive content
lock and deletes the tuple (by setting XMAX) and some other backend
later sees the old buffer contents after having in the meanwhile taken
a new snapshot; or if likewise someone grabs an exclusive-lock, adds a
tuple, and then your backend takes a new snapshot and then sees the
old buffer contents. Basically, any time someone grabs an
exclusive-lock and releases it, it's necessary for all observers to
see the updated contents by the time the exclusive lock is released.

In other cases after obtaining lock check is done if buffer has
associated updatable buffer and if local "mirror" has it too, then swap
should take place.

I think this check would have to be done every time someone
share-locks the buffer, which seems rather expensive.

I don't treat as issues, but it's disadvantage.

Logic about updatable buffers is similar to "shared buffers", each
updatable buffer has pin count, and updatable buffer can't be free if
someone uses it, but in contrast to "normal buffers", updatable buffers
doesn't have any support for locking etc. Updatable buffers exists only
on free list, or when associated with buffer.

I don't see how you're going to get away with removing buffer locks.
They exist for a reason, and adding mmap() to the mix is going to
require MORE locking, not less.

In future, I will change version to shared segment id, something like
relation's oid + block, but ids will have continuous numbering 1,2,3...,
so I will be able to bypass smgr/md during read, and tag version check -
this looks like faster solution.

I don't understand this part at all.

Versioning is witch approach where I thought about really often changes of
mmaped areas, I allocated part of segments, but now the segment is mmaped with
reservation, to it's full possible size, addresses of segments can't change
(problem is only with segment deletion).

Regards,
Radek

Robert Haas <robertmhaas@gmail.com> writes:

... Maybe someone out there is under the impression
that I get high off of rejecting patches; but the statistics you cite
from the CF app don't exactly support the contention that I'm going
around looking for reasons to reject things, or if I am, I'm doing a
pretty terrible job finding them.

Hm ... there are people out there who think *I* get high off rejecting
patches. I have a t-shirt to prove it. But I seem to be pretty
ineffective at it too, judging from these numbers.

regards, tom lane

Greg Smith <greg@2ndquadrant.com> writes:

The last time I tried to do this a few years ago I failed miserably and
never came back. I know way more about building software now though,
and just got this to work for the first time.

BTW, another thing that should be in the try-try-again category is
seeing how close we could get to pgindent's results with GNU indent.
It seems clear to me that a process based on GNU indent would be a
lot easier for a lot of people. We tried that once before, and couldn't
get close enough to want to consider switching, but maybe it just needs
a more determined effort and/or more recent versions of GNU indent.
(ISTR that we hit some things that seemed to be outright bugs in GNU
indent, but this was quite a few years ago.)

regards, tom lane

On Sun, 17 Apr 2011 21:06:17 -0400, Robert Haas wrote:

On Sun, Apr 17, 2011 at 5:32 PM, Radosław Smogura
<rsmogura@softperience.eu> wrote:

Each process has simple "mirror" of shared descriptors.

I "believe" that modifications to buffer content may be only done
when holding
exclusive lock (with some simple exceptions) (+ MVCC), actually I
saw only two
things that can change already loaded data and cause damage, you
have
described (setting hint bits during scan, and vacuum - 1st may only
cause, I
think, that two processes will ask for same transaction statuses
<except
vacuum>, 2nd one is impossible as vacumm requires exclusive pin).
When buffer
tag is changed the version of buffer is bumped up, and checked
against local
version - this about reading buffer.

Yes, an exclusive lock is required for substantive content changes.
But if vacuum cleaning up the buffer is an issue for your patch, then
it's probably also a problem if someone grabs an exclusive content
lock and deletes the tuple (by setting XMAX) and some other backend
later sees the old buffer contents after having in the meanwhile
taken
a new snapshot; or if likewise someone grabs an exclusive-lock, adds
a
tuple, and then your backend takes a new snapshot and then sees the
old buffer contents. Basically, any time someone grabs an
exclusive-lock and releases it, it's necessary for all observers to
see the updated contents by the time the exclusive lock is released.

In other cases after obtaining lock check is done if buffer has
associated
updatable buffer and if local "mirror" has it too, then swap should
take
place.

I think this check would have to be done every time someone
share-locks the buffer, which seems rather expensive.

Logic about updatable buffers is similar to "shared buffers", each
updatable
buffer has pin count, and updatable buffer can't be free if someone
uses it,
but in contrast to "normal buffers", updatable buffers doesn't have
any
support for locking etc. Updatable buffers exists only on free list,
or when
associated with buffer.

I don't see how you're going to get away with removing buffer locks.
They exist for a reason, and adding mmap() to the mix is going to
require MORE locking, not less.

In future, I will change version to shared segment id, something
like
relation's oid + block, but ids will have continuous numbering
1,2,3..., so I
will be able to bypass smgr/md during read, and tag version check -
this looks
like faster solution.

I don't understand this part at all.

To my previous post I want to clarify that "updatable buffers" are
implemented in shared memory, so there is no way that process has own
copy of data.

Regards,
Radek.

On 04/18/2011 12:48 AM, Greg Smith wrote:

Andrew Dunstan wrote:

Now we could certainly make this quite a bit slicker. Apart from
anything else, we should change the indent source code tarball so it
unpacks into its own directory. Having it unpack into the current
directory is ugly and unfriendly. And we should get rid of the "make
clean" line in the install target of entab's makefile, which just
seems totally ill-conceived.

I think the script I submitted upthread has most of the additional
slickness needed here. Looks like we both were working on documenting
a reasonable way to do this at the same time the other day. The idea
of any program here relying on being able to write to /usr/local/bin
as your example did makes this harder for people to run; that's why I
made everything in the build tree and just pushed the appropriate
directories into the PATH.

Since I see providing a script to automate this whole thing as the
preferred way to make this easier, re-packaging the indent source
tarball to extract to a directory doesn't seem worth the backwards
compatibility trouble it will introduce. Improving the entab makefile
I don't have an opinion on.

Personally, I want pgindent installed in /usr/local/ or similar. That
way I can have multiple trees and it will work in all of them without my
having to build it for each. What I don't want is for the installed
patched BSD indent to conflict with the system's indent, which is why I
renamed it. If you still think that's a barrier to easy use, then I
think we need a way to provide hooks in the makefiles for specifying the
install location, so we can both be satisfied.

Since there's no script I know of other than your prototype, I don't
think repackaging is likely to break anything. That makes it worth doing
*now* rather than later.

But frankly, I'd rather do without an extra script if possible.

It might also be worth setting it up so that instead of having to
pass a path to a typedefs file on the command line, we default to a
file sitting in, say, /usr/local/etc. Then you'd just be able to say
"pgindent my_file.c".

OK, so I need to update my script to handle either indenting a single
file, or doing all of them.

Yes, very much.

cheers

andrew

Excerpts from Tom Lane's message of lun abr 18 02:50:22 -0300 2011:

Robert Haas <robertmhaas@gmail.com> writes:

... Maybe someone out there is under the impression
that I get high off of rejecting patches; but the statistics you cite
from the CF app don't exactly support the contention that I'm going
around looking for reasons to reject things, or if I am, I'm doing a
pretty terrible job finding them.

Hm ... there are people out there who think *I* get high off rejecting
patches. I have a t-shirt to prove it. But I seem to be pretty
ineffective at it too, judging from these numbers.

Does this mean we need an auction to get Robert a nice $1000 t-shirt?

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Robert, Tom,

Hm ... there are people out there who think *I* get high off rejecting
patches. I have a t-shirt to prove it. But I seem to be pretty
ineffective at it too, judging from these numbers.

It's a question of how we reject patches, especially first-time patches. We can reject them in a way which makes the submitter more likely to fix them and/or work on something else, or we can reject them in a way which discourages people from submitting to PostgreSQL at all.

For example, the emails to Radoslaw mentioned nothing about pg_ident, documented spacing requirements, accidental inclusion of files he didn't mean to touch, etc. Instead, a couple of people told him he should abandon his chosen development IDE in favor of emacs or vim. Radoslaw happens to be thick-skinned and persistent, but other first-time submitters would have given up at that point and run off to a more welcoming project.

Mind, even better would be to get our "so you're submitting a patch" documentation and tools into shape; that way, all we need to do is send the first-time submitter a link. Will work on that between testing ...

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

Does this mean we need an auction to get Robert a nice $1000 t-shirt?

... starting hunting through Robert's emails for a good quote ...

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/17/2011 11:07 PM, Tom Lane wrote:

BTW, another thing that should be in the try-try-again category is
seeing how close we could get to pgindent's results with GNU indent.
It seems clear to me that a process based on GNU indent would be a
lot easier for a lot of people. We tried that once before, and couldn't
get close enough to want to consider switching, but maybe it just needs
a more determined effort and/or more recent versions of GNU indent.
(ISTR that we hit some things that seemed to be outright bugs in GNU
indent, but this was quite a few years ago.)

That seems like a definite win possibility there.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Developement
Organizers of the PostgreSQL Conference - http://www.postgresqlconference.org/
@cmdpromptinc - @postgresconf - 509-416-6579

On Mon, Apr 18, 2011 at 12:52 PM, Joshua Berkus <josh@agliodbs.com> wrote:

Robert, Tom,

Hm ... there are people out there who think *I* get high off rejecting
patches. I have a t-shirt to prove it. But I seem to be pretty
ineffective at it too, judging from these numbers.

It's a question of how we reject patches, especially first-time patches.   We can reject them in a way which makes the submitter more likely to fix them and/or work on something else, or we can reject them in a way which discourages people from submitting to PostgreSQL at all.

For example, the emails to Radoslaw mentioned nothing about pg_ident, documented spacing requirements, accidental inclusion of files he didn't mean to touch, etc.  Instead, a couple of people told him he should abandon his chosen development IDE in favor of emacs or vim.  Radoslaw happens to be thick-skinned and persistent, but other first-time submitters would have given up at that point and run off to a more welcoming project.

Actually, the first reply was a very polite reply from Heikki pointing
out the problem very gently and asking for a theory of operation.

Radoslaw replied and said that he understood the formatting problem,
but his editor was mangling it:

Yes, but, hmm... in Netbeans I had really long gaps (probably 8 spaces, from tabs), so deeper "ifs", comments at the and of variables, went of out my screen. I really wanted to not format this, but sometimes I needed.

That prompted one - ONE! - person to reply and suggest that the use of
another editor might work better. At which point, we got an
apparently-exasperated note from you suggesting that a 10% performance
improvement wasn't enough (which I disagree with) and that it was
wrong for people to worry about whether they could read the patch well
enough to understand it (which I also disagree with). Conceding that
some of the following discussion may have gotten a little harsh
(though frankly I think that was mostly directed at your remark, not
the OP), what prompted that original note? Here it is:

Guys, can we *please* focus on the patch for now, rather than the formatting, which is fixable with sed?

So first of all, no it's not fixable with sed. But secondly, writing
"*please*" here seems to evince a level of frustration which is
entirely out of proportion to the really rather mild comments which
preceded it. What made you write it that way?

I think that the harshness of the reaction to your statement is a
reflection of some underlying frustration on my part and perhaps also
on the part of other reviewers - to this continual commentary that we
are not nice enough to people, especially newcomers. Well, OK, maybe
we're not. But you know what? We're trying really hard, and getting
accused of being nasty when we actually weren't is kind of a tough
pill to swallow. I would really like to see someone go back and look
at every patch from a newcomer that's been submitted in the last year,
and rate the reaction to that patch on an A-F scale. Then let's have
a discussion about what percentage we did well on, and what percentage
we did poorly on, and how we could have done better. When we actually
start raking someone over the coals, I think it's great and a helpful
service for you to jump in and say - hold on a minute, timeout. But
in this case I think you were too quick off the trigger, and I don't
think that acting as if it's unreasonable to want a patch that
conforms to our submission guidelines is doing anyone any favors.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 4/18/11 10:57 AM, Robert Haas wrote:

So first of all, no it's not fixable with sed. But secondly, writing
"*please*" here seems to evince a level of frustration which is
entirely out of proportion to the really rather mild comments which
preceded it. What made you write it that way?

I'll admit that the conversation I'd had at the Drizzle BOF the previous
night strongly influenced me.

to this continual commentary that we
are not nice enough to people, especially newcomers. Well, OK, maybe
we're not. But you know what? We're trying really hard, and getting
accused of being nasty when we actually weren't is kind of a tough
pill to swallow ... But
in this case I think you were too quick off the trigger

Well, my apologies to you. You are probably correct.

In any case, I think the answer to this is constructive; better
documentation and tools to let submitters get their code into good shape
in the first place so that we don't have discussions about formatting.
That way we waste *neither* the reviewers' nor the submitters' time.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/18/2011 01:46 PM, Joshua D. Drake wrote:

On 04/17/2011 11:07 PM, Tom Lane wrote:

BTW, another thing that should be in the try-try-again category is
seeing how close we could get to pgindent's results with GNU indent.
It seems clear to me that a process based on GNU indent would be a
lot easier for a lot of people. We tried that once before, and couldn't
get close enough to want to consider switching, but maybe it just needs
a more determined effort and/or more recent versions of GNU indent.
(ISTR that we hit some things that seemed to be outright bugs in GNU
indent, but this was quite a few years ago.)

That seems like a definite win possibility there.

If you're aware of any changes in GNU indent that would overcome the
previous issues, then by all means spend the time on it. If not, it
seems a bit like the definition of insanity ("repeating an experiment
with the expectation of a different result").

cheers

andrew

Excerpts from Andrew Dunstan's message of lun abr 18 19:20:30 -0300 2011:

On 04/18/2011 01:46 PM, Joshua D. Drake wrote:

On 04/17/2011 11:07 PM, Tom Lane wrote:

BTW, another thing that should be in the try-try-again category is
seeing how close we could get to pgindent's results with GNU indent.
It seems clear to me that a process based on GNU indent would be a
lot easier for a lot of people. We tried that once before, and couldn't
get close enough to want to consider switching, but maybe it just needs
a more determined effort and/or more recent versions of GNU indent.
(ISTR that we hit some things that seemed to be outright bugs in GNU
indent, but this was quite a few years ago.)

That seems like a definite win possibility there.

If you're aware of any changes in GNU indent that would overcome the
previous issues, then by all means spend the time on it. If not, it
seems a bit like the definition of insanity ("repeating an experiment
with the expectation of a different result").

The source of GNU indent itself is 3x what it was when the experiment
was last reported. (I checked this about a year ago with an eye on
"repeating the experiment" but then I failed to actually do it.) It
seems fair to say that, yes, it has changed a bit in the meantime.

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:

Excerpts from Andrew Dunstan's message of lun abr 18 19:20:30 -0300 2011:

On 04/17/2011 11:07 PM, Tom Lane wrote:

... but maybe it just needs
a more determined effort and/or more recent versions of GNU indent.

If you're aware of any changes in GNU indent that would overcome the
previous issues, then by all means spend the time on it. If not, it
seems a bit like the definition of insanity ("repeating an experiment
with the expectation of a different result").

The source of GNU indent itself is 3x what it was when the experiment
was last reported. (I checked this about a year ago with an eye on
"repeating the experiment" but then I failed to actually do it.) It
seems fair to say that, yes, it has changed a bit in the meantime.

Also, my recollection of the previous go-round is that we gave up rather
quickly. Maybe by now there is somebody willing to put more than
minimal effort into getting the options just so.

regards, tom lane

On Mon, Apr 18, 2011 at 3:17 PM, Josh Berkus <josh@agliodbs.com> wrote:

In any case, I think the answer to this is constructive; better
documentation and tools to let submitters get their code into good shape
in the first place so that we don't have discussions about formatting.
That way we waste *neither* the reviewers' nor the submitters' time.

Well, I'm all in favor of better documentation, but I think the
biggest thing we need to do is get the word out:

1. We realize we have been too trigger-happy sometimes.
2. But we really want you to participate.
3. And we are trying very hard to do better.
4. And please tell us if we screw up, so we can keep working on it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert,

1. We realize we have been too trigger-happy sometimes.
2. But we really want you to participate.
3. And we are trying very hard to do better.
4. And please tell us if we screw up, so we can keep working on it.

I received a private offlist email from someone who didn't feel
comfortable bringing up their issues with this list publicly. Let me
quote from it, because I think it pins part of the issue:

"I believe this is due to the current postgresql "commitfest" process
whereby there is no real way to present new ideas or technologies
without coming to the table with a fully-baked plan and patch. This is
obvious even in the name "commitfest" since the expectation is that
every patch presented is considered ready-to-commit by the patch
presenter. This makes a novice or experimental contribution less likely."

You'll notice that this has been a complaint of veteran contributors as
well; WIP patches either get no review, or get reviewed as if they were
expected to be committable.

The person who e-mailed me suggests some form of PostgreSQL Incubator as
a solution. I'm not sure about that, but it does seem to me that we
need somewhere or some way that people can submit patches, ideas, git
forks, etc., for discussion without that discussion needing to
immediately move to the cleanliness/maintainability/supportable status
of the patch.

I'm concerned though that if these WIP projects don't get to -hackers,
then their creators won't get the feedback they really need.

Thoughts?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/18/2011 06:38 PM, Robert Haas wrote:

On Mon, Apr 18, 2011 at 3:17 PM, Josh Berkus<josh@agliodbs.com> wrote:

In any case, I think the answer to this is constructive; better
documentation and tools to let submitters get their code into good shape
in the first place so that we don't have discussions about formatting.
That way we waste *neither* the reviewers' nor the submitters' time.

Well, I'm all in favor of better documentation, but I think the
biggest thing we need to do is get the word out:

1. We realize we have been too trigger-happy sometimes.
2. But we really want you to participate.
3. And we are trying very hard to do better.
4. And please tell us if we screw up, so we can keep working on it.

I think Robert has hit the nail on the head. As I mentioned at #PgWest,
we are a 1000 person dysfunctional family. David Fetter reminded me
gently (yes really) that as far as 1000 person families go, we're doing
pretty good. We are one of the last true communities left.

We need to find a way to let people know that we are only gruff, because
of experience and that although we can be rough we welcome the
participation and we try really hard. We are engineers (well, I'm
not...) but most of us are.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Developement
Organizers of the PostgreSQL Conference - http://www.postgresqlconference.org/
@cmdpromptinc - @postgresconf - 509-416-6579

On Mon, Apr 18, 2011 at 19:50, Josh Berkus <josh@agliodbs.com> wrote:

You'll notice that this has been a complaint of veteran contributors as
well; WIP patches either get no review, or get reviewed as if they were
expected to be committable.

I don't see this changing anytime in the future. We have a hard enough
time getting "finished" patches reviewed.

The person who e-mailed me suggests some form of PostgreSQL Incubator as
a solution.   I'm not sure about that, but it does seem to me that we
need somewhere or some way that people can submit patches, ideas, git
forks, etc., for discussion without that discussion needing to
immediately move to the cleanliness/maintainability/supportable status
of the patch.

Reminds me a bit of what linux is doing with the "staging" tree. I
don't see anyway for that to work with postgres (lower the bar for
-contrib?).

You can fork fairly easy with github nowdays. For example the replace
GEQ with SA is on one of those git sites. Does that mean it gets any
attention? *shrug*

On Mon, Apr 18, 2011 at 9:50 PM, Josh Berkus <josh@agliodbs.com> wrote:

Robert,

1. We realize we have been too trigger-happy sometimes.
2. But we really want you to participate.
3. And we are trying very hard to do better.
4. And please tell us if we screw up, so we can keep working on it.

I received a private offlist email from someone who didn't feel
comfortable bringing up their issues with this list publicly.  Let me
quote from it, because I think it pins part of the issue:

"I believe this is due to the current postgresql "commitfest" process
whereby there is no real way to present new ideas or technologies
without coming to the table with a fully-baked plan and patch. This is
obvious even in the name "commitfest" since the expectation is that
every patch presented is considered ready-to-commit by the patch
presenter. This makes a novice or experimental contribution less likely."

You'll notice that this has been a complaint of veteran contributors as
well; WIP patches either get no review, or get reviewed as if they were
expected to be committable.

The person who e-mailed me suggests some form of PostgreSQL Incubator as
a solution.   I'm not sure about that, but it does seem to me that we
need somewhere or some way that people can submit patches, ideas, git
forks, etc., for discussion without that discussion needing to
immediately move to the cleanliness/maintainability/supportable status
of the patch.

I'm concerned though that if these WIP projects don't get to -hackers,
then their creators won't get the feedback they really need.

Thoughts?

I think the quality of review that WIP patches get depends very much
on how specific the submitter is about what they'd like to get out of
the process. If you submit a patch and say "I have this cool patch
that allows FTL travel, but it's WIP, please review" then you're
basically asking some poor schmuck to reverse engineer what the patch
is doing, and, when they find problems with it, guess which of those
problems were things that you didn't think of and which were things
that you knew about but haven't gotten around to fixing yet because
you're still working on it. This is a pretty thankless task for the
reviewer, and it's not surprising that it doesn't go well. However,
if you say "I have this cool patch that allows FTL travel. It current
plays havoc with the transporter beams and the dilithium crystals tend
to shatter if you exceed Warp 3, but I'd like to get a check as to
whether the basic design is sound, and if anyone can see why the
Heisenburg compensator is destabilizing, please let me know", your
chances of getting some useful feedback are pretty good. Sometimes it
even provokes a rather competitive spot-the-bug race...

Also, I think the reason why we have a process called CommitFest and
not a process called BrainstormingFest is because, when we didn't have
a CommitFest process, patches fell on the floor. Since we've added
that process, that problem has largely gone away. But it is generally
not difficult to get a review of a "big idea" for which no code has
been written yet - in fact it's often much faster and easier than
getting a patch reviewed. It's true that there have been occasional
times when people have gotten lightly toasted for bringing up big new
ideas in the middle of a CF or beta period, but I think we've gotten
less pedantic about that. Certainly, there are no shortage of ideas
that have been proposed and commented on over the last few weeks, even
as we have been working to get 9.1beta1 out the door. Code is not
really getting reviewed right now, but ideas *are*. I'm not going to
claim that this works perfectly: the way that ideas are presented and
the relative level of interest and/or exhaustion of the people
responding certainly play a role, but it is a pretty rare for an email
to -hackers to get no answer at all. Maybe we need some formal
process here just to make people more comfortable, but it's not
necessary from a workflow perspective.

Thinking back over the kinds of things that have lead to people
getting jumped on, I think I can identify a pattern: people tend to
get jumped on when they allege that our code sucks, or that they're
smarter than we are. Whether or not they actually meant to imply
those things turns out not to matter - it rubs people the wrong way,
and everyone's a volunteer, so when you rub them the wrong way, they
get annoyed. I make an effort, as I think most of us do, to be aware
that just because someone makes an annoying remark doesn't necessarily
mean that they are an annoying person; it just means they haven't
quite figured it all out yet. But there are still people who get
flamed that way far more than they probably deserve. That's an area
we can improve, but in the meantime, approaching the topic with a bit
of humility goes a long way. I can't remember the last time someone
said "I was thinking about working on ... and I thought I might
approach it by ... Does this seem like a good idea? Is it likely to
be too hard for me to tackle? My skillset is ..." and got flamed for
it. Some people here (myself included) get a bit pricklier than we
probably ought to from time to time, but everyone is well-meaning and
sincerely wants to help. The list of users who have had Tom fix a bug
for them within hours of posting a question is not short, and the list
of people who have spent time and energy helping newcomers get started
with PostgreSQL tuning, hacking, or whatever is very long.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Apr 18, 2011 at 11:15 PM, Alex Hunsaker <badalex@gmail.com> wrote:

On Mon, Apr 18, 2011 at 19:50, Josh Berkus <josh@agliodbs.com> wrote:

You'll notice that this has been a complaint of veteran contributors as
well; WIP patches either get no review, or get reviewed as if they were
expected to be committable.

I don't see this changing anytime in the future. We have a hard enough
time getting "finished" patches reviewed.

Sadly so.

As much as I think we have gotten a LOT of useful milage out of the
"commitfest" concept, it does, conceptually, have a strong bias
(including in its very name) towards the assumption that changes are
pretty much ready to commit.

Two items still undergoing work (collations, sync rep) weren't at that
level of readiness, needing some mere "dusting off" to make them
ready. Rather, they needed substantial examination and modification
before they'd be ready. And, while this has doubtless aroused some
ire, it doesn't intrinsically make those items "broken."

The Apache guys may be onto something in having the "incubator"
moniker, for things that aren't "so ready we're calling them
Commitable."

There may be merit to separating out "easy to commit" and "tougher to
commit" items, and having different kinds of pickiness for them, the
former being good fodder for "Easy CommitFest" and the latter being
"PG Incubation."

Though I'm not sure the latter makes it any easier to get tough
features like synchronous replication into place.

The person who e-mailed me suggests some form of PostgreSQL Incubator as
a solution.   I'm not sure about that, but it does seem to me that we
need somewhere or some way that people can submit patches, ideas, git
forks, etc., for discussion without that discussion needing to
immediately move to the cleanliness/maintainability/supportable status
of the patch.

Reminds me a bit of what linux is doing with the "staging" tree. I
don't see anyway for that to work with postgres (lower the bar for
-contrib?).

You can fork fairly easy with github nowdays. For example the replace
GEQ with SA is on one of those git sites. Does that mean it gets any
attention? *shrug*

Well, the project hasn't been on Git for all that spectacularly long a
time, so the comfort level with managing via forks maybe isn't quite
there yet.

Forking isn't as magically delicious as GitHub might make some
imagine; it's fine and useful to have a bunch of forks, and eventually
merge useful ones, when they are remaining pretty close together, and
don't conflict. That's likely to work out happily for features that
are essentially independent. If you and I are hacking on different
contrib modules, that's pretty "essentially independent."

Unfortunately, deeper features are more likely to be more
interdependent, and forks aren't so readily productive in that case.

If we hack around with formatting, that would muck with *everything*
else, as an even worse "for instance."
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On Mon, Apr 18, 2011 at 11:50 PM, Christopher Browne <cbbrowne@gmail.com> wrote:

Two items still undergoing work (collations, sync rep) weren't at that
level of readiness, needing some mere "dusting off" to make them
ready.  Rather, they needed substantial examination and modification
before they'd be ready.  And, while this has doubtless aroused some
ire, it doesn't intrinsically make those items "broken."

I don't think it really aroused that much ire. It's pretty clear that
both of those patches cost us something on the schedule, and I would
have preferred to see them committed sooner and with fewer bugs. But
they are great features. Unfortunately, we have a tendency to leave
things to the last minute, and that's something I think we could
improve. We have gotten a bit better but there is clearly room for
further improvement. With beta having gotten pushed out to the end of
the month, there is a real chance that we are going to end up
releasing in the fall again, and I would have much preferred July 1.
But given how long CF4 lasted and how much surgery was required
afterwards, it was an unfixable problem. It's not going to get any
better unless we get more serious about getting these big features
done early in the cycle, or postponing them to the next release if
they aren't. Anyway, I'm drifting off topic: nothing against the
patches, at least on my part, just want to make the schedule.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Josh Berkus <josh@agliodbs.com> writes:

I received a private offlist email from someone who didn't feel
comfortable bringing up their issues with this list publicly. Let me
quote from it, because I think it pins part of the issue:

"I believe this is due to the current postgresql "commitfest" process
whereby there is no real way to present new ideas or technologies
without coming to the table with a fully-baked plan and patch. This is
obvious even in the name "commitfest" since the expectation is that
every patch presented is considered ready-to-commit by the patch
presenter. This makes a novice or experimental contribution less likely."

As Robert noted, the purpose of the commitfest mechanism is mostly to
ensure that patches that *are* committable, or close to it, don't fall
through the cracks. I'm not sure we're doing anybody any favors by
trying to shoehorn reviews of WIP ideas into that same process. At the
very least it seems we'd need a different set of review guidelines for
WIP items, and we don't have one.

I think useful reviewing of WIP stuff has to focus much more on design
concepts and much less on code reading. The reason why the mmap patch
was getting such negative feedback was that there was no way to provide
such a review except by reverse-engineering the design out of some very
messily-presented code. So if we're going to do anything about this,
what we have to do is tell people that the first thing to present for
a WIP review is a design document. If they feel a need to write some
throwaway code to help them clarify their ideas, fine ... but *don't
show us that code*. Write a design document. Get that reviewed.
Then see about coding it, or bringing your first-draft code up to the
point where it can stand the light of day.

I don't know if we need a formal process akin to CFs for reviewing
design documents. I think people are usually plenty willing to discuss
ideas on -hackers, unless maybe you hit them at a particularly bad time
like when they're already burnt out towards the end of a CF.

regards, tom lane

Excerpts from Tom Lane's message of mar abr 19 03:34:34 -0300 2011:

As Robert noted, the purpose of the commitfest mechanism is mostly to
ensure that patches that *are* committable, or close to it, don't fall
through the cracks. I'm not sure we're doing anybody any favors by
trying to shoehorn reviews of WIP ideas into that same process. At the
very least it seems we'd need a different set of review guidelines for
WIP items, and we don't have one.

I think this is historical revisionism. Commitfests were mostly created
because of pressure due to the lateness of the HOT patch. Probably
there were other factors too but this is likely the single most
important reason. (I think the term "commitfest" was coined later, but
I don't think this invalidates my point.)

And the way we considered things at the time is that we had failed to
timely review the concepts in the WIP HOT patch that was presented. So
we wanted to ensure that we provided good feedback to WIP patches (to
all patches really) to avoid this failure from repeating. All patches
*and WIP ideas* were supposed to be reviewed by someone, and if they
were to be rejected, some rationale was to be provided.

Somewhere down the line this seems to have been forgotten and we are now
using commitfests just to track finished patches.

So if we want to stick to the original principles we should have some
sort of "different set of review guidelines". Or perhaps we could just
decide that we don't care much about this problem and toss it aside.

Maybe this is something to discuss at the next developer's meeting.

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:

I think this is historical revisionism. ...
Somewhere down the line this seems to have been forgotten and we are now
using commitfests just to track finished patches.

So if we want to stick to the original principles we should have some
sort of "different set of review guidelines". Or perhaps we could just
decide that we don't care much about this problem and toss it aside.

Well, I absolutely think that we need to encourage people to get
feedback at the design and prototype stages. The problem with the
commitfest mechanism for that is that when you are trying to work out a
patch, you don't want to wait around for a couple months for comments.
The time delay that's built into the CF process means that it's
fundamentally not very good for anything except finished patches that
can sit on a shelf for awhile before they get applied.

I think that ideally, WIP reviews would be something that happens
quickly on pgsql-hackers, and probably it would be best if they were
explicitly *not* encouraged while a CF is on. I know that I tend to see
discussions of unfinished patches as something of a distraction when
I'm up to my ears in committing finished ones, and certainly there's
less mental bandwidth available then.

Maybe this is something to discuss at the next developer's meeting.

I'd rather talk about it on-list so we can get comments from a wider
circle of people.

regards, tom lane

On Wed, Apr 20, 2011 at 5:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Well, I absolutely think that we need to encourage people to get
feedback at the design and prototype stages.  The problem with the
commitfest mechanism for that is that when you are trying to work out a
patch, you don't want to wait around for a couple months for comments.
The time delay that's built into the CF process means that it's
fundamentally not very good for anything except finished patches that
can sit on a shelf for awhile before they get applied.

From my point of view I definitely thought the rationale for
commitfests was as a kind of checkpoint to make sure there weren't any
developers waiting for feedback for a long time. My concurrent index
build patch ended up needing to be reworked and I would have liked to
be involved but it wasn't until feature freeze that you found all
these problems and then it was too late to wait for me to recode
things instead of having you just do it.

I admit though this whole concept of "finished patches" seems foreign
to me. I always have additional stuff I want to do and if the patch
sits on the shelf I'm essentially stuck unable to work on the next
great thing that that patch enables. Developers either have the option
to go off on their own with no feedback and risk having initial
assumptions questioned later and all their work invalidated or go and
work on something unrelated leaving this direction stunted with only
one round of features implemented. I think this is how we ended up
with partitioning that's only halfway useful and selinux that had tons
of code written that needed to be reworked.

Core developers attention is precious and we can't really dictate that
Tom must respond to every email within a week or anything crazy like
that. The commitfests are a dramatic improvement over waiting until
feature freeze which was what was happening before. They also help
bring in new committers and having Robert and Heikki and Peter and
others giving substantive feedback has also improved things
dramatically.

To use a database analogy I think of the commitfests as a checkpoint
-- that doesn't mean we don't also need bgwriter and don't
occasionally need to flush dirty buffers to enable the database to
make progress in the mean-time. But if we didn't have checkpoints at
all things would definitely fall through the cracks and get lost to
bitrot and brainfade.

--
greg

On Wed, Apr 20, 2011 at 12:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

I think this is historical revisionism. ...
Somewhere down the line this seems to have been forgotten and we are now
using commitfests just to track finished patches.

So if we want to stick to the original principles we should have some
sort of "different set of review guidelines".  Or perhaps we could just
decide that we don't care much about this problem and toss it aside.

Well, I absolutely think that we need to encourage people to get
feedback at the design and prototype stages.  The problem with the
commitfest mechanism for that is that when you are trying to work out a
patch, you don't want to wait around for a couple months for comments.
The time delay that's built into the CF process means that it's
fundamentally not very good for anything except finished patches that
can sit on a shelf for awhile before they get applied.

I think that ideally, WIP reviews would be something that happens
quickly on pgsql-hackers, and probably it would be best if they were
explicitly *not* encouraged while a CF is on.  I know that I tend to see
discussions of unfinished patches as something of a distraction when
I'm up to my ears in committing finished ones, and certainly there's
less mental bandwidth available then.

Ditto.

Unfortunately, my memory of this project only goes back to about
September 2008, which isn't far enough to remember why CommitFests
were created in the first place. So Alvaro may be correct in saying
that things have mutated over time, but that isn't necessarily a bad
thing. Maybe we've settled into something that works reasonably well.
Or maybe we should make some changes; nothing is set in stone.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, 2011-04-20 at 17:52 +0100, Greg Stark wrote:

I admit though this whole concept of "finished patches" seems foreign
to me. I always have additional stuff I want to do and if the patch
sits on the shelf I'm essentially stuck unable to work on the next
great thing that that patch enables. Developers either have the option
to go off on their own with no feedback and risk having initial
assumptions questioned later and all their work invalidated or go and
work on something unrelated leaving this direction stunted with only
one round of features implemented. I think this is how we ended up
with partitioning that's only halfway useful and selinux that had tons
of code written that needed to be reworked.

Yeah, there appear to be occasional assumptions that one ought to work
on one major feature per release, and ideally you'd have the plan ready
for the first commit fest and the code mostly ready for the third commit
fest. Whereas I agree with you that it's often rather the case that you
want to work on say three incremental features, and order for that to
work out under this process, you really have to get the first increment
perfect for the first commit fest already. Which is difficult if no one
pays attention until the commit fest starts.

On Wed, 2011-04-20 at 12:21 -0400, Tom Lane wrote:

Well, I absolutely think that we need to encourage people to get
feedback at the design and prototype stages. The problem with the
commitfest mechanism for that is that when you are trying to work out
a patch, you don't want to wait around for a couple months for
comments. The time delay that's built into the CF process means that
it's fundamentally not very good for anything except finished patches
that can sit on a shelf for awhile before they get applied.

I think that ideally, WIP reviews would be something that happens
quickly on pgsql-hackers, and probably it would be best if they were
explicitly *not* encouraged while a CF is on. I know that I tend to
see discussions of unfinished patches as something of a distraction
when I'm up to my ears in committing finished ones, and certainly
there's less mental bandwidth available then.

We'll the current process certainly places a lot of emphasis on the
"finishing" part. You have commit fests that nominally account for 50%
of development time, and then beta, RC, limbo, backbranch releases -- I
blogged about this a while ago, if you follow all these guidelines and
encouragements, you are left with all of about 20 days per year for
discussion, collaborative planning and coding. Which is obviously
silly, which is why the process breaks down. People do other things as
commit fests fade out, but they subconsciously fear they will get the
stink for it, so public discussion and planning is effectively stifled.

I think we should put less temporal emphasis on the finishing part, but
use the time better. I would imagine one commit fest per month, but
it's only a week long. Then everyone can really concentrate on the
commit fest, people get faster feedback, but there is ultimately more
time to do other things. Something to think about.

Robert,

Unfortunately, my memory of this project only goes back to about
September 2008, which isn't far enough to remember why CommitFests
were created in the first place. So Alvaro may be correct in saying
that things have mutated over time, but that isn't necessarily a bad
thing. Maybe we've settled into something that works reasonably well.
Or maybe we should make some changes; nothing is set in stone.

Review of design concepts and WIP patches has *always* been a problem
for this project. Andrew Sullivan bitched about it at some length back
in 2004 ("Why there is no traffic on pgsql-replicationhooks", but
Andrew's blog is down now unfortunately). And I've gotten complaints
from numerous people: the Drizzle student, the person who e-mailed me,
Afilias, Greenplum, Aster Data, others. It's just a broken process, and
it particularly leads PostgreSQL forks to not contribute back stuff.

We tell people to submit a design concept, but then such submissions are
often ignored. When they're not ignored, they often are subject to
either extreme bikeshedding or a lot of negativity around things the
author hasn't implemented yet ... even if the author warns that they're
not implemented.

(btw, I'm not talking about the MMAP patch here, which has gotten
excellent review at this point. I'm talking about a lot of other patches)

I think that Robert is right and what we need is a completely different
process for WIP patches and design concepts. It's pretty clear that
none of the processes we've tried so far ("just post it to
pgsql-hackers", "get a submission mentor" and "commitfest") have worked
consistently.

So in the spirit of NOT reinventing the wheel: ReviewBoard. Yes,
really. One of the big issues with working through design reviews etc.
on this mailing list is the lack of continuity and timeliness in
comments on the idea/WIP patch. Having an interface which presents all
of the discussion around a specific patch in a threaded and
chronological way would help cut down on bikeshedding and dogpiling, as
well as allowing both the idea/patch author to review all commentary in
a coherent way.

Maybe we don't want to use ReviewBoard specifically. Maybe we want to
use bugzilla or Crucible or Redmine something more specific for
patch/spec review. But I think it's time to try something else, maybe
several other things.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Peter Eisentraut <peter_e@gmx.net> writes:

I think we should put less temporal emphasis on the finishing part, but
use the time better. I would imagine one commit fest per month, but
it's only a week long. Then everyone can really concentrate on the
commit fest, people get faster feedback, but there is ultimately more
time to do other things. Something to think about.

Yeah, maybe. To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes. But that might be a good thing
all around. I find this idea attractive.

regards, tom lane

On Wednesday, April 20, 2011 08:50:04 PM Tom Lane wrote:

I think we should put less temporal emphasis on the finishing part, but
use the time better. I would imagine one commit fest per month, but
it's only a week long. Then everyone can really concentrate on the
commit fest, people get faster feedback, but there is ultimately more
time to do other things. Something to think about.

Yeah, maybe. To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes. But that might be a good thing
all around. I find this idea attractive.

Actually as a patch submitter I would somewhat prefer that as well. Its not
exactly easy to learn what wasn't optimal with your patch at times.

On the other hand for some issues its pretty hard to fix the more involved
issues without e.g. Tom's involvement.

Andres

On Wed, Apr 20, 2011 at 2:39 PM, Josh Berkus <josh@agliodbs.com> wrote:

Review of design concepts and WIP patches has *always* been a problem
for this project.  Andrew Sullivan bitched about it at some length back
in 2004 ("Why there is no traffic on pgsql-replicationhooks", but
Andrew's blog is down now unfortunately).  And I've gotten complaints
from numerous people: the Drizzle student, the person who e-mailed me,
Afilias, Greenplum, Aster Data, others.  It's just a broken process, and
it particularly leads PostgreSQL forks to not contribute back stuff.

We tell people to submit a design concept, but then such submissions are
often ignored.

Please provide the evidence that this is a problem that exists now, as
opposed to seven years ago. I leave pgsql-hackers emails marked
unread until they have gotten a response, especially if it's something
important like a design proposal. I have 10 unread threads at the
moment; and I don't think any of them are design proposals except
possibly "Still more REINDEX fun", which was posted 9 minutes ago by
Tom - presumably not the case you are concerned about. I have worked
extremely hard to make sure that we do NOT ignore such submissions,
and I would like to hold your feet to the fire on this one a little
bit: let's hear the list of design ideas that have been proposed in
the last year and been ignored. If the process is as bad as you are
alleging, you should find it easy to come up with numerous, recent
examples. I bet you can't.

When they're not ignored, they often are subject to
either extreme bikeshedding or a lot of negativity around things the
author hasn't implemented yet ... even if the author warns that they're
not implemented.

I concede that this happens, but I don't believe it happens nearly as
often as it used to, and, again, let's have some recent examples. I
don't care what happened three years ago; a lot has changed in the
last three years.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 4/20/11 12:00 PM, Robert Haas wrote:

Please provide the evidence that this is a problem that exists now, as
opposed to seven years ago.

Since you're clearly already made up your mind that no problem exists, I
don't have the energy to fight it out with you.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Wed, Apr 20, 2011 at 2:53 PM, Andres Freund <andres@anarazel.de> wrote:

On Wednesday, April 20, 2011 08:50:04 PM Tom Lane wrote:

I think we should put less temporal emphasis on the finishing part, but
use the time better.  I would imagine one commit fest per month, but
it's only a week long.  Then everyone can really concentrate on the
commit fest, people get faster feedback, but there is ultimately more
time to do other things.  Something to think about.

Yeah, maybe.  To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes.  But that might be a good thing
all around.  I find this idea attractive.

Actually as a patch submitter I would somewhat prefer that as well. Its not
exactly easy to learn what wasn't optimal with your patch at times.

On the other hand for some issues its pretty hard to fix the more involved
issues without e.g. Tom's involvement.

This would amount to reducing the amount of time we spend
in-CommitFest from 50% to slightly less than 25%. That would
certainly be pleasant from my point of view, but for the average patch
to get the same amount of attention, we'd need twice as many
volunteers, or the existing people to volunteer twice as much time, or
everyone to work twice as fast as they already are. That's not
impossible, if the new system inspires more people to contribute, but
2x is a lot, especially when you correct for relative skill levels:
we're not going to find another Tom Lane.

Still, it's an interesting thought.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wednesday, April 20, 2011 08:39:47 PM Josh Berkus wrote:

Robert,

Unfortunately, my memory of this project only goes back to about
September 2008, which isn't far enough to remember why CommitFests
were created in the first place. So Alvaro may be correct in saying
that things have mutated over time, but that isn't necessarily a bad
thing. Maybe we've settled into something that works reasonably well.

Or maybe we should make some changes; nothing is set in stone.

Review of design concepts and WIP patches has *always* been a problem
for this project. Andrew Sullivan bitched about it at some length back
in 2004 ("Why there is no traffic on pgsql-replicationhooks", but
Andrew's blog is down now unfortunately). And I've gotten complaints
from numerous people: the Drizzle student, the person who e-mailed me,
Afilias, Greenplum, Aster Data, others. It's just a broken process, and
it particularly leads PostgreSQL forks to not contribute back stuff.

Well. But very few company people to contribute back in reviewing stuff from
others. At least in the time I have somewhat regularly

We tell people to submit a design concept, but then such submissions are
often ignored. When they're not ignored, they often are subject to
either extreme bikeshedding or a lot of negativity around things the
author hasn't implemented yet ... even if the author warns that they're
not implemented.

I can see that point.

I think that Robert is right and what we need is a completely different
process for WIP patches and design concepts. It's pretty clear that
none of the processes we've tried so far ("just post it to
pgsql-hackers", "get a submission mentor" and "commitfest") have worked
consistently.

So in the spirit of NOT reinventing the wheel: ReviewBoard. Yes,
really. One of the big issues with working through design reviews etc.
on this mailing list is the lack of continuity and timeliness in
comments on the idea/WIP patch. Having an interface which presents all
of the discussion around a specific patch in a threaded and
chronological way would help cut down on bikeshedding and dogpiling, as
well as allowing both the idea/patch author to review all commentary in
a coherent way.

I don't believe a second that problem is solved by any tool. In my opinion
there simply are very few people being able to do in-depth reviews of complex
patches. And those are also needed to implement complex features or do parts
of features others could not do.

A RRR like process doesn't really help in those cases except catch the most
obvious problems.

Andres

On 04/20/2011 12:05 PM, Josh Berkus wrote:

On 4/20/11 12:00 PM, Robert Haas wrote:

Please provide the evidence that this is a problem that exists now, as
opposed to seven years ago.

Since you're clearly already made up your mind that no problem exists, I
don't have the energy to fight it out with you.

Well, you aren't fighting alone. We have significant problems in this
area. As you said, we always have. There is also a bizarre, almost
insane objection to using tools that "aren't invented here" to solve
problems. The problems you (Josh) present are real, regardless of
Robert's opinion. The thing that is important for everyone to remember
is PERCEPTION IS REALITY.

If people PERCEIVE there is a problem, THERE IS A PROBLEM.

So Robert, with respect to your "show me the money", the money is at
your feet on the floor. JB and I can list multitudes of hackers and
contributors who have the perception of this problem and that perception
is hurting the project because frankly, Astor Data isn't going to waste
it's valuable time (money) fighting our community. We have to make it
damn freaking easy for them or we lose their interest, and thus the
community loses.

From the whales of discontentment society,

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Developement
Organizers of the PostgreSQL Conference - http://www.postgresqlconference.org/
@cmdpromptinc - @postgresconf - 509-416-6579

On Wednesday, April 20, 2011 09:09:48 PM Robert Haas wrote:

On Wed, Apr 20, 2011 at 2:53 PM, Andres Freund <andres@anarazel.de> wrote:

On Wednesday, April 20, 2011 08:50:04 PM Tom Lane wrote:

Yeah, maybe. To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes. But that might be a good thing
all around. I find this idea attractive.

Actually as a patch submitter I would somewhat prefer that as well. Its
not exactly easy to learn what wasn't optimal with your patch at times.
On the other hand for some issues its pretty hard to fix the more
involved issues without e.g. Tom's involvement.

This would amount to reducing the amount of time we spend
in-CommitFest from 50% to slightly less than 25%. That would
certainly be pleasant from my point of view, but for the average patch
to get the same amount of attention, we'd need twice as many
volunteers, or the existing people to volunteer twice as much time, or
everyone to work twice as fast as they already are. That's not
impossible, if the new system inspires more people to contribute, but
2x is a lot, especially when you correct for relative skill levels:
we're not going to find another Tom Lane.
Still, it's an interesting thought.

Additional points:
* perhaps it also frees up time if committers balk earlier if a patch doesn't
meet some requirement
* Patch submitters learn more:
* so they submit better patches in the future
* so they can apply the same standards when they review other patches

Andres

On Wed, Apr 20, 2011 at 3:13 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

On 04/20/2011 12:05 PM, Josh Berkus wrote:

On 4/20/11 12:00 PM, Robert Haas wrote:

Please provide the evidence that this is a problem that exists now, as
opposed to seven years ago.

Since you're clearly already made up your mind that no problem exists, I
don't have the energy to fight it out with you.

Well, you aren't fighting alone. We have significant problems in this area.
As you said, we always have. There is also a bizarre, almost insane
objection to using tools that "aren't invented here" to solve problems. The
problems you (Josh) present are real, regardless of Robert's opinion. The
thing that is important for everyone to remember is PERCEPTION IS REALITY.

If people PERCEIVE there is a problem, THERE IS A PROBLEM.

Absolutely. And I am perfectly well aware that we have screwed this
up from time to time. But I also know that I have spent a very large
amount of time over the last few years trying to improve things. It
would be nice to know whether that has had any impact. If it hasn't,
then half of what I have spent the last two years doing has been a
waste of time.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wednesday, April 20, 2011 08:53:34 PM Andres Freund wrote:

On Wednesday, April 20, 2011 08:50:04 PM Tom Lane wrote:

I think we should put less temporal emphasis on the finishing part, but
use the time better. I would imagine one commit fest per month, but
it's only a week long. Then everyone can really concentrate on the
commit fest, people get faster feedback, but there is ultimately more
time to do other things. Something to think about.

Yeah, maybe. To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes. But that might be a good thing
all around. I find this idea attractive.

Actually as a patch submitter I would somewhat prefer that as well. Its not
exactly easy to learn what wasn't optimal with your patch at times.

Perhaps we should adapt something like the kernel's checkpatch.pl for our
needs?
I.e. something that checks that the most obvious style issues are addressed
(tabs, trailing whitespaces, spacing around braces etc).

Andres

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Apr 20, 2011 at 2:53 PM, Andres Freund <andres@anarazel.de> wrote:

On Wednesday, April 20, 2011 08:50:04 PM Tom Lane wrote:

Yeah, maybe. To do that, we'd have to strongly resist the temptation to
spend a lot of time fixing up submitted patches --- if it's not pretty
darn close to committable, back it goes. But that might be a good thing
all around. I find this idea attractive.

Actually as a patch submitter I would somewhat prefer that as well. Its not
exactly easy to learn what wasn't optimal with your patch at times.
On the other hand for some issues its pretty hard to fix the more involved
issues without e.g. Tom's involvement.

This would amount to reducing the amount of time we spend
in-CommitFest from 50% to slightly less than 25%. That would
certainly be pleasant from my point of view, but for the average patch
to get the same amount of attention, we'd need twice as many
volunteers, or the existing people to volunteer twice as much time, or
everyone to work twice as fast as they already are.

Well, no, that's not the whole story. To me, what the above idea
implies is shifting more of the burden of fixing up patches away from
the committer and back to the patch author. Instead of spending time
fixing up not-quite-ready patches myself, I'd be much more ready to
tell the patch author "do X, Y, and Z, and come back next month".

From the committers' standpoint, this is a great idea precisely because
it suggests we might get to put only 25% and not 50% of our time into
commitfests. But it also makes the work more distributed, and it forces
patch authors to learn the things that committers might otherwise have
done for them silently, which in the long run will make everything work
better.

The key point is that we do have to have much more frequent commitfests.
It's hard to bounce back a patch when you know it will then be delayed
two months, especially if the patch is already two months old and the
author has probably forgotten half of it himself. For me anyway,
"I'll just take half a day and make this look the way I think it should"
is a continual temptation. A shorter CF cycle would weaken the argument
to do that.

I haven't spent any time in the role of a non-committer reviewer, but
I think that the same dynamic might work for reviewers. Basically
what a short cycle would do is encourage people to hit the high points
and turn the review around quickly, dumping the big issues back into the
patch author's lap for fixing. You wouldn't spend time sweating details
until the patch had gotten into a state that justified it. Of course
we'd need to tweak the review guidelines to encourage this sort of
multiple-iterations review approach --- right now the guidelines are
pretty much one-size-fits-all, and this type of approach cannot work
with that.

regards, tom lane

On 4/20/11 12:35 PM, Tom Lane wrote:

Well, no, that's not the whole story. To me, what the above idea
implies is shifting more of the burden of fixing up patches away from
the committer and back to the patch author. Instead of spending time
fixing up not-quite-ready patches myself, I'd be much more ready to
tell the patch author "do X, Y, and Z, and come back next month".

Yes, definitely! For that matter, booting a patch which got no review
is less of a problem if we're only booting it for 3 weeks.

The whole purpose of the CFs was not to help submitters -- it was to
help reviewers. If we just wanted to help submitters, we'd do
Continuous Integration, and review all the time. But the reviewers need
"time off".

I think we should try this for 9.2. Given the accumulation between then
and now, I think the first CF should be 2 weeks, and then we can move to
monthly/weeklong CFs after that. So it would look like:

CF1: July 16-31
CF2: August 1-7
CF3: September 1-7
CF4: October 1-7
CF5: November 1-7
CF6: December 1-7
CF7: January 3-10
CF8: February until done

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Robert,

Absolutely. And I am perfectly well aware that we have screwed this
up from time to time. But I also know that I have spent a very large
amount of time over the last few years trying to improve things. It
would be nice to know whether that has had any impact. If it hasn't,
then half of what I have spent the last two years doing has been a
waste of time.

That would take pretty significant research; it's not like we have a
database of idea/WIP submissions. It's all e-mail.

Not that it wouldn't be worth doing, but it would be an entire day of
someone's time.

BTW, I do still believe that step 1 is tremendously expanding the "so
you want to submit a patch" documentation, and linking it in many places
so that newbies read it. On my TODO list.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Wed, Apr 20, 2011 at 3:05 PM, Josh Berkus <josh@agliodbs.com> wrote:

On 4/20/11 12:00 PM, Robert Haas wrote:

Please provide the evidence that this is a problem that exists now, as
opposed to seven years ago.

Since you're clearly already made up your mind that no problem exists, I
don't have the energy to fight it out with you.

It is not possible for me to work any harder on anything than I have
worked on this problem. I do not deny the existence of the problem.
But I believe that we have greatly mitigated it in the last few
release cycles, and that much of what remains is a problem of
perception, not reality.

You can disagree, but if no one has the energy to find real examples
and talk about them, then it is hard to see how we will be able to
improve the situation.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Peter Eisentraut <peter_e@gmx.net> writes:

I would imagine one commit fest per month, but
it's only a week long.

BTW, just as a thought experiment: what about a one-day CF once a week?
"Patch Tuesdays", if you will. Spend all day reviewing/committing,
bounce back whatever is not ready, patch authors try again next week.

Really large patches are not going to fit into that paradigm, probably,
but an awful lot of stuff would --- and it might help encourage more
incremental development of the big ones, too.

regards, tom lane

On Wed, Apr 20, 2011 at 21:54, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

I would imagine one commit fest per month, but
it's only a week long.

BTW, just as a thought experiment: what about a one-day CF once a week?
"Patch Tuesdays", if you will.  Spend all day reviewing/committing,
bounce back whatever is not ready, patch authors try again next week.

I think that would pretty much kill the process for any committer who
is not employed to work full-time on postgresql *development*. Those
who have other dayjobs (which may well be postgresql consulting or
training or whatever) will probably end up dealing with significantly
fewer patches, leaving even more of the burden on those who do have
the dedicated schedule. I know I don't do as much reviewing/comitting
as I'd like to do during the commitfests, but with a process like
that, it would probably become more or less zero.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 04/20/2011 04:09 PM, Magnus Hagander wrote:

On Wed, Apr 20, 2011 at 21:54, Tom Lane<tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut<peter_e@gmx.net> writes:

I would imagine one commit fest per month, but
it's only a week long.

BTW, just as a thought experiment: what about a one-day CF once a week?
"Patch Tuesdays", if you will. Spend all day reviewing/committing,
bounce back whatever is not ready, patch authors try again next week.

I think that would pretty much kill the process for any committer who
is not employed to work full-time on postgresql *development*. Those
who have other dayjobs (which may well be postgresql consulting or
training or whatever) will probably end up dealing with significantly
fewer patches, leaving even more of the burden on those who do have
the dedicated schedule. I know I don't do as much reviewing/comitting
as I'd like to do during the commitfests, but with a process like
that, it would probably become more or less zero.

Yeah, I can't organize my time that way either.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

On 04/20/2011 04:09 PM, Magnus Hagander wrote:

On Wed, Apr 20, 2011 at 21:54, Tom Lane<tgl@sss.pgh.pa.us> wrote:

BTW, just as a thought experiment: what about a one-day CF once a week?
"Patch Tuesdays", if you will. Spend all day reviewing/committing,
bounce back whatever is not ready, patch authors try again next week.

I think that would pretty much kill the process for any committer who
is not employed to work full-time on postgresql *development*.

Yeah, I can't organize my time that way either.

True, and any fixed day of the week would let out X number of people
anyway. But ignoring scheduling difficulties, my point here is that
it seems like the shorter the cycle, the better, for a lot of purposes.
Can we do any better than once-a-month, or is that the limit given that
people need flexible schedules within the fest?

regards, tom lane

On Wed, 2011-04-20 at 11:39 -0700, Josh Berkus wrote:

Maybe we don't want to use ReviewBoard specifically. Maybe we want
to use bugzilla or Crucible or Redmine something more specific for
patch/spec review. But I think it's time to try something else, maybe
several other things.

I had suggested ideatorrent before. But I agree with you in principle.

Tom,

True, and any fixed day of the week would let out X number of people
anyway. But ignoring scheduling difficulties, my point here is that
it seems like the shorter the cycle, the better, for a lot of purposes.
Can we do any better than once-a-month, or is that the limit given that
people need flexible schedules within the fest?

Also consider that the PostgreSQL development world represents a lot of
different time zones. For me to have some dialog with Tatsuo about a
patch, for example, takes at least 24 hours for a simple back-and-forth.

If we were a full-time development shop in a single time zone, we could
use scrum and do a *daily* integration. Many of my clients do. But for
a high-distributed volunteer-based organization, I don't think it's
practical.

I also find the one-day-a-week attractive. It would make patch review
much more immediate. However, not only would it raise issues with
people's schedules, it would also require us to adopt new tools or
modify the CF code.

Is there anything between one-week-a-month and one-day-a-week?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Wed, 2011-04-20 at 15:09 -0400, Robert Haas wrote:

This would amount to reducing the amount of time we spend
in-CommitFest from 50% to slightly less than 25%. That would
certainly be pleasant from my point of view, but for the average patch
to get the same amount of attention, we'd need twice as many
volunteers, or the existing people to volunteer twice as much time, or
everyone to work twice as fast as they already are.

I think in reality people don't spend more than 50% of their time during
commit fests on the commit fest. By making the commit fests shorter and
tighter, we could perhaps increase that number. More "quality time" if
you will.

On 04/20/2011 12:22 PM, Robert Haas wrote:

Well, you aren't fighting alone. We have significant problems in this area.
As you said, we always have. There is also a bizarre, almost insane
objection to using tools that "aren't invented here" to solve problems. The
problems you (Josh) present are real, regardless of Robert's opinion. The
thing that is important for everyone to remember is PERCEPTION IS REALITY.

If people PERCEIVE there is a problem, THERE IS A PROBLEM.

Absolutely. And I am perfectly well aware that we have screwed this
up from time to time. But I also know that I have spent a very large
amount of time over the last few years trying to improve things. It
would be nice to know whether that has had any impact. If it hasn't,
then half of what I have spent the last two years doing has been a
waste of time.

I don't think anyone would argue that your efforts have not improved the
situation. I certainly wouldn't. However, the perception (and reality of
the problem) definitely still applies. I wouldn't suggest that you stop
what you are doing but that doesn't mean the problem or variances of the
problem don't still exist and need to be addressed.

Sincerely,

jD

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Developement
Organizers of the PostgreSQL Conference - http://www.postgresqlconference.org/
@cmdpromptinc - @postgresconf - 509-416-6579

On Wed, 2011-04-20 at 16:25 -0400, Tom Lane wrote:

But ignoring scheduling difficulties, my point here is that
it seems like the shorter the cycle, the better, for a lot of
purposes. Can we do any better than once-a-month, or is that the
limit given that people need flexible schedules within the fest?

If you want to keep the basic idea of predictable periods of activity
and rest, I think that's as far as you can go.

I'm personally not terribly tied to that; I'm more interested in the
tool support that the CF gives us. I might also like, for example, just
a permanent patch queue with patches sorted by date. Multiple
approaches like that could also very well exist in parallel, even within
the existing commitfest application framework.

Excerpts from Robert Haas's message of mié abr 20 16:22:24 -0300 2011:

If people PERCEIVE there is a problem, THERE IS A PROBLEM.

Absolutely. And I am perfectly well aware that we have screwed this
up from time to time. But I also know that I have spent a very large
amount of time over the last few years trying to improve things. It
would be nice to know whether that has had any impact. If it hasn't,
then half of what I have spent the last two years doing has been a
waste of time.

It may very well be fixed, but if the guys doing the submission (or,
more precisely failing to do it) don't know that things have changed,
they will continue to avoid submitting stuff.

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Wed, Apr 20, 2011 at 3:42 PM, Josh Berkus <josh@agliodbs.com> wrote:

On 4/20/11 12:35 PM, Tom Lane wrote:

Well, no, that's not the whole story.  To me, what the above idea
implies is shifting more of the burden of fixing up patches away from
the committer and back to the patch author.  Instead of spending time
fixing up not-quite-ready patches myself, I'd be much more ready to
tell the patch author "do X, Y, and Z, and come back next month".

Yes, definitely!  For that matter, booting a patch which got no review
is less of a problem if we're only booting it for 3 weeks.

The whole purpose of the CFs was not to help submitters -- it was to
help reviewers.   If we just wanted to help submitters, we'd do
Continuous Integration, and review all the time.  But the reviewers need
"time off".

I think we should try this for 9.2.  Given the accumulation between then
and now, I think the first CF should be 2 weeks, and then we can move to
monthly/weeklong CFs after that.  So it would look like:

CF1: July 16-31
CF2: August 1-7
CF3: September 1-7
CF4: October 1-7
CF5: November 1-7
CF6: December 1-7
CF7: January 3-10
CF8: February until done

I am concerned that this will get us back into the land of the
interminable last CommitFest. I believe that one of the reasons why
things didn't go as smoothly before we had the CommitFest was because
patches didn't get dealt with until the end of the cycle. I think
that if, as proposed, we are faster about pushing patches back on the
submitters when they're not up to snuff, then we will end up having
more stuff bounce along for many CommitFests without actually getting
committed, which will tend to exacerbate the pile-up at the end of the
cycle. The basic underlying problem here is that there is tremendous
reluctance to boot anything when it means pushing it out to the next
release, and I think that's just terrible project management. If we
had punted collations and sync rep to 9.2, we would be on beta2 right
now, instead of still trying to get things squared away for beta1. If
we allow people to submit patches up until supposed feature freeze - 7
days instead of proposed feature freeze - 31 days, that's not going to
help.

Now, maybe if we branched the tree immediately after the last CF of
the release and continued having week-long CFs, we might be able to
make it work. Then, at least if you didn't get your stuff committed
to the right release, you could still get it committed somewhere. But
even then I think we'd have this problem of people being unwilling to
give up on jamming stuff into a release, regardless of the scheduling
impact of doing so. I actually think the problem of getting releases
out on time is a *much* bigger problem for us than how long or short
CommitFests are.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Apr 20, 2011 at 7:00 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

Excerpts from Robert Haas's message of mié abr 20 16:22:24 -0300 2011:

If people PERCEIVE there is a problem, THERE IS A PROBLEM.

Absolutely.  And I am perfectly well aware that we have screwed this
up from time to time.  But I also know that I have spent a very large
amount of time over the last few years trying to improve things.  It
would be nice to know whether that has had any impact.  If it hasn't,
then half of what I have spent the last two years doing has been a
waste of time.

It may very well be fixed, but if the guys doing the submission (or,
more precisely failing to do it) don't know that things have changed,
they will continue to avoid submitting stuff.

Yep.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, 2011-04-20 at 21:09 -0400, Robert Haas wrote:

But
even then I think we'd have this problem of people being unwilling to
give up on jamming stuff into a release, regardless of the scheduling
impact of doing so. I actually think the problem of getting releases
out on time is a *much* bigger problem for us than how long or short
CommitFests are.

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months. Otherwise, the current 12
to 14 month horizon is just too long psychologically.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Apr 20, 2011 at 11:39:47AM -0700, Josh Berkus wrote:

[...]

Review of design concepts and WIP patches has *always* been a problem
for this project [...]

We tell people to submit a design concept, but then such submissions are
often ignored. When they're not ignored, they often are subject to
either extreme bikeshedding or a lot of negativity around things the
author hasn't implemented yet ... even if the author warns that they're
not implemented.

I'm not a committer. So take this data point for what it's worth. But I
have been following this list for quite a while, and I must say: I (very
respectfully!) disagree. Having watched mailing lists for other
projects, the quality of the answers one gets here is outstanding. The
tone might be sometimes a bit tight (but never disrespectful or
flaming), but seriously: what do I get off a friendly answer if there is
no content?

The same goes to -GENERAL. I've always got answers to my (sometimes, in
hindsight quite stupid) questions which actually *helped* to solve my
problem.

It's OK to strive to improve the process, but I think you all are quite
good.

[...]

So in the spirit of NOT reinventing the wheel: ReviewBoard. Yes,
really [...]
[...] But I think it's time to try something else, maybe
several other things.

Maybe. But I *do* understand the unwillingness to change that. I've
contributed (tiny) patches to more that one project, and it's
frustrating to fight the bug-tracker-du-jour system. This one won't talk
to me unless my browser talks Javascript. That one... (you get the
idea). I strongly appreciate the free-flowing mailing list style here
(maybe it's just an age problem ;-)

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFNr9IhBcgs9XrR2kYRAkw+AJoDFJcnpR06VpGNVAzsbx/eZpQcxACfUv//
vFsZsPiYlM78fxsjCLQvbHw=
=A+7H
-----END PGP SIGNATURE-----

On Wed, Apr 20, 2011 at 8:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

I would imagine one commit fest per month, but
it's only a week long.

BTW, just as a thought experiment: what about a one-day CF once a week?
"Patch Tuesdays", if you will.  Spend all day reviewing/committing,
bounce back whatever is not ready, patch authors try again next week.

Really large patches are not going to fit into that paradigm, probably,
but an awful lot of stuff would --- and it might help encourage more
incremental development of the big ones, too.

I'm responding to this post with mostly general comments, not directed
specifically at Tom.

Speeding up the process means that people with more time get a bigger
say and people with less time get a smaller input than before. I'm
already concerned that the gap between patch submission and patch
commit is so short it effectively means feedback is impossible.

The more frequently we do integration, the greater proportion of our
time is spent doing that.

My concern is there are a relatively low number of people working on
features that lots of people care about. Senior time should not be
wasted on endless integration.

We should be encouraging people to spend more time on more useful
features, not an endless stream of trivial patches, integration and
release processes. None of our users give a flying, err, squirrel,
about our small patch review process. Especially when its absolutely
brilliant already.

My model of contributing to this project has always been to spend time
with customers, understanding solutions and problems, then bringing
that back to the community. That has brought both the funding to allow
me to contribute and a stream of ideas with a clear focus. I encourage
others to do the same. I don't think we should be working on an
interrupt driven model, we should be planning our contributions and
making sure we make the biggest impact possible with real code, not
just twittering about it constantly. If we spend too much time with
each other we will be exactly like the larger commercial development
groups who never meet users only each other. Even the General list
isn't fully representative of the actual/potential user base.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Peter Eisentraut <peter_e@gmx.net> wrote:

you need to think about shorter release cycles overall, like every
6 months.

With the current time between feature freeze and release, that
wouldn't leave a lot of time for development.

-Kevin

On Thu, 2011-04-21 at 14:01 +0100, Simon Riggs wrote:

We should be encouraging people to spend more time on more useful
features, not an endless stream of trivial patches, integration and
release processes.

Hence the proposal to cut that time down and make it count better.

Which direction were you thinking?

On Thu, 2011-04-21 at 08:42 -0500, Kevin Grittner wrote:

you need to think about shorter release cycles overall, like every
6 months.

With the current time between feature freeze and release, that
wouldn't leave a lot of time for development.

Presumably, one would aim to cut all the other things in half as well.

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

On Wed, 2011-04-20 at 21:09 -0400, Robert Haas wrote:

But
even then I think we'd have this problem of people being unwilling to
give up on jamming stuff into a release, regardless of the scheduling
impact of doing so.  I actually think the problem of getting releases
out on time is a *much* bigger problem for us than how long or short
CommitFests are.

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months.  Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree. I am in favor of a shorter release cycle. But I think that
a shorter release cycle won't work well if there is still four month
long integration period at the end of each series of CommitFests. The
problem is a bit circular here: because release cycles are long,
people really, really want to slip as much as possible in at the end.
But being under time pressure to get things committed results in a
higher bug count, which means more things that have to be fixed after
feature freeze, which translates into a long release cycle.

I think that it's not too bad if the process of a release getting out
the door results in effectively missing one CommitFest. For example,
if we imagine one-month CommitFests starting every two months, and we
had a CommitFest starting on January 15th, it wouldn't be too painful
if we skipped a hypothetical March 15th CommitFest to get the release
done, and then started up the process again on May 15th. However, in
practice, what happens is we miss *two* CommitFests: the expectation
is that the next CommitFest will be on the order of July 15th, which
is just too long. Similarly, if we did shorter CommitFests and
shorter releases - say, five one-week-a-month CommitFests in July,
August, September, October, and November, I'd want to kick a release
out in December and reopen for development in January, not get stuck
with the same six-month feature freeze we have now, or even a
four-month feature freeze. But that isn't going to work if people do
the same sort of throwing everything into the kitchen sink at the last
minute that we have been doing for at least the last couple of
releases.

In fact, I don't believe that the current CF cycle really forces a
huge amount of waiting-for-feedback. It's true that if you submit a
patch at a randomly chosen time, you will have to wait up to two
months for a CommitFest to start, and then you might not get a review
until late in the CommitFest, so it could take you up to three months
to get a review. In practice, patches are not submitted at random
times - in fact, probably 50% of the patches come in during the last
week before the CF starts, and typically perhaps 50% of the patches
get a review in the first week, and maybe 80% within the first two
weeks. Some patches also get an initial review between CommitFests,
which further improves the average. Overall, I bet the average time
between patch submission and first review is <3 weeks. You can
typically get 2 or 3 followup reviews during the same cycle with only
a few days latency for each. Even though it would be nice to do
better, for an all-volunteer project, I think it's respectable. I
can't say the same thing about our process from getting from feature
freeze to release. It's really long, and it's nearly all fixing bugs
in code that was committed in the last CF, and the last CF produces
exponentially more bugs than the earlier ones, and it's often the case
that people don't fix their own bugs and someone else has to jump in
to pick up the slack. Meanwhile, the regular flow of reviewing and
committing patches is completely disrupted; and once in a while
someone gets flamed for so much as bringing up a new feature that
they're interested in working on for the next release (which I think
is totally unwarranted; now is the PERFECT time to begin roughing out
plans for 9.2 work... but I digress).

So while I'm mildly interested in the idea of shifting the CF cycle
around to provide more timely review, I can't really get that excited
about it, especially if there's any risk that we are just shifting
more of the work from the CommitFest cycle to the
end-of-release-interminable-integration-period. However, if there's
some way of avoiding the phenomenon where all hell breaks loose
because people jam four major new features into the tree in as many
weeks, sign me up.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months. �Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree. I am in favor of a shorter release cycle.

I'm not. I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year. It's hard enough
to get people to migrate that often.

Another problem is that if you halve the release interval, you either
double the amount of work spent on maintaining back branches, or halve
the support lifetime of a branch. Neither of those is attractive.

Now, it certainly would be nice to spend less time in beta mode as
opposed to development, and I think most of the points being made here
are really about how to cut that. But reducing the release interval is
not going to reduce the total amount of time we spend in beta mode;
in fact I'd expect it to increase. Halving the amount of development
time per release doesn't mean that you can cut beta time proportionally.
It just takes time to cut a release, and time for testers to try it.

regards, tom lane

On 04/21/2011 11:16 AM, Tom Lane wrote:

Robert Haas<robertmhaas@gmail.com> writes:

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut<peter_e@gmx.net> wrote:

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months. Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree. I am in favor of a shorter release cycle.

I'm not. I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year. It's hard enough
to get people to migrate that often.

I agree.

Another problem is that if you halve the release interval, you either
double the amount of work spent on maintaining back branches, or halve
the support lifetime of a branch. Neither of those is attractive.

I *really* *really* agree.

cheers

andrew

On Thu, Apr 21, 2011 at 11:16:45AM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months. �Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree. I am in favor of a shorter release cycle.

I'm not. I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year. It's hard enough
to get people to migrate that often.

In fact, I predict that the observed behavior would be for even more end
users to start skipping releases. Some already do - it's common not to
upgrade unless there's a feature you really need, but for those who do
stay on the 'current' upgrade path, you'll lose some who can't afford to
spend more than one integration-testing round a year.

Ross
--
Ross Reedstrom, Ph.D. reedstrm@rice.edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE

On Thu, Apr 21, 2011 at 11:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months.  Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree.  I am in favor of a shorter release cycle.

I'm not.  I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year.  It's hard enough
to get people to migrate that often.

I agree there's probably little user demand, and back-branch
maintenance is an issue, but I think if it removed the temptation to
cram major new features into the tree at the last minute, it might be
worth it. However, a possibly more likely outcome is that we'd still
have that temptation, just more frequently; and end up with even less
of the year open to new patches than is currently the case.

Another problem is that if you halve the release interval, you either
double the amount of work spent on maintaining back branches, or halve
the support lifetime of a branch.  Neither of those is attractive.

Now, it certainly would be nice to spend less time in beta mode as
opposed to development, and I think most of the points being made here
are really about how to cut that.  But reducing the release interval is
not going to reduce the total amount of time we spend in beta mode;
in fact I'd expect it to increase.  Halving the amount of development
time per release doesn't mean that you can cut beta time proportionally.
It just takes time to cut a release, and time for testers to try it.

I believe that the problem is much more related to the fact that we
commit things at the end of the cycle that aren't really done than it
is to the amount of time beta testers need to try things. If we were
only waiting on testing, we could branch the tree and call the release
du jour beta for another N months, then release, meanwhile continuing
development. In fact, you and I and three or four other people have
spent most of our visible PG time over the last 2 months fixing MANY
bugs, mostly in the six or so major features committed between
February 7th and March 6th. (By way of comparison, notice how few
bugs that have been in the major patches from CF3 - because those
things were actually pretty much working *when they were committed*.)

Now, we're getting to the point where that might actually be a
reasonable way to go. It wouldn't bother me a bit to branch the tree
just after beta1 and start a new cycle of CommitFests on May 15th, and
we could begin integrating some of the big stuff that didn't make it
into 9.1: key locks, range types, additional sync rep modes, snapshot
cloning, parallel pg_dump, etc. It would be great to start working on
that stuff while it's still mildly fresh in people's minds, and at the
*beginning* of the release cycle. We're probably doomed to another
fall release at this point anyway, so it's not clear to me that the
inevitable loss of focus that will ensue is really costing anything.
Had we gotten to beta1 on March 1st, I'd probably be in favor of going
all in to get the release out in June or maybe on July 1, but at this
point that seems unlikely to be realistic.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Apr 21, 2011 at 11:37 AM, Ross J. Reedstrom <reedstrm@rice.edu> wrote:

On Thu, Apr 21, 2011 at 11:16:45AM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Thu, Apr 21, 2011 at 2:43 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

I think to really address that problem, you need to think about shorter
release cycles overall, like every 6 months.  Otherwise, the current 12
to 14 month horizon is just too long psychologically.

I agree.  I am in favor of a shorter release cycle.

I'm not.  I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year.  It's hard enough
to get people to migrate that often.

In fact, I predict that the observed behavior would be for even more end
users to start skipping releases. Some already do - it's common not to
upgrade unless there's a feature you really need, but for those who do
stay on the 'current' upgrade path, you'll lose some who can't afford to
spend more than one integration-testing round a year.

Well, that aspect of the problem doesn't bother me, much. I don't
really care whether people upgrade to each new release the moment it
comes out anyway. It would require us to keep any
backward-compatibility hacks around for more releases, but we're
pretty good about that anyway. 8.3 broke the world, but the last few
releases have been pretty smooth for most people, I think.

Not to say that there aren't OTHER problems with the idea...

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[ another thought on this topic ]

Robert Haas <robertmhaas@gmail.com> writes:

I think that it's not too bad if the process of a release getting out
the door results in effectively missing one CommitFest. ...
But that isn't going to work if people do
the same sort of throwing everything into the kitchen sink at the last
minute that we have been doing for at least the last couple of
releases.

In fact, I don't believe that the current CF cycle really forces a
huge amount of waiting-for-feedback. It's true that if you submit a
patch at a randomly chosen time, you will have to wait up to two
months for a CommitFest to start, and then you might not get a review
until late in the CommitFest, so it could take you up to three months
to get a review. In practice, patches are not submitted at random
times - in fact, probably 50% of the patches come in during the last
week before the CF starts, and typically perhaps 50% of the patches
get a review in the first week, and maybe 80% within the first two
weeks.

But aren't those two sides of the same coin, ie, people's natural
tendency to work to a deadline? If you approve of a lot of patches
showing up just in time for a commitfest, why don't you approve of
big patches showing up just in time for a release? I mean, I've been
heard to complain about that too, but complaining hasn't changed
anyone's behavior and it's foolish to expect that it will in the
future. (See insanity, definition of.)

We need to find a way to work with that behavior, not try to change it.
I don't know what exactly.

One idea that comes to mind is to give up on the linear development-mode-
then-beta-mode management model, ie, allow development of release N+1
to start while beta is still going on for release N. The principal
objection to this in the past has been that the PG development community
is too small to do more than one thing at once, but maybe that's not
true anymore. The thing I'd be most worried about is how we get enough
energy directed at the release-stabilization part of the work, when for
most developers the new-development part is much more interesting/fun.
But we have that problem in some form already --- it's not clear to me
how much of the community really engages in what happens during beta,
rather than quietly working on stuff for the next release.

regards, tom lane

On Thursday, April 21, 2011 05:43:16 PM Robert Haas wrote:

On Thu, Apr 21, 2011 at 11:37 AM, Ross J. Reedstrom <reedstrm@rice.edu>

wrote:

On Thu, Apr 21, 2011 at 11:16:45AM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

I agree. I am in favor of a shorter release cycle.

I'm not. I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year. It's hard enough
to get people to migrate that often.

In fact, I predict that the observed behavior would be for even more end
users to start skipping releases. Some already do - it's common not to
upgrade unless there's a feature you really need, but for those who do
stay on the 'current' upgrade path, you'll lose some who can't afford to
spend more than one integration-testing round a year.

Well, that aspect of the problem doesn't bother me, much. I don't
really care whether people upgrade to each new release the moment it
comes out anyway.
Not to say that there aren't OTHER problems with the idea...

One could argue that its causing bad PR for postgres. I have seen several
parties planning to migrate away or not migrate to postgres because of
performance evaluations they made. With 7.4, 8.0 and 8.2. In 2010.

Andres

On Thu, Apr 21, 2011 at 11:46 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

But aren't those two sides of the same coin, ie, people's natural
tendency to work to a deadline?  If you approve of a lot of patches
showing up just in time for a commitfest, why don't you approve of
big patches showing up just in time for a release?  I mean, I've been
heard to complain about that too, but complaining hasn't changed
anyone's behavior and it's foolish to expect that it will in the
future.  (See insanity, definition of.)

Well, I guess I approve of the first behavior because it doesn't feel
like having a red-hot iron spike driven through my foot, and I
disapprove of the second one because it does. That may not be
entirely consistent taken in the abstract, but it has some solid
practical roots.

We need to find a way to work with that behavior, not try to change it.
I don't know what exactly.

One idea that comes to mind is to give up on the linear development-mode-
then-beta-mode management model, ie, allow development of release N+1
to start while beta is still going on for release N.  The principal
objection to this in the past has been that the PG development community
is too small to do more than one thing at once, but maybe that's not
true anymore.  The thing I'd be most worried about is how we get enough
energy directed at the release-stabilization part of the work, when for
most developers the new-development part is much more interesting/fun.
But we have that problem in some form already --- it's not clear to me
how much of the community really engages in what happens during beta,
rather than quietly working on stuff for the next release.

I totally agree. In fact, I think that trying to close off that
activity is one of the most self-destructive things we could possibly
do. It makes missing the release far more painful if you're thinking
about not only a 12-month slip on GA but also a 6-month slip on any
meaningful further review. Encouraging people to hold off major
proposals for the next release while we are focusing on beta also
tends to slow them down, which then exacerbates the pile-up at the end
of the release cycle. I would like to blow the doors on that wide
open and encourage people to start submitting design proposals for 9.2
NOW. NOW, NOW, NOW! Not in July! And *really* not next January!
And frankly, the sooner we can realistically start working on
integrating the code that has *already* been written for 9.2, the
better. The patches are going to land on us at some point, and
dealing with them earlier will allow those people to move on to other
things (which is good), reduce the pile-up at the end of the cycle
(even better), or possibly both.

I'm willing to make a serious commitment to being involved in the
release stabilization work and to give it some degree of priority over
new patches, if that's what it takes to make the process work
smoothly. We are fundamentally resource-constrained, and no process
is going to change that unless the process change, of itself, causes
more people to contribute more time. But even if the first CommitFest
involves a slightly higher bounce rate due to lack of
reviewer/committer bandwidth, it's still better than not having one.
There have been maybe half a dozen people who have been principally
responsible for the stabilization that we have done since CF4, and the
community is much larger than that. Everyone else is either doing
nothing (which is bad), or working without on-list discussion (which
is also bad). Even for the people who are deeply committed to release
stabilization would probably be happier and more motivated to continue
contributing if they weren't being limited to ONLY that.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Apr 21, 2011 at 11:48 AM, Andres Freund <andres@anarazel.de> wrote:

On Thursday, April 21, 2011 05:43:16 PM Robert Haas wrote:

On Thu, Apr 21, 2011 at 11:37 AM, Ross J. Reedstrom <reedstrm@rice.edu>

wrote:

On Thu, Apr 21, 2011 at 11:16:45AM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

I agree.  I am in favor of a shorter release cycle.

I'm not.  I don't think there is any demand among *users* (as opposed to
developers) for more than one major PG release a year.  It's hard enough
to get people to migrate that often.

In fact, I predict that the observed behavior would be for even more end
users to start skipping releases. Some already do - it's common not to
upgrade unless there's a feature you really need, but for those who do
stay on the 'current' upgrade path, you'll lose some who can't afford to
spend more than one integration-testing round a year.

Well, that aspect of the problem doesn't bother me, much.  I don't
really care whether people upgrade to each new release the moment it
comes out anyway.
Not to say that there aren't OTHER problems with the idea...

One could argue that its causing bad PR for postgres. I have seen several
parties planning to migrate away or not migrate to postgres because of
performance evaluations they made. With 7.4, 8.0 and 8.2. In 2010.

That's certainly true. It's clearly insane to benchmark with anything
other than the latest major release - on any product - if you want to
have any pretense of fairness. However, for users who have
applications that work and perform acceptably, I don't think it
benefits us to be too aggressive in trying to get them onto a later
major release. If we wanted to do that, we could maintain
back-branches for two years instead of five, but I don't think that
would be doing anyone any favors.

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small. I guess the brunt of the work would
actually fall on the packagers. It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Apr 21, 2011 at 11:48 AM, Andres Freund <andres@anarazel.de> wrote:

One could argue that its causing bad PR for postgres. I have seen several
parties planning to migrate away or not migrate to postgres because of
performance evaluations they made. With 7.4, 8.0 and 8.2. In 2010.

Well evaluating based on things past that can't be changed in the
absence of time machines doesn't offer us much guidance, as there
isn't anything that can be done in the present to fix such.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

[ man, this thread has totally outlived its title, could we change that?
I'll start with this subtopic ]

Robert Haas <robertmhaas@gmail.com> writes:

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small. I guess the brunt of the work would
actually fall on the packagers. It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

I agree that the incremental effort would not be so large, but what
makes you think that the situation will change given another year?
My expectation is that'd just mean people will do nothing about
migrating for a year longer.

More generally: it took a lot of argument to establish the current EOL
policy, and bending it the first time anyone feels any actual pain
will pretty much destroy the whole concept.

regards, tom lane

On Thursday, April 21, 2011 06:39:44 PM Robert Haas wrote:

On Thu, Apr 21, 2011 at 11:48 AM, Andres Freund <andres@anarazel.de> wrote:

On Thursday, April 21, 2011 05:43:16 PM Robert Haas wrote:

On Thu, Apr 21, 2011 at 11:37 AM, Ross J. Reedstrom <reedstrm@rice.edu>

wrote:

On Thu, Apr 21, 2011 at 11:16:45AM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

I agree. I am in favor of a shorter release cycle.

I'm not. I don't think there is any demand among *users* (as opposed
to developers) for more than one major PG release a year. It's hard
enough to get people to migrate that often.

In fact, I predict that the observed behavior would be for even more
end users to start skipping releases. Some already do - it's common
not to upgrade unless there's a feature you really need, but for
those who do stay on the 'current' upgrade path, you'll lose some who
can't afford to spend more than one integration-testing round a year.

Well, that aspect of the problem doesn't bother me, much. I don't
really care whether people upgrade to each new release the moment it
comes out anyway.
Not to say that there aren't OTHER problems with the idea...

One could argue that its causing bad PR for postgres. I have seen several
parties planning to migrate away or not migrate to postgres because of
performance evaluations they made. With 7.4, 8.0 and 8.2. In 2010.

That's certainly true. It's clearly insane to benchmark with anything
other than the latest major release - on any product - if you want to
have any pretense of fairness.

The usual argument against that is that $version is the only available on
$platform in version $version...

And I doubt that a higher number of new pg versions will lead to more
supported releases in distributions...

Andres

On Thu, Apr 21, 2011 at 5:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

[ man, this thread has totally outlived its title, could we change that?
 I'll start with this subtopic ]

Robert Haas <robertmhaas@gmail.com> writes:

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small.  I guess the brunt of the work would
actually fall on the packagers.  It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

I agree that the incremental effort would not be so large, but what
makes you think that the situation will change given another year?
My expectation is that'd just mean people will do nothing about
migrating for a year longer.

More generally: it took a lot of argument to establish the current EOL
policy, and bending it the first time anyone feels any actual pain
will pretty much destroy the whole concept.

It would also make at least one packager very unhappy as the 8.2
Windows build is by far the hardest and most time consuming to do and
I happen to know he's been counting the days until it goes.

More generally, keeping it for longer means we might end up supporting
6 major releases at once. That may not be so much work on a day to day
basis, but it adds up to a lot at release times, which was one of the
reasons why we agreed on the 5 year support window.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Apr 21, 2011 at 11:05 AM, Robert Haas <robertmhaas@gmail.com> wrote:

I agree.  I am in favor of a shorter release cycle.  But I think that
a shorter release cycle won't work well if there is still four month
long integration period at the end of each series of CommitFests.  The
problem is a bit circular here: because release cycles are long,
people really, really want to slip as much as possible in at the end.
But being under time pressure to get things committed results in a
higher bug count, which means more things that have to be fixed after
feature freeze, which translates into a long release cycle.

If we somehow were able to come up with a 6 week release cycle, we'd
still have the problem that there are features that take more than 6
weeks to integrate into a release. (HOT and SyncRep, I'm looking at
you!) Any such larger features would "blow this up," quite forcibly.

I don't think our release cycle is vastly too long; it takes enough
time to plan upgrades for systems that my colleagues at Afilias aren't
keen on using every PG release in production that comes out as it
stands now.

Peter Eisentraut points out that with the way things are, now, "...
you are left with all of about 20 days per year for discussion,
collaborative planning and coding. Which is obviously silly, which is
why the process breaks down."

I think the CommitFests have been a *super* tool for addressing such
problems as:
- patches getting lost
- getting review effort put onto the easier patches

But they aren't the only thing we conceptually need to have. For
tougher features, they're not great. And they're completely useless
at addressing discussions surrounding things we know we want done, but
don't have a strategy for yet. Those things aren't "patches", there's
nothing yet to commit.

My sense is that something else is needed as a process to help with
those "nebulous large changes." I'm not sure quite what it looks
like. Maybe there's some tooling that would be helpful, but we really
need some experimentation to figure out what the process should look
like.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

All,

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small. I guess the brunt of the work would
actually fall on the packagers. It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

Better that someone should just focus on whipping Robert's (or was it
Greg's?) replace-the-missing-casts package into shape as an extension.

I'm sure some kind of corporate sponsorship would be available for this
if someone wanted to work on it. Enough companies are facing this as
upgrade pain to want to fix it. If someone wants to work on it, let me
know; I'll start a fundraising campaign.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Thu, Apr 21, 2011 at 06:04:09PM +0100, Dave Page wrote:

On Thu, Apr 21, 2011 at 5:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

[ man, this thread has totally outlived its title, could we change that?
?I'll start with this subtopic ]

Robert Haas <robertmhaas@gmail.com> writes:

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small. ?I guess the brunt of the work would
actually fall on the packagers. ?It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

I agree that the incremental effort would not be so large, but what
makes you think that the situation will change given another year?
My expectation is that'd just mean people will do nothing about
migrating for a year longer.

More generally: it took a lot of argument to establish the current EOL
policy, and bending it the first time anyone feels any actual pain
will pretty much destroy the whole concept.

It would also make at least one packager very unhappy as the 8.2
Windows build is by far the hardest and most time consuming to do and
I happen to know he's been counting the days until it goes.

More generally, keeping it for longer means we might end up supporting
6 major releases at once. That may not be so much work on a day to day
basis, but it adds up to a lot at release times, which was one of the
reasons why we agreed on the 5 year support window.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

+1 for cutting the cord on 8.2. People using it still will need
to use the last release available, upgrade, or consult to have
a back-port/build made.

Regards,
Ken

On Thu, Apr 21, 2011 at 12:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

[ man, this thread has totally outlived its title, could we change that?
 I'll start with this subtopic ]

Robert Haas <robertmhaas@gmail.com> writes:

In fact, I've been wondering if we shouldn't consider extending the
support window for 8.2 past the currently-planned December 2011.
There seem to be quite a lot of people running that release precisely
because the casting changes in 8.3 were so painful, and I think the
incremental effort on our part to extend support for another year
would be reasonably small.  I guess the brunt of the work would
actually fall on the packagers.  It looks like we've done 5 point
releases of 8.2.x in the last year, so presumably if we did decide to
extend the EOL date by a year or so that's about how much incremental
effort would be needed.

I agree that the incremental effort would not be so large, but what
makes you think that the situation will change given another year?
My expectation is that'd just mean people will do nothing about
migrating for a year longer.

More generally: it took a lot of argument to establish the current EOL
policy, and bending it the first time anyone feels any actual pain
will pretty much destroy the whole concept.

I don't think that's quite a fair description of the proposal. I
don't think that having a general policy about EOL should preclude us
from making exceptions when there is some particularly compelling
reason to do so, and "it's particularly difficult to upgrade to
release X+1" seems to me to be something that might merit a bit of
consideration in that area. It is hard to imagine that 8.3, 8.4, 9.0,
or 9.1 could justify special treatment on similar grounds, nor did
7.4, 8.0, or 8.1, which we recently retired under this policy.

However, I can see that I'm way, way in the minority on this one, so
never mind! It was just a thought...

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Dave Page <dpage@pgadmin.org> writes:

On Thu, Apr 21, 2011 at 5:59 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I agree that the incremental effort would not be so large, but what
makes you think that the situation will change given another year?

It would also make at least one packager very unhappy as the 8.2
Windows build is by far the hardest and most time consuming to do and
I happen to know he's been counting the days until it goes.

Well, if we did extend support for 8.2, we could specifically exclude
Windows. But I'm still unclear on what would really be accomplished
by extending support for it. Sooner or later we have to get people
to migrate up from it, and I see no reason to think that supporting
it for just a year more will change anything.

regards, tom lane

Josh Berkus <josh@agliodbs.com> writes:

Better that someone should just focus on whipping Robert's (or was it
Greg's?) replace-the-missing-casts package into shape as an extension.

I think Peter originated that, actually. My recollection is that there
didn't seem to be any way to extend it to a complete solution, and
besides which it's really a crutch to avoid fixing bugs in your
application. Still, if someone does want to expend more work on it
I wouldn't object.

regards, tom lane

On Thu, Apr 21, 2011 at 12:35 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

But I'm still unclear on what would really be accomplished
by extending support for it.  Sooner or later we have to get people
to migrate up from it, and I see no reason to think that supporting
it for just a year more will change anything.

And people is more likely to migrate if they see some kind of hard
line, specially when migrate means a lot of work.
Actually, someone i know is targeting to migrate before the EOL, just
because the EOL exists.

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL

On Thu, Apr 21, 2011 at 6:18 PM, Robert Haas <robertmhaas@gmail.com> wrote:

However, I can see that I'm way, way in the minority on this one, so
never mind!  It was just a thought...

Fwiw I would have agreed with you on the basic question. Just because
we've said that users can count on N years of support doesn't mean
there's anything binding us to *not* support things for N+x years. The
argument that we should cut refuse to back-patch security fixes and
bug fixes that we could handle without much effort to versions that
users are using just because we think we know better than them and
know they should upgrade is a bad path imho.

However your theory was all predicated on the idea that supporting 8.2
was not much incremental effort and Dave said that's not true so this
is all moot. Doing it Windows-excluded seems not worth the effort ---
unless... what version of Postgres was shipped in the last supported
releases of major distributions? I think it was 8.1 in Ubuntu Hardy
and 8.4 in Ubuntu Lucid so that's irrelevant. What about Redhat and
Debian?

--
greg

Greg Stark <gsstark@mit.edu> writes:

Fwiw I would have agreed with you on the basic question. Just because
we've said that users can count on N years of support doesn't mean
there's anything binding us to *not* support things for N+x years.

Certainly. The question is what's the point --- and perhaps even more
to the point, if we extend 8.2 support, when are we going to stop
extending it?

However your theory was all predicated on the idea that supporting 8.2
was not much incremental effort and Dave said that's not true so this
is all moot. Doing it Windows-excluded seems not worth the effort ---
unless... what version of Postgres was shipped in the last supported
releases of major distributions? I think it was 8.1 in Ubuntu Hardy
and 8.4 in Ubuntu Lucid so that's irrelevant. What about Redhat and
Debian?

So far as Red Hat is concerned, neither 8.2 nor 8.3 are of any interest
whatsoever. I'm still on the hook for 7.4 and 8.1 to some extent, but
only very severe security issues are likely to be considered for those.

regards, tom lane

On 04/21/2011 05:17 PM, Greg Stark wrote:

what version of Postgres was shipped in the last supported
releases of major distributions? I think it was 8.1 in Ubuntu Hardy
and 8.4 in Ubuntu Lucid so that's irrelevant. What about Redhat and
Debian?

IIRC RedHat has a ten year EOL policy, so what they have shipped in old
releases should not really bind us. In any case, our EOL policy only
affects what formal releases we make. We can commit fixes to branches
past their EOL date, and IIRC Tom did this not long ago.

cheers

andrew

On tor, 2011-04-21 at 13:39 -0400, Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Better that someone should just focus on whipping Robert's (or was it
Greg's?) replace-the-missing-casts package into shape as an extension.

I think Peter originated that, actually. My recollection is that there
didn't seem to be any way to extend it to a complete solution, and
besides which it's really a crutch to avoid fixing bugs in your
application. Still, if someone does want to expend more work on it
I wouldn't object.

http://petereisentraut.blogspot.com/2008/03/readding-implicit-casts-in-postgresql.html

There are some problems if you just add *all* the casts back without
thinking. In particular, the || appears to be causing problems.

But other than those few specific cases, that tool kit fixes all known
problems. So anyone who is willing to spend more than zero minutes on
planning and executing a major version upgrade shouldn't really have any
problems with this aspect.

On tor, 2011-04-21 at 22:17 +0100, Greg Stark wrote:

However your theory was all predicated on the idea that supporting 8.2
was not much incremental effort and Dave said that's not true so this
is all moot. Doing it Windows-excluded seems not worth the effort ---
unless... what version of Postgres was shipped in the last supported
releases of major distributions? I think it was 8.1 in Ubuntu Hardy
and 8.4 in Ubuntu Lucid so that's irrelevant. What about Redhat and
Debian?

Debian: 8.3 in oldstable (<1 year left), 8.4 in stable, probably 9.1 in
next