cvs head? initdb?

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.
--
select 'cbbrowne' || '@' || 'libertyrms.info';
<http://dev6.int.libertyrms.com/&gt;
Christopher Browne
(416) 646 3304 x124 (land)

They are no longer in sync. Jan has started committing some of his ARC
work (though I think it might be backed out currently) and Tom has
committed his cross data-type index work. Both have reported errors with
initdb so it might not be completely stable right now. generally it
should be, but not always, you can report errors if you want or wait
until they sort it out.

You should still be able to get an RC2 build with the appropriate tags
applied. HTH

Robert Treat

On Wed, 2003-11-12 at 21:04, elein wrote:

What is the status of CVS head? Isn't it in sync
with 7.4.RC2? I just upgraded from CVS and rebuilt
clean and initdb now gives this lovely informative
initdb failed message.

I haven't had trouble previously with cvs head or
other versions.

$ sudo make install
...
PostgreSQL installation complete.
/local/src/postgres-cvs/pgsql[elein:cookie]
$ type initdb
initdb is hashed (/local/pghead/bin/initdb)
/local/src/postgres-cvs/pgsql[elein:cookie]
$ initdb
The database cluster will be initialized with locales
COLLATE: C
CTYPE: en_US
MESSAGES: en_US
MONETARY: en_US
NUMERIC: en_US
TIME: en_US
initdb: failed
initdb: removing contents of data directory "/local/pghead/data"
creating directory "/local/pghead/data" ... /local/src/postgres-cvs/pgsql[elein:cookie]

--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

and the problem seen here with the order of messages sent by initdb has
been fixed by Tom in the head branch (which is the only place it would
have occurred).

cheers

andrew

Robert Treat wrote:

Christopher Browne <cbbrowne@libertyrms.info> writes:

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

It's back in again, and appears to work now (at least the regression
tests pass ... no idea about performance ...)

To answer Elein's question, HEAD is no longer catalog-compatible with
7.4 because of the cross-type indexing work I committed a couple days
ago.

regards, tom lane

Christopher Browne wrote:

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

and starting postmaster with -d1

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

It's back in again, and appears to work now (at least the regression
tests pass ... no idea about performance ...)

I actually managed to hang my process when I first did a make check.
After cleaning all that up, I did a make installcheck and that was fine.
Then I did a make check again and this time it didn't hang. It hung
on select_views/portals_p2, but I haven't been able to reproduce it...

Chris

Jan Wieck wrote:

Christopher Browne wrote:

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

and starting postmaster with -d1

It is not on by default?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Jan Wieck wrote:

Christopher Browne wrote:

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

and starting postmaster with -d1

It is not on by default?

Sure not. Why would someone turn on debug messages by default?

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

On Fri, 2003-11-14 at 10:32, Jan Wieck wrote:

Bruce Momjian wrote:

Jan Wieck wrote:

Christopher Browne wrote:

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

and starting postmaster with -d1

It is not on by default?

Sure not. Why would someone turn on debug messages by default?

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

Robert Treat wrote:

On Fri, 2003-11-14 at 10:32, Jan Wieck wrote:

Bruce Momjian wrote:

Jan Wieck wrote:

Christopher Browne wrote:

elein@varlena.com (elein) writes:

What is the status of CVS head? Isn't it in sync with 7.4.RC2? I
just upgraded from CVS and rebuilt clean and initdb now gives this
lovely informative initdb failed message.

No, I believe that they have started applying 7.5 patches, notably
including reimplementing initdb in C.

Jan checked in the ARC changes last night, and he and Tom ran into
some problems, leading to that being pulled back out, while he
revisits the code.

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

and starting postmaster with -d1

It is not on by default?

Sure not. Why would someone turn on debug messages by default?

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies. Then
again, who will really test this and try to tune ALL of them to find the
best choice, and is this possible at all given that all databases under
one postmaster share the same buffer pool?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

The above line is the one I was asking about? Does that just control
debug output interval? If so, it should be named more appropriately.

and starting postmaster with -d1

It is not on by default?

Sure not. Why would someone turn on debug messages by default?

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

No, I don't see much value to this and would encourage people to play
with something that probably is is of little value.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Jan Wieck wrote:

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

The above line is the one I was asking about? Does that just control
debug output interval? If so, it should be named more appropriately.

Yes, all it controls is the DEBUG1 output eveny N seconds. What is
inappropriate about it? It is exactly that, it shows the sizes and
recent hit rates for the 4 different CDB queues of the strategy. Have a
better one?

The output is not really of any use yet. The background writer though
will add a third line to this output, the number of clean buffers in
front of the T1 and T2 queues. That will be the #1 tool for adjusting
the background writer parameters so that it keeps the cache clean enough
that backends don't need to write, and dirty enough that high frequently
used blocks don't get written too often.

and starting postmaster with -d1

It is not on by default?

Sure not. Why would someone turn on debug messages by default?

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

No, I don't see much value to this and would encourage people to play
with something that probably is is of little value.

Me neither, just wanted to have asked.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

Bruce Momjian wrote:

Jan Wieck wrote:

Yeah, there was a problem with *extreme* sharing ... the code tried to
use the same buffer for multiple disk blocks at the same time, and
somehow the backends did not agree on the correct content. But it's
fixed and back in. You can see ARC working by setting

buffer_strategy_status_interval = 10 # seconds

The above line is the one I was asking about? Does that just control
debug output interval? If so, it should be named more appropriately.

Yes, all it controls is the DEBUG1 output eveny N seconds. What is
inappropriate about it? It is exactly that, it shows the sizes and
recent hit rates for the 4 different CDB queues of the strategy. Have a
better one?

I figured it should begin with debug_ or log_, maybe:

debug_shared_buffers = 10 # seconds

I don't think "interval" is needed and it clearly links to the
shared_buffers parameter.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Jan Wieck <JanWieck@yahoo.com> writes:

Robert Treat wrote:

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

To me, the question is whether it's worth the additional complexity
for users and administrators, and to a lesser extent the code
complexity. (I think the answer is "no")

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies.

Granted -- but IMHO it would be better to concentrate on making sure
that ARC adapts to any access pattern so that the set of access
patterns where you _really want_ LRU is a small as possible, if not
empty.

-Neil

On Friday 14 November 2003 12:03, Jan Wieck wrote:

Robert Treat wrote:

On Fri, 2003-11-14 at 10:32, Jan Wieck wrote:

Or did you mean ARC itself? Since it replaced the old LRU code, it is
the only choice you have now. Which sort of raises the question if we
would want to have multiple choices, like a config option

buffer_replacement_strategy = lru|lru2|arc

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies. Then
again, who will really test this and try to tune ALL of them to find the
best choice, and is this possible at all given that all databases under
one postmaster share the same buffer pool?

I could see people like the OSDB folks or some of the folks on -performance at
least doing some testing against the different backends. Probably not
extensive but I bet enough to see if there is a clear winner for some types
of work. You might not be able to test them in parallel, but certainly you
could serially.

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

On Friday 14 November 2003 14:23, Neil Conway wrote:

Jan Wieck <JanWieck@yahoo.com> writes:

Robert Treat wrote:

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

To me, the question is whether it's worth the additional complexity
for users and administrators, and to a lesser extent the code
complexity. (I think the answer is "no")

i don't think the complexity for users is that high... depending on
implementation. assuming we end up with clearly definable cases where one is
more usefull than the other. (which seems just as likely not to happen as to
happen)

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies.

Granted -- but IMHO it would be better to concentrate on making sure
that ARC adapts to any access pattern so that the set of access
patterns where you _really want_ LRU is a small as possible, if not
empty.

but how do you test this if you cant run them both against each other to
compare? (initally running vs 7.4 does tell you something, but even now, 7.5
improved cross datatype index improvments could skew the results of any
comparisons)

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

Our philosophy has never been to give people configuration options just
in case they might be valuable to them. If we did that, we would be
like Oracle.

We give config options only if we can't decide the best default. For
testing, you can have an #ifdef and we can test it ourselves. If we can
find the best default, no need to burden the user with it.

---------------------------------------------------------------------------

Robert Treat wrote:

On Friday 14 November 2003 14:23, Neil Conway wrote:

Jan Wieck <JanWieck@yahoo.com> writes:

Robert Treat wrote:

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

To me, the question is whether it's worth the additional complexity
for users and administrators, and to a lesser extent the code
complexity. (I think the answer is "no")

i don't think the complexity for users is that high... depending on
implementation. assuming we end up with clearly definable cases where one is
more usefull than the other. (which seems just as likely not to happen as to
happen)

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies.

Granted -- but IMHO it would be better to concentrate on making sure
that ARC adapts to any access pattern so that the set of access
patterns where you _really want_ LRU is a small as possible, if not
empty.

but how do you test this if you cant run them both against each other to
compare? (initally running vs 7.4 does tell you something, but even now, 7.5
improved cross datatype index improvments could skew the results of any
comparisons)

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Our philosophy has never been to give people configuration options just
in case they might be valuable to them. If we did that, we would be
like Oracle.

We give config options only if we can't decide the best default. For
testing, you can have an #ifdef and we can test it ourselves. If we can
find the best default, no need to burden the user with it.

Please tell me what is the best default cache replacement strategy for
PostgreSQL. Comparing the old LRU to the new ARC the winner is clear.
But what about MQ, 2Q, FBR, LFRU, LRU2 LFU and MRU?

Jan

---------------------------------------------------------------------------

Robert Treat wrote:

On Friday 14 November 2003 14:23, Neil Conway wrote:

Jan Wieck <JanWieck@yahoo.com> writes:

Robert Treat wrote:

people would always want to have those choices (especially for doing
development/testing/benchmarking between the different methods) the
question is is it worth the effort to give people those options?

To me, the question is whether it's worth the additional complexity
for users and administrators, and to a lesser extent the code
complexity. (I think the answer is "no")

i don't think the complexity for users is that high... depending on
implementation. assuming we end up with clearly definable cases where one is
more usefull than the other. (which seems just as likely not to happen as to
happen)

And in the case of the cache strategy, the point is that different
access patterns might be served better by different strategies.

Granted -- but IMHO it would be better to concentrate on making sure
that ARC adapts to any access pattern so that the set of access
patterns where you _really want_ LRU is a small as possible, if not
empty.

but how do you test this if you cant run them both against each other to
compare? (initally running vs 7.4 does tell you something, but even now, 7.5
improved cross datatype index improvments could skew the results of any
comparisons)

Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

Bruce Momjian wrote:

Our philosophy has never been to give people configuration options just
in case they might be valuable to them. If we did that, we would be
like Oracle.

We give config options only if we can't decide the best default. For
testing, you can have an #ifdef and we can test it ourselves. If we can
find the best default, no need to burden the user with it.

Please tell me what is the best default cache replacement strategy for
PostgreSQL. Comparing the old LRU to the new ARC the winner is clear.
But what about MQ, 2Q, FBR, LFRU, LRU2 LFU and MRU?

Is there a BJM? I would like to have one named after me too! :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

I figured it should begin with debug_ or log_, maybe:

debug_shared_buffers = 10 # seconds

If it's just that and since nobody else seemed to care ... changed.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Robert Treat wrote:

but how do you test this if you cant run them both against each other to
compare? (initally running vs 7.4 does tell you something, but even now, 7.5
improved cross datatype index improvments could skew the results of any
comparisons)

Right. But with the current two candidates (LRU and ARC) in the field
only, there isn't much to compare. If someone wants to implement another
algorithm, I will be happy to put the required switchboard (something
like the jump table in smgr) into the code. Right now I see much more
bang for the buck in creating the background writer.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

Robert Treat wrote:

but how do you test this if you cant run them both against each other to
compare? (initally running vs 7.4 does tell you something, but even now, 7.5
improved cross datatype index improvments could skew the results of any
comparisons)

Right. But with the current two candidates (LRU and ARC) in the field
only, there isn't much to compare. If someone wants to implement another
algorithm, I will be happy to put the required switchboard (something
like the jump table in smgr) into the code. Right now I see much more
bang for the buck in creating the background writer.

Better Journalistic Manipulator ==> BJM. :-)

_B_ruce _J_. _M_omjian.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073