creating a cluster

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:

Does anyone have any new ways to create clusters without using initdb
or bootstrap mode? I need to be able to create one without those 2
things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Find a bug in a program, and fix it, and the program will work today.
Show the program how to find and fix a bug, and the program
will work forever" (Oliver Silfridge)

Alexander Cohen <alex@toomuchspace.com> writes:

Does anyone have any new ways to create clusters without using initdb
or bootstrap mode? I need to be able to create one without those 2
things. Any ideas?

Perhaps you should explain *why* you think you need this?

regards, tom lane

Alvaro Herrera wrote:

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:

Does anyone have any new ways to create clusters without using initdb
or bootstrap mode? I need to be able to create one without those 2
things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

Btw, I've been doing this for a binary distribution on Windows (Cygwin)
and Linux. Primarily because initdb-ing + doing a bunch of SQL commands
to the db takes a long time on Cygwin. Seems fine so far.

--
dave

On Jun 23, 2004, at 10:18 AM, David Garamond wrote:

Alvaro Herrera wrote:

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:

Does anyone have any new ways to create clusters without using
initdb or bootstrap mode? I need to be able to create one without
those 2 things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

Btw, I've been doing this for a binary distribution on Windows
(Cygwin) and Linux. Primarily because initdb-ing + doing a bunch of
SQL commands to the db takes a long time on Cygwin. Seems fine so far.

And how do you take care of users for your distribution. If you created
the cluster on your computer, does it not have your user name as the
main root user? That needs to be changed when copying over the cluster,
how do i that?

Alex

David Garamond <lists@zara.6.isreserved.com> writes:

Alvaro Herrera wrote:

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:

Does anyone have any new ways to create clusters without using initdb
or bootstrap mode? I need to be able to create one without those 2
things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

Btw, I've been doing this for a binary distribution on Windows (Cygwin)
and Linux.

Yeah, that would work fine as long as the "somewhere else" is using an
identical Postgres build. I found out in off-list conversation that
Alexander wants to build a hacked-up version of Postgres with all
bootstrap code removed (and, I suppose, a bunch of other changes too).
Seems to me that file-level compatibility would be difficult to
guarantee under such circumstances, so I told him he ought to put back
the bootstrap support ... it's not like it's large ...

regards, tom lane

On Jun 23, 2004, at 11:36 AM, Tom Lane wrote:

David Garamond <lists@zara.6.isreserved.com> writes:

Alvaro Herrera wrote:

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:

Does anyone have any new ways to create clusters without using
initdb
or bootstrap mode? I need to be able to create one without those 2
things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

Btw, I've been doing this for a binary distribution on Windows
(Cygwin)
and Linux.

Yeah, that would work fine as long as the "somewhere else" is using an
identical Postgres build. I found out in off-list conversation that
Alexander wants to build a hacked-up version of Postgres with all
bootstrap code removed (and, I suppose, a bunch of other changes too).
Seems to me that file-level compatibility would be difficult to
guarantee under such circumstances, so I told him he ought to put back
the bootstrap support ... it's not like it's large ...

For the meantime, i ended up compiling a normal version of postgres and
using that with initdb, then switching it over to my "hacked-up"
version. It works, and thats all i need for now!

Alex

Hi all,

Can anyone tell me if postgresql has problems with xeon processors?
If so, there is any fix or project of fix it?

Thanx in advance,

Jaime Casanova

---------------------------------
Do You Yahoo!?
Todo lo que quieres saber de Estados Unidos, Am�rica Latina y el resto del Mundo.
Vis�ta Yahoo! Noticias.

Hello,

I seem to recall that HyperThreading and PostgreSQL != good stuff...
There was a whole bunch of stuff recently on this... google the archives.

Sincerely,

Joshua D. Drake

Jaime Casanova wrote:

Hi all,

Can anyone tell me if postgresql has problems with xeon processors?
If so, there is any fix or project of fix it?

Thanx in advance,

Jaime Casanova

------------------------------------------------------------------------
*Do You Yahoo!?*
<http://espanol.yahoo.com/mail_tagline/*http://espanol.news.yahoo.com&gt;
Todo lo que quieres saber de Estados Unidos, América Latina y el resto
del Mundo.
Visíta Yahoo! Noticias
<http://espanol.yahoo.com/mail_tagline/*http://espanol.news.yahoo.com&gt;.

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

thanx

"Joshua D. Drake" <jd@commandprompt.com> wrote:Hello,

I seem to recall that HyperThreading and PostgreSQL != good stuff...
There was a whole bunch of stuff recently on this... google the archives.

Sincerely,

Joshua D. Drake

Jaime Casanova wrote:

Hi all,

Can anyone tell me if postgresql has problems with xeon processors?
If so, there is any fix or project of fix it?

Thanx in advance,

Jaime Casanova

------------------------------------------------------------------------
*Do You Yahoo!?*

Todo lo que quieres saber de Estados Unidos, Am�rica Latina y el resto
del Mundo.
Vis�ta Yahoo! Noticias
.

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL
begin:vcard
fn:Joshua D. Drake
n:Drake;Joshua D.
org:Command Prompt, Inc.
adr:;;PO Box 215;Cascade Locks;Oregon;97014;USA
email;internet:jd@commandprompt.com
title:Consultant
tel;work:503-667-4564
tel;fax:503-210-0034
note:Command Prompt, Inc. is the largest and oldest US based commercial PostgreSQL support provider. We provide the only commercially viable integrated PostgreSQL replication solution, but also custom programming, and support. We authored the book Practical PostgreSQL, the procedural language plPHP, and adding trigger capability to plPerl.
x-mozilla-html:FALSE
url:http://www.commandprompt.com/
version:2.1
end:vcard

---------------------------------
Do You Yahoo!?
Todo lo que quieres saber de Estados Unidos, Am�rica Latina y el resto del Mundo.
Vis�ta Yahoo! Noticias.

Centuries ago, Nostradamus foresaw when systemguards@yahoo.com (Jaime Casanova) would write:

Can anyone tell me if postgresql has problems with xeon processors?

If so, there is any fix or project of fix it?

Well, there's a known issue that IA-32 systems with more than 4GB of
memory use an extension known as "PAE" to bank-switch between the
banks of memory.

Any time you switch banks, there's a fair little bit of work to be
done. That includes multitasking systems that need to context switch
a few thousand times per second.

The "fix" for this problem is to rewrite all of your applications so
that they become conscious of which bits of memory they're using so
they can tune their own behaviour. This, of course, requires
discarding useful notions such as "virtual memory" that are _assumed_
by most modern operating systems.

The fix is to buy hardware that hasn't been hacked up so badly.
--
select 'cbbrowne' || '@' || 'ntlug.org';
http://www3.sympatico.ca/cbbrowne/lsf.html
"There is something in the lecture course which may not have been
visible so far, which is reality ..." -- Arthur Norman

On Fri, 2004-06-25 at 14:13, Jaime Casanova wrote:

Hi all,

Can anyone tell me if postgresql has problems with xeon processors?
If so, there is any fix or project of fix it?

To PostgreSQL, there's no difference between a dual CPU machine with no
hyperthreading, and a single CPU machine with hyperthreading.

HOWEVER, there have been some issues with certain Operating Systems
running slower with hyperthreading enabled than without it. Late model
Linux kernels (2.6) seem to have gotten enough logic into the scheduler
to get good performance on a hyperthreaded system.

On Sat, Jun 26, 2004 at 07:31:59PM -0600, Scott Marlowe wrote:

On Fri, 2004-06-25 at 14:13, Jaime Casanova wrote:

Hi all,

Can anyone tell me if postgresql has problems with xeon processors?
If so, there is any fix or project of fix it?

To PostgreSQL, there's no difference between a dual CPU machine with no
hyperthreading, and a single CPU machine with hyperthreading.

At some point there was trouble with spinlocks on some of the newer
Xeons (maybe those with hyperthreading?). I think there was a good deal
of discussion and resulting development because of that. According to
my archives it seems to be around december 2003. There's even a CVS
entry, listed by cvs2cl as

2003-12-27 17:58 tgl

* src/: backend/storage/lmgr/s_lock.c, include/storage/s_lock.h:
Improve spinlock code for recent x86 processors: insert a PAUSE
instruction in the s_lock() wait loop, and use test before
test-and-set in TAS() macro to avoid unnecessary bus traffic.
Patch from Manfred Spraul, reworked a bit by Tom.

Not sure if this made the 7.4.3 release ...

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"You knock on that door or the sun will be shining on places inside you
that the sun doesn't usually shine" (en Death: "The High Cost of Living")

Christopher Browne <cbbrowne@acm.org> writes:

Centuries ago, Nostradamus foresaw when systemguards@yahoo.com (Jaime Casanova) would write:

Can anyone tell me if postgresql has problems with xeon processors?

If so, there is any fix or project of fix it?

Well, there's a known issue that IA-32 systems with more than 4GB of
memory use an extension known as "PAE" to bank-switch between the
banks of memory.

AIUI, it's not really "bank switching" in the old 8-bit sense.
Rather, there is a big linear 36-bit physical address space, and the
processor's page tables have been extended so they can point to any
page in this space. Any one process still sees a 4GB (32-bit) address
space since that's how big the registers are.

Any time you switch banks, there's a fair little bit of work to be
done. That includes multitasking systems that need to context switch
a few thousand times per second.

I don't think this is any more overhead than a "normal" context
switch--cache misses, TLB flush etc.

The "fix" for this problem is to rewrite all of your applications so
that they become conscious of which bits of memory they're using so
they can tune their own behaviour. This, of course, requires
discarding useful notions such as "virtual memory" that are _assumed_
by most modern operating systems.

This is only if you need to address more than 32-bits worth of address
in a single process, which is not always the case on server-class
systems (on scientific/numerical workloads, it's often a big win). In
which case you are certainly right:

The fix is to buy hardware that hasn't been hacked up so badly.

64-bit systems get cheaper all the time... :)

-Doug

Christopher Browne wrote:

The "fix" for this problem is to rewrite all of your applications so
that they become conscious of which bits of memory they're using so
they can tune their own behaviour. This, of course, requires
discarding useful notions such as "virtual memory" that are _assumed_
by most modern operating systems.

This is misleading: PAE means that a 32-bit cpu can have more that 4 GB
physical memory. Each process can map at most 4 (in reality: ~2) GB memory.
Many databases manage their own, huge buffer pool and read/write the
database tables with O_DIRECT. These apps must support buffer pools > 2
GB, which requires some work. Linux and Solaris contain a special
syscall that helps Oracle to manage it's buffer pool for such setups
(remap_page_rage()).
OTHO postgres has a small user space buffer pool, the majority of the
file buffers are handled by OS. Thus no changes are required inside
postgres for PAE, all it needs is an OS that support PAE for the buffer
pool.

Regarding hyperthreading: I'm aware of two changes:
- busy loops must contain PAUSE instructions. Postgres does that.
- virtual aliases should be avoided: If two processes access memory at
the same virtual address, then this can cause cache collisions and then
misses. I think this is handled by the C library by randomizing the
return addresses of malloc() and Intel mitigated the issue by improving
the cache.

--
Manfred