WIN32 native ... lets start?!?

Actually, take a look at the thread starting at:

http://archives.postgresql.org/pgsql-hackers/2002-05/msg00665.php

Right now, IMHO, the big show stopper is passing global variables to the
child processes in Windows ... the above thread talks about a method of
pulling together the global variables *cleanly* that Tom seems to feel
wouldn't add much in the way of long term maintenance headaches ... *and*,
as I understand it, would provide us with a means to use threading in
future developments if deemed appropriate ...

From what I read by those 'in the know' about Windows programming, if we
could centralize the global variables somewhat, using CreateProcess in
Windows shouldn't be a big deal, eliminiating the whole fork() headache
...

On Thu, 16 May 2002, Joerg Hessdoerfer wrote:

On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:

So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.

Getting it to compile and then "somehow" run on MinGW seems a good first
step on road to full native Win32 PG.

----------
Hannu

On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:

So, my route would be to get it to run *somehow* without paying
attention to speed and not to change much of the existing code,
THEN see how we could get rid of fork() on windows.

What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying the memory allocations.
How fast that would be regarding the context switches, i don't know, but i'm willing to experiment some to see how feesible this is...

Anyone tried this before?

Magnus

On Thursday 16 May 2002 22:10, you wrote:
[...]

What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying
the memory allocations. How fast that would be regarding the context
switches, i don't know, but i'm willing to experiment some to see how
feesible this is...

Anyone tried this before?

Magnus

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on windows, it's
creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg.hessdoerfer@sea-gmbh.com
WWW: http://www.sea-gmbh.com

Joerg Hessdoerfer <Joerg.Hessdoerfer@sea-gmbh.com> wrote:
[snip]

The problem is not the fork() call itself, this has been done (MinGW
and cygwin I know of, possibly others) but the speed of fork() on
windows, it's creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

I'm not so familiar with the win32 kernel mode stuff.
But i've seen programs using .vxd (kernelmode, ring X ?) helpers for getting more privileges, maybe cross process ones.
Well, i'll look into this sometime if it's possible to reduce the context switches by going vxd.
There must be some way to read protection of the pages and map them as COW or RO in the new process to get rid of much of the copy, but then again, we're talking microsoft here :)
I once did a .exe loader that used the MapViewOfFile (win32 mmap) of the .exe itself to accomplish shared loadable modules that worked on x86 linux and win32 without recompile (might be something like the XFree86 binary gfx card drivers).

Good luck on the IPC work!

Magnus

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Chris

On Fri, 17 May 2002, Christopher Kings-Lynne wrote:

Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

"Marc G. Fournier" <scrappy@hub.org> writes:

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

On Friday 17 May 2002 22:16, you wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

Yes, indeed. I would also like to discuss matters on this list, as one get's
a 'heads up' from people in the know much easier.

BTW, I'm in the process of doing the 'really only what is necessary for pg'
ipc-stuff, and was wondering if anybody already did some configuration of the
source tree towards MinGW?? How should we go about that? I would rather like
not using cygwin's sh for that ;-), and we have no 'ln' !!

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg.hessdoerfer@sea-gmbh.com
WWW: http://www.sea-gmbh.com

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Tuesday, June 04, 2002 9:34 PM
To: PostgreSQL-development
Subject: [HACKERS] Roadmap for a Win32 port

OK, I think I am now caught up on the Win32/cygwin
discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies
working on Win32
ports of PostgreSQL. We always said there would not be
PostgreSQL forks
if we were doing our job to meet user needs. Well,
obviously, a number
of groups see a need for a better Win32 port and we aren't
meeting that
need, so they are. I believe this is one of the few cases
where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users.
We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

Nothing else. It is better than any commercial tools in current use.
An excellent piece of work.

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

Do not try to do a fork() on Win32. The one at PW32 is better, but
still awful. Win32 just does not have fascilities for fork().

If you use Cygwin, it will kill the project for commercial use (at least
for many institutions). That's fine, but it will become an academic
exercise instead of a viable commercial tool. If they are comfortable
in that [Cygwin] environment, it makes no sense to use Cygwin instead of
Redhat. The Redhat version will fork() 100 times faster. After all, if
they are going to use unix tools in a unix interface with Unix scripts
you might as well use UNIX. And Cygwin requires a license for
commercial use.
http://cygwin.com/licensing.html

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

CreateProcess() works well for Win32. That is the approach that we used
and also the approach used by the Japanese team.
It is very simple. Simply do a create process call and then perform the
same operations that were done up to that point. It isn't difficult.
Threading is another possibility. I think create process is better,
because you can clone the rights of the one who attaches for the spawned
server (if you want to do that).

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

We wrote our own from scratch. Cygwin will kill it. If there is a
MinGW version it might be OK, but if MinGW is GPL, that will kill it.
Have a look at ACE:
http://www.cs.wustl.edu/~schmidt/ACE.html
Their license is on the same level as a BSD license. Now, they use C++,
but you can always write:
extern "C" {
}
wrappers for stuff and keep PostgreSQL itself in pure, vanilla C. GCC
does come with a C++ compiler, so it isn't going to cut anyone off.

ENVIRONMENT

Lots of our code requires a Unix shell and utilities. Will
we continue
using cygwin for this?

We wrote our own utilities from scratch (e.g. initdb). The Japanese
group that did the port did the same thing.

--------------------------------------------------------------
-------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

If you want a roadmap, the Japanese group laid it out for you. They
did the exact same steps as we did. Now, I don't know if we will be
able to contribute or not (it is very much up in the air). And we had
to do a lot of hacking of the source, so you might not want it if we
volunteered.

Suggestion:
Ask the Japanese group if they would like to post their changes back or
expose them so that the programming team can get ideas form it.

I actually like what they did better than what we did (A giant DLL and
all the binaries are microscopic -- it was how I suggested to do it here
but it was vetoed).

Anyway, here is a roadmap laid out for you exactly. Just do what it
says and you will be fine:
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html

Look at where it says "Gists for patch" and do that.

On Wed, 2002-06-05 at 16:33, Bruce Momjian wrote:

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 853-3000
+  If your life is a hard drive,     |  830 Blythe Avenue
+  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

regards

Mark

Dan,

The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.

On Tue, Jun 04, 2002 at 10:02:14PM -0700, Dann Corbit wrote:

And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html

The above is not necessarily true:

Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.

Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.

Jason

Jason Tishler wrote:

Dan,

The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.

On Tue, Jun 04, 2002 at 10:02:14PM -0700, Dann Corbit wrote:

And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html

The above is not necessarily true:

Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.

Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.

So apps written using client libraries are BSD, while server-side
changes would have to release source. Makes sense, though we have never
had this distinction before. I assume plpgsql stored procedures would
have be open source, but of course those are stored in plaintext on the
server so that isn't a problem. If companies created custom C stored
procedures, those would have to be open source if using cygwin.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mark kirkwood wrote:

Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

Absolutely worth seeing how MySQL does it. They use cygwin, and I
assume they aren't seeing the fork() issue because they are threaded,
and perhaps they don't use SysV IPC like we do.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

-- igor

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Tuesday, June 04, 2002 11:33 PM
Subject: [HACKERS] Roadmap for a Win32 port

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

--------------------------------------------------------------------------

-

Igor Kovalenko wrote:

I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

...

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

I would think that we would build on our strengths of having a fork/exec
model for separate clients. A threaded model *could* benefit individual
clients who are doing queries on multiprocessor servers, and I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well supported
targets. If it came for free then great, but if it came with a high cost
then the choice is not as obvious. It is also not a *requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

- Thomas

-----Original Message-----
From: Thomas Lockhart [mailto:thomas@fourpalms.org]
Sent: Wednesday, June 05, 2002 3:03 PM
To: Bruce Momjian
Cc: Igor Kovalenko; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

...

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a

web backend

where many sessions are a single query, people may want to

give up the

stability of fork() and go with threads, even on Unix.

I would think that we would build on our strengths of having
a fork/exec
model for separate clients. A threaded model *could* benefit
individual
clients who are doing queries on multiprocessor servers, and
I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well
supported
targets. If it came for free then great, but if it came with
a high cost
then the choice is not as obvious. It is also not a
*requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

Notion:
Have one version do both. Your server can fork(), and your sever can
thread. It can fork() and thread, it can fork() or thread.

That gives the best of all worlds. One client who has his attachments
to a database all setup might want to do a bunch of similar queries.
Hence a threaded model is nice.

A server may be set up to clone the rights of the attaching process for
security reasons. Then you launch a new server with fork().

...

Notion:
Have one version do both. Your server can fork(), and your sever can
thread. It can fork() and thread, it can fork() or thread.
That gives the best of all worlds. One client who has his attachments
to a database all setup might want to do a bunch of similar queries.
Hence a threaded model is nice.
A server may be set up to clone the rights of the attaching process for
security reasons. Then you launch a new server with fork().

Right. If/when that is possible then let's do it, as long as the cost is
not too high. But the intermediate steps are a possibility also, and are
not precluded from discussion.

This will all work out as a *convergence* of interests imho. And there
is no great identifiable benefit for our current crop of platforms for
going to a threaded model *unless* that enables queries for a single
client to execute in parallel (all imho of course ;).

So our convergence of interests for all platforms is in enabling
threading for these two purposes, and focusing on enabling the
multithreaded single client *first* means that the current crop of
clients don't have to accept all negatives while we start on the road to
better support of Win32 machines.

- Thomas

One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model. Please see:

http://state-threads.sourceforge.net/docs/st.html

Thus, on platforms where it can be supported, we should keep in mind that a
hybrid multiprocess/multithreaded postgresql might be the fastest
solution...

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Igor Kovalenko" <Igor.Kovalenko@motorola.com>
Cc: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

Igor Kovalenko wrote:

I might be naive here, but would not proper threading model remove the

need

for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32

threads...

I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the

bright

side, performance should be better on some platforms (note however,

Apache

group still can't come up with implementation of threaded model which

would

provide better performance than forked or other models). The need to

deal

with possibility of 'alien' postmaster running along with orphaned

backends

would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things

will

probably have to be revamped. But in long term this is probably best way

if

you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would

be

something like a thread pool with low & high watermarks. Main thread

would

On Wed, 5 Jun 2002 18:50:46 -0400
"Jon Franz" <coventry@one.net> wrote:

One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.

I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.

However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?

Cheers,

Neil

--
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

Yes I proposed to use the GNU Pth library instead. It's an event
demultiplexer just like the sgi library, but has a posix thread interface.
This architecture is actually the more robust and also the more scalable. On
a single processor server, you don't have the multi-thread synchronization
and context switching overhead and you also take full advantage of
multi-processor servers when you create several processes. Plus you have
much less concern about global variables.

Also for those concerned about the licence of this library here is an
abstract of it:
"The author places this library under the LGPL to make sure that it
can be used both commercially and non-commercially provided that
modifications to the code base are always donated back to the official
code base under the same license conditions. Please keep in mind that
especially using this library in code not staying under the GPL or
the LGPL _is_ allowed and that any taint or license creap into code
that uses the library is not the authors intention. It is just the
case that _including_ this library into the source tree of other
applications is a little bit more inconvinient because of the LGPL.
But it has to be this way for good reasons. And keep in mind that
inconvinient doesn't mean not allowed or even impossible."

So it can be used in both commercial and non commercial project.

----- Original Message -----
From: "Jon Franz" <coventry@one.net>
To: <pgsql-hackers@postgresql.org>
Sent: Thursday, June 06, 2002 8:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

One note: SGI developers discovered they could get amazing performance

using

as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model. Please see:

http://state-threads.sourceforge.net/docs/st.html

Thus, on platforms where it can be supported, we should keep in mind that

a

hybrid multiprocess/multithreaded postgresql might be the fastest
solution...

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Igor Kovalenko" <Igor.Kovalenko@motorola.com>
Cc: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

Igor Kovalenko wrote:

I might be naive here, but would not proper threading model remove the

need

for fork() altogether? On both Unix and Win32? Should not be too hard

to

come up with abstraction which encapsulates POSIX, BeOS and Win32

threads...

I am not sure how universal POSIX threads are by now. Any important

Unix

platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the

bright

side, performance should be better on some platforms (note however,

Apache

group still can't come up with implementation of threaded model which

would

provide better performance than forked or other models). The need to

deal

with possibility of 'alien' postmaster running along with orphaned

backends

would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some

things

will

probably have to be revamped. But in long term this is probably best

way

if

you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it

would

be

something like a thread pool with low & high watermarks. Main thread

would

handle thread pool and hand over requests to worker threads (blocked

on

condvar). How does that sound?

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 853-3000
+  If your life is a hard drive,     |  830 Blythe Avenue
+  Christ can be your backup.        |  Drexel Hill, Pennsylvania

19026

Neil Conway wrote:

On Wed, 5 Jun 2002 18:50:46 -0400
"Jon Franz" <coventry@one.net> wrote:

One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.

I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.

However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?

I know Red Hat is interested in AIO. Only a few OS's support it so it
was hard to get exited about it at the time, but with threading, a
AIO-specific module could be attempted.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Hello Thomas,

Wednesday, June 5, 2002, 7:02:33 PM, you wrote:

TL> ...

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

TL> I would think that we would build on our strengths of having a fork/exec
TL> model for separate clients. A threaded model *could* benefit individual
TL> clients who are doing queries on multiprocessor servers, and I would be
TL> supportive of efforts to enable that.
Just a note - this is also the solution adopted by Interbase/Firebird
and it seems interesting. They already had the same problems
PostgreSQL has been under today.
Those interested in read about Interbase's architeture, please refer
to http://community.borland.com/article/0,1410,23217,00.html.
"Classic" is the fork() model, and the "SuperServer" is the threaded
model.
-------------
Best regards,
Steve Howe mailto:howe@carcass.dhs.org

Gnu Pth also supports AIO
----- Original Message -----
From: "Nicolas Bazin" <nbazin@ingenico.com.au>
To: "Jon Franz" <coventry@one.net>; <pgsql-hackers@postgresql.org>
Sent: Thursday, June 06, 2002 10:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

Yes I proposed to use the GNU Pth library instead. It's an event
demultiplexer just like the sgi library, but has a posix thread interface.
This architecture is actually the more robust and also the more scalable.

On

a single processor server, you don't have the multi-thread synchronization
and context switching overhead and you also take full advantage of
multi-processor servers when you create several processes. Plus you have
much less concern about global variables.

Also for those concerned about the licence of this library here is an
abstract of it:
"The author places this library under the LGPL to make sure that it
can be used both commercially and non-commercially provided that
modifications to the code base are always donated back to the official
code base under the same license conditions. Please keep in mind that
especially using this library in code not staying under the GPL or
the LGPL _is_ allowed and that any taint or license creap into code
that uses the library is not the authors intention. It is just the
case that _including_ this library into the source tree of other
applications is a little bit more inconvinient because of the LGPL.
But it has to be this way for good reasons. And keep in mind that
inconvinient doesn't mean not allowed or even impossible."

So it can be used in both commercial and non commercial project.

----- Original Message -----
From: "Jon Franz" <coventry@one.net>
To: <pgsql-hackers@postgresql.org>
Sent: Thursday, June 06, 2002 8:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

One note: SGI developers discovered they could get amazing performance

using

as hybrid threaded and forked-process model with apache - we might want

to

look into this. They even have a library for network-communication
utilizing thier 'state threads' model. Please see:

http://state-threads.sourceforge.net/docs/st.html

Thus, on platforms where it can be supported, we should keep in mind

that

a

hybrid multiprocess/multithreaded postgresql might be the fastest
solution...

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Igor Kovalenko" <Igor.Kovalenko@motorola.com>
Cc: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

Igor Kovalenko wrote:

I might be naive here, but would not proper threading model remove

the

need

for fork() altogether? On both Unix and Win32? Should not be too

hard

to

come up with abstraction which encapsulates POSIX, BeOS and Win32

threads...

I am not sure how universal POSIX threads are by now. Any important

Unix

platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the

bright

side, performance should be better on some platforms (note however,

Apache

group still can't come up with implementation of threaded model

which

would

provide better performance than forked or other models). The need to

deal

with possibility of 'alien' postmaster running along with orphaned

backends

would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some

things

will

probably have to be revamped. But in long term this is probably best

way

if

you want to have efficient and uniform Unix AND Win32

implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it

would

be

something like a thread pool with low & high watermarks. Main thread

would

handle thread pool and hand over requests to worker threads (blocked

on

condvar). How does that sound?

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 853-3000
+  If your life is a hard drive,     |  830 Blythe Avenue
+  Christ can be your backup.        |  Drexel Hill, Pennsylvania

19026

---------------------------(end of

broadcast)---------------------------

TIP 1: subscribe and unsubscribe commands go to

majordomo@postgresql.org

Hello Bruce,

Wednesday, June 5, 2002, 1:33:44 AM, you wrote:

BM> INSTALLER
BM> ---------

BM> We clearly need an installer that is zero-hassle for users. We need to
BM> decide on a direction for this.
I suggest Nullsoft install system
(http://www.nullsoft.com/free/nsis/). It's real good and very simple
to use. I can help on this if you want.

BM> ENVIRONMENT

BM> Lots of our code requires a unix shell and utilities. Will we continue
BM> using cygwin for this?
There are other ports ( http://unxutils.sourceforge.net/ ) that won't
require Cygwin but they won't provide an environment so complete as
Cygwin does.

I also would like to empathize that probably a small GUI for
controlling the PostgreSQL service/application would be nice. I think
about something sitting in the system tray like MSSQL, Oracle,
Interbase, etc. does.
I could code this in Delphi if you like. I don't have experience in
writing GUI apps in C. There is an open source versions of Delphi so
it won't be a problem compiling it.
Also coming with this, a code for starting the PostgreSQL as a service
would be really nice. For those from UNIX world that don't know what a
service is, think about it as a daemon for Windows. A service can be
automatically started when the machine boots up.

-------------
Best regards,
Steve Howe mailto:howe@carcass.dhs.org

I think SGI gets amazing performance because they have very good (efficient)
synchronisation primitives on SGI. Some proprietary light-weight mutexes.
Using threaded or mixed model just by itself is not going to do a miracle.
Threads will save you some context switch time, but that will probably
translate into lower CPU usage rather than performance boost. And if your
mutexes are not fast or awkwardly implemented (say Linux), it might be even
worse. Apache is not all that fast on Linux as on SGI, whatever model you
chose. I also doubt that purely threaded model would be slower than mixed
one.

Now about the AIO model. It is useful when you need to do something else
while I/O requests are being processed as long as platform does it in some
useful way. If all you can do is to submit requests and keep sitting in
select/poll then AIO does not buy you anything you can't get by just using
threaded model. However, if you can tag the requests and set up
notifications, then few I/O threads could handle relatively large number of
requests from different clients. Note, this means you don't have any
association between clients and servers at all, there is pool of generic I/O
threads which serve requests from whoever they come. It saves system
resources and scales very well. It also provides interesting possibilities
for fault recovery - since handlers are generic all the state information
would have to be kept in some kind of global context area. That area can be
saved into persistent memory or dumped onto disk and *recovered* after a
forced restart. Server and library could be designed in such a way that
clients may continue where they left with a recoverable error.

In POSIX AIO model you can tag requests and set up notifications via
synchronous signals. You wait for them *synchronously* in 'waiter' thread
via sigwaitinfo() and avoid the headache of asynchronous signals hitting you
any time... Unfortunately on some platforms (Solaris) the depth of
synchronous signal queue is fixed at magic value 32 (and not adjustable).
This may not be a problem if you're sure that waiting thread will be able to
drain the queue faster than it gets filled with notifications... but I'm not
sure there is a portable way to guarantee that, so you need to check for
overloads and handle them... that complicates things. On Solaris you also
need a mile of compiler/linker switches to even get this scheme to work and
I am afraid other platforms may not support it at all (but then again, they
may not support AIO to begin with).

And speaking about getting best of all worlds. Note how Apache spent nearly
3 years developing their portable Multi-Processing Modules scheme. What they
got for that is handful of models neither of which perform noticeably better
than original pre-fork() model. Trying to swallow all possible ways to
handle things on all possible platforms usually does not produce very fast
code. It tends to produce very complex code with mediocre performance and
introduces extra complexity into configuration process. If you consider all
that was done mostly to support Win32, one might doubt if it was worth the
while.

What I am trying to say is, extra complexity in model to squeeze few percent
of performance is not a wise investment of time and efforts. On Win32 you
don't really compete in terms of performance. You compete in terms of
easyness and features. Spend 3 years trying to support Windows and Unix in
most optimal way including all subvariants of Unix ... meanwhile MSFT will
come up with some bundled SQL server. It probably will have more features
since they will spend time doing features rather than inventing a model to
support gazillion of platforms. Chances are, it will be faster too - due to
better integration with OS and better compiler.

I am not in position to tell you what to do guys. But if I was asked, I'd
say supporting Win32 is only worth it if it comes as a natural result of a
simple, coherent and uniform model applied to Unix. Threaded model may not
have as much inherent stability as forked/mixed, but it has inherent
simplicity and better Unix/Windows/BeOS portability. It can be done faster
and simpler code will make work on features easier.

Regards,
- Igor

"There are 2 ways to design an efficient system - first is to design it so
complex that there are no obvious deficiencies, second is to design it so
simple that there are obviously no deficiencies. Second way is much harder"
(author unknown to me)

----- Original Message -----
From: "Neil Conway" <nconway@klamath.dyndns.org>
To: "Jon Franz" <coventry@one.net>
Cc: <pgsql-hackers@postgresql.org>
Sent: Wednesday, June 05, 2002 7:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port

On Wed, 5 Jun 2002 18:50:46 -0400
"Jon Franz" <coventry@one.net> wrote:

One note: SGI developers discovered they could get amazing performance

using

as hybrid threaded and forked-process model with apache - we might want

to

Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.

---------------------------------------------------------------------------

INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/

GUI
---
pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/

BINARY
------

FORK()

cygwin fork() http://cygwin.com
CreateProcess() and copy global area

THREADING

Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST http://state-threads.sourceforge.net/docs/st.html
(single-session multi-threading possible)
(Posix AIO is possible)

IPC

Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE http://www.cs.wustl.edu/~schmidt/ACE.html
APR http://apr.apache.org/
Our own

ENVIRONMENT

Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb

IMPLEMENTATIONS
---------------
PostgreSQLe http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html
Dbexperts http://www.dbexperts.net/postgresql
Connx http://www.connx.com/
gborg http://gborg.postgresql.org/project/winpackage/projdisplay.php
Interbase http://community.borland.com/article/0,1410,23217,00.html

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Igor Kovalenko wrote:

I think SGI gets amazing performance because they have very good (efficient)
synchronization primitives on SGI. Some proprietary light-weight mutexes.
Using threaded or mixed model just by itself is not going to do a miracle.
Threads will save you some context switch time, but that will probably
translate into lower CPU usage rather than performance boost. And if your
mutexes are not fast or awkwardly implemented (say Linux), it might be even
worse. Apache is not all that fast on Linux as on SGI, whatever model you
chose. I also doubt that purely threaded model would be slower than mixed
one.

Let me throw out an idea. I have been mentioning full fork, light
fork(copy globals only), and threading as possible solutions.

Another idea uses neither threading nor copying. It is the old system
we used before I removed exec() from our code. We used to pass the
database name as an argument to an exec'ed postgres binary that
continued with the database connection.

We removed the exec, then started moving what we could into the
postmaster so each backend didn't need to do the initialization.

One solution is to return to that for Win32 only, so instead of doing:

initialization()
want for connection()
fork backend()

we do for Win32:

want for connection()
exec backend()
initialization()

It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Hi Bruce,

You obviosuly missed my recent posting advertising the homepage
of Konstantin Knizhnik?

Make sure to have a look: http://www.garret.ru/~knizhnik/

You find there -everything- concerning multiplatform IPC,
threading and even some extraordinary, complete database
backends that are superior to the database backends
previously available as open source (including PostgreSQL,
I'm afraid...). The licensing of all of this stuff is -public domain-.
I think this should really be worth a look/discussion/mentioning.

Here an excerpt of my last email, describing the furios list
of features abailable in GOODS:

Some core features of the GOODS backend (as they come to my mind):
-> full ACID transaction support, incl. distributed transactions
-> Multiple stoarge servers distributed over a TCP/ID network
-> multible reader/single writer (MVCC)
-> dual client side object cache
-> online backup (snapshot backup AND permanent backup)
-> nested transactions on object level
-> transaction isolation levels on object level
-> object level shared and exclusive locks
-> excellent C++ programming interface
-> WAL
-> garbage collection for no longer reference database objects (online VACUUM)
-> fully thread safe client interface
-> JAVA client API
-> very high performance as a result of a lot of fine tuning (better
perfomance than berkeley db in my benchmarks!!!)
-> asyncrous event notification on object instance modification
-> extremly high code quality
-> a one person effort, hence a very clean design
-> the most relevant platforms are supported out of the box
-> complete build is done in less than a minute on my machine
-> it's documented
-> it's tested and found to be working for a while now
...

kind regards,
Robert Schrem

Bruce Momjian wrote:

Let me throw out an idea. I have been mentioning full fork, light
fork(copy globals only), and threading as possible solutions.

Another idea uses neither threading nor copying. It is the old system
we used before I removed exec() from our code. We used to pass the
database name as an argument to an exec'ed postgres binary that
continued with the database connection.

We removed the exec, then started moving what we could into the
postmaster so each backend didn't need to do the initialization.

One solution is to return to that for Win32 only, so instead of doing:

initialization()
want for connection()
fork backend()

we do for Win32:

want for connection()
exec backend()
initialization()

Summarizes pretty much what we discussed Monday on the phone.
Except that the postmaster still has to initialize the shared
memory and other stuff. It's just that the backends and
helper processes need to reinitialize themself (attach).

It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.

All I see so far is the reading of the postgresql.conf, the
pg_hba.conf and the password files. Nothing fancy and the
postmaster could easily write out a binary content only file
that the backends then read, eliminating the parsing
overhead.

The bad news is that Tom is right. We did a terrible job in
using the new side effect, that the shared memory segment is
at the same address in all forked processes, after removing
the need to reattach.

In detail the XLog code, the FreeSpaceMap code and the
"shared memory" hashtable code now use pointers, located in
shared memory. For the XLog and FreeSpace code this is
understandable, because they where developed under the fork()
only model. But the dynahash code used offsets only until
v7.1!

All three (no claim that that's all) make it impossible to
ever have someone attaching to the shared memory from the
outside. So with these moves we made the shared memory a
"Postmaster and children" only thing. Raises the question,
why we need an IPC key at all any more.

Anyhow, looks as if I can get that fork() vs. fork()+exec()
feature done pretty soon. It'll be controlled by another
Postmaster commandline switch. After cleaning up the mess I
did to get it working quick, I'll provide a patch for
discussion.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Added to the list. Thanks.

---------------------------------------------------------------------------

Robert Schrem wrote:

Hi Bruce,

You obviosuly missed my recent posting advertising the homepage
of Konstantin Knizhnik?

Make sure to have a look: http://www.garret.ru/~knizhnik/

You find there -everything- concerning multiplatform IPC,
threading and even some extraordinary, complete database
backends that are superior to the database backends
previously available as open source (including PostgreSQL,
I'm afraid...). The licensing of all of this stuff is -public domain-.
I think this should really be worth a look/discussion/mentioning.

Here an excerpt of my last email, describing the furios list
of features abailable in GOODS:

Some core features of the GOODS backend (as they come to my mind):
-> full ACID transaction support, incl. distributed transactions
-> Multiple stoarge servers distributed over a TCP/ID network
-> multible reader/single writer (MVCC)
-> dual client side object cache
-> online backup (snapshot backup AND permanent backup)
-> nested transactions on object level
-> transaction isolation levels on object level
-> object level shared and exclusive locks
-> excellent C++ programming interface
-> WAL
-> garbage collection for no longer reference database objects (online VACUUM)
-> fully thread safe client interface
-> JAVA client API
-> very high performance as a result of a lot of fine tuning (better
perfomance than berkeley db in my benchmarks!!!)
-> asyncrous event notification on object instance modification
-> extremly high code quality
-> a one person effort, hence a very clean design
-> the most relevant platforms are supported out of the box
-> complete build is done in less than a minute on my machine
-> it's documented
-> it's tested and found to be working for a while now
...

kind regards,
Robert Schrem

On Thursday 06 June 2002 04:57, you wrote:

Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.

---------------------------------------------------------------------------

INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/

GUI
---
pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1 pgaccess
http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/

BINARY
------

FORK()

cygwin fork() http://cygwin.com
CreateProcess() and copy global area

THREADING

Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST
http://state-threads.sourceforge.net/docs/st.html (single-session
multi-threading possible)
(Posix AIO is possible)

IPC

Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE
http://www.cs.wustl.edu/~schmidt/ACE.html APR
http://apr.apache.org/
Our own

ENVIRONMENT

Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb

IMPLEMENTATIONS
---------------
PostgreSQLe
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html Dbexperts
http://www.dbexperts.net/postgresql Connx
http://www.connx.com/
gborg
http://gborg.postgresql.org/project/winpackage/projdisplay.php Interbase

http://community.borland.com/article/0,1410,23217,00.html

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Jan Wieck wrote:

One solution is to return to that for Win32 only, so instead of doing:

initialization()
want for connection()
fork backend()

we do for Win32:

want for connection()
exec backend()
initialization()

Summarizes pretty much what we discussed Monday on the phone.
Except that the postmaster still has to initialize the shared
memory and other stuff. It's just that the backends and
helper processes need to reinitialize themself (attach).

Yes, obviously I simplified, and I do believe our optimizations are
helping on Unix. It is just that I think for Win32 the fork is more
harmful than removing those optimizations.

One thing that may not have been clear is that we don't need to play
with globals at all. We just pass whatever info we want to the child
via command-line arguments, rather than shared memory.

It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.

All I see so far is the reading of the postgresql.conf, the
pg_hba.conf and the password files. Nothing fancy and the
postmaster could easily write out a binary content only file
that the backends then read, eliminating the parsing
overhead.

Yes, that is clearly possible. Another option is to just write out a
no-comments, no-whitespace version of each file and just have the
backends read those. The advantage is that we can use the same code to
read them, and I don't think it would be any slower than a binary file.

The bad news is that Tom is right. We did a terrible job in
using the new side effect, that the shared memory segment is
at the same address in all forked processes, after removing
the need to reattach.

In detail the XLog code, the FreeSpaceMap code and the
"shared memory" hashtable code now use pointers, located in
shared memory. For the XLog and FreeSpace code this is
understandable, because they where developed under the fork()
only model. But the dynahash code used offsets only until
v7.1!

All three (no claim that that's all) make it impossible to
ever have someone attaching to the shared memory from the
outside. So with these moves we made the shared memory a
"Postmaster and children" only thing. Raises the question,
why we need an IPC key at all any more.

Well, we could force shmat() to bind to the same address, but I suspect
that might fail in some cases.

Anyhow, looks as if I can get that fork() vs. fork()+exec()
feature done pretty soon. It'll be controlled by another
Postmaster commandline switch. After cleaning up the mess I
did to get it working quick, I'll provide a patch for
discussion.

Yes, very little impact. We then need someone to do some Win32 timings
to see if things have improved. As Tom mentioned, we need some hard
numbers for these things. In fact, I would like a Win32 test that takes
our code and compares fork(), then exit(), with CreateProcess(), exit().
It doesn't have create a db session, but I would like to see some
timings to know what we are gaining. Heck, time CreateThread too and
let's see what that shows.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian writes:

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

We should probably get rid of using shell scripts for application programs
altogether, for a number of reasons besides this one, such as the
inability to properly handle input values with spaces, commas, etc. (we
probably don't handle very long values either on some platforms), the
inability to maintain open database connections so that createlang needs
to prompt for the same password thrice, general portable scripting
headaches, and the lack of internationalization facilities.

I'd even volunteer to do this. Comments?

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

We should probably get rid of using shell scripts for application programs
altogether, for a number of reasons besides this one, such as the
inability to properly handle input values with spaces, commas, etc. (we
probably don't handle very long values either on some platforms), the
inability to maintain open database connections so that createlang needs
to prompt for the same password thrice, general portable scripting
headaches, and the lack of internationalization facilities.

I'd even volunteer to do this. Comments?

I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.

What language where you thinking of using? C?

Also, it seems Win32 doesn't need these scripts, except initdb.
PostgreSQLe didn't use the, it just did initdb, and the rest were done
using a GUI. However, initdb would remain a problem. PostgreSQLe wrote
its own.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-----Original Message-----
From: Steve Howe [mailto:howe@carcass.dhs.org]
Sent: 06 June 2002 02:37
To: Bruce Momjian
Cc: PostgreSQL-development
Subject: Re: Roadmap for a Win32 port

Hello Bruce,

Wednesday, June 5, 2002, 1:33:44 AM, you wrote:

BM> INSTALLER
BM> ---------

BM> We clearly need an installer that is zero-hassle for
users. We need
BM> to decide on a direction for this.
I suggest Nullsoft install system
(http://www.nullsoft.com/free/nsis/). It's > real good and very
simple to use. I can help on this if you want.

I think that a Windows Installer compatible package would be better as
it would allow us to build the package as a merge module which others
could use in their installers for their PostgreSQL based apps, allowing
one installation to install everything they require easily and (more
importantly) correctly. An example of this can be found in the psqlODBC
installer.

I can handle this if required.

BM> ENVIRONMENT

I also would like to empathize that probably a small GUI for
controlling the PostgreSQL service/application would be nice.

I'm happy to add such code to pgAdmin - seems like the natural thing to
do (to me at least!).

Regards, Dave.

Bruce Momjian writes:

GUI
---
pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/

Surely Unix folks would like a GUI as well?

--
Peter Eisentraut peter_e@gmx.net

How about a SOAP interface and a web-based front end that provides the cross
platform support? My company's TIBET framework would provide a solid
foundation for this kind of admin suite. In fact, we're already in the
planning stages on doing just that.

ss

Scott Shattuck
Technical Pursuit Inc.

----- Original Message -----
From: "Peter Eisentraut" <peter_e@gmx.net>
To: "Bruce Momjian" <pgman@candle.pha.pa.us>
Cc: "PostgreSQL-development" <pgsql-hackers@postgreSQL.org>
Sent: Friday, June 07, 2002 11:42 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port

Bruce Momjian writes:

GUI
---
pgAdmin2

http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1

pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)

http://sourceforge.net/projects/dev-cpp/

Bruce Momjian writes:

I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.

My point is that it is not, for the reasons that I listed. Handling
spaces is a small part of one of the several problems, there are problems
with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
read you lose one character.

What language where you thinking of using? C?

Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
use the established internationalization facilities, and use libpq
directly in create* and drop*.

Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.

I have been seriously considering converting pg_dumpall to C anyway,
because it's already *very* messy, and I don't see any reasonable
way to make it support dumping per-database and per-user config
settings. (Do you really want to try to parse array values in a
shell script?)

(I'd actually consider making pg_dumpall a part of the pg_dump
executable; then it could invoke pg_dump as a subroutine call...)

If Peter's got the time/energy to convert 'em all, I'm for it.

regards, tom lane

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.

I have been seriously considering converting pg_dumpall to C anyway,
because it's already *very* messy, and I don't see any reasonable
way to make it support dumping per-database and per-user config
settings. (Do you really want to try to parse array values in a
shell script?)

(I'd actually consider making pg_dumpall a part of the pg_dump
executable; then it could invoke pg_dump as a subroutine call...)

If Peter's got the time/energy to convert 'em all, I'm for it.

Yea, shame it will now take 15 lines of C code to do what we could do in
1 line of shell script but I don't see any other option.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Yea, shame it will now take 15 lines of C code to do what we could do in
1 line of shell script but I don't see any other option.

In places we are using 15 lines of shell to do what would take 1 line
in C ;-). Yes, it'll probably be bigger overall, but I think you are
overstating the penalty.

regards, tom lane

Peter Eisentraut wrote:

Bruce Momjian writes:

I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.

My point is that it is not, for the reasons that I listed. Handling
spaces is a small part of one of the several problems, there are problems
with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
read you lose one character.

What language where you thinking of using? C?

Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
use the established internationalization facilities, and use libpq
directly in create* and drop*.

Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.

I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: 08 June 2002 22:48
To: Peter Eisentraut
Cc: PostgreSQL-development
Subject: Re: Roadmap for a Win32 port

Also, it seems Win32 doesn't need these scripts, except initdb.

The utility of these programs is independent of the

platform. If we

think pg_dumpall is not useful, then let's remove it.

I think the first two targets for C-ification would be
pg_dumpall and initdb. The others have SQL equivalents.
Maybe pg_ctl too.

I looked at this issue some time ago & came to the conclusion that the
only scripts that Win32 really needed were pg_dumpall, initdb &
initlocation.

The others have SQL equivalents as you say, apart from pg_ctl which
under Windows should probably (and generally is) be replaced by the SCM
(Service Control Manager). The only thing that comes to mind that the
SCM can't do is a reload.

Regards, Dave.

Scott,

I just started a java admin tool project on sf called
www.sf.net/projects/jpgadmin, which should be able to handle web based
interfaces, the idea being to seperate the model and view so that we can
support a swing or web interface.

Dave

Bruce Momjian writes:

I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut wrote:

Bruce Momjian writes:

I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Jan Wieck wrote:

And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.

No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to do that than
a shell script.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:

Peter Eisentraut wrote:

Bruce Momjian writes:

I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.

No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?

It will be at least another copy of the postmaster (dot exe).

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being
right. #
# Let's break this rule - forgive
me. #
#==================================================
JanWieck@Yahoo.com #

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Monday, June 17, 2002 6:01 PM
To: Jan Wieck
Cc: Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Jan Wieck wrote:

And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.

No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but a spawn or a
create process to create threaded Postgres instances?

Just a notion.

Dann Corbit wrote:

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but a spawn or a
create process to create threaded Postgres instances?

Can't. postmaster/postgres are symlinks to the same file, and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary

somehow reporting

the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but

a spawn or a

create process to create threaded Postgres instances?

Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

Dann Corbit wrote:

Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

We used to fork() and exec(), but that was slow. Now we preload stuff
in the postmaster for each backend. It is faster.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Dann Corbit wrote:

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary

somehow reporting

the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but

a spawn or a

create process to create threaded Postgres instances?

Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.

exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Peter Eisentraut <peter_e@gmx.net> writes:

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

Like what?

The thing pg_ctl needs to know is where PGDATA is, and that
unfortunately isn't going to be known any better just by sharing
executables.

I don't object to C-ifying pg_ctl, but I don't see that it will
automatically improve pg_ctl's robustness materially.

regards, tom lane

On Tue, 2002-06-18 at 09:07, Jan Wieck wrote:

Dann Corbit wrote:

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)

Well, if you're going to go through the trouble of rewriting postmaster
to be win32 specific, you might as well make it hook into the services
control panel.

If I recall, shared memory is "owned" by a process in Win32 (please
correct as needed...as I've slept since I last looked). That means that
the postmaster process not only owns the shared memory but needs to make
sure that it's persists as the rest of postgres is expecting.

Please provide more details as to the nature of your Win32 changes. I'm
certainly curious. If you've already covered them, just say so...I have
no problem going back to the archives! :)

Greg

Hello,

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

They have had to turn their multi-process model used on Unix into a fully
multi-threaded one on Win32. I have the feeling that they have had the same
debate that the one you have.

The CreateProcess() syscall is very costly on Windows. Some improvements
have been done in Windows XP but it is still far more costly than a Unix
fork().

I have been programming with threads on NT for a long time now.
They are quiet robust and efficient. I fear that it is the only successful
way to port PostgreSQL.

Sorry for this interruption,
Serge

-----Original Message-----
From: Jan Wieck [mailto:JanWieck@Yahoo.com]
Sent: Tuesday, June 18, 2002 16:07
To: Dann Corbit
Cc: Bruce Momjian; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary

somehow reporting

the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but

a spawn or a

create process to create threaded Postgres instances?

Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.

exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

I know that Apache Group created special library to handle difference
between different platforms (including win32). They had similar problems
porting Apache to Windows. They build very portable threads api (win32,
POSIX, native Linux thread and more) There is also all IPC stuff (mutex,
signals mmap etc.) and many more. This functions work both on unix and
windows and use most effective implementation (e.g. POSIX functions on
Winodws are slow compared to native).

http://apr.apache.org/

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Serge Adda
Sent: Tuesday, June 18, 2002 6:43 PM
To: 'Jan Wieck'; 'Dann Corbit'
Cc: 'Bruce Momjian'; 'Peter Eisentraut'; 'PostgreSQL-development'
Subject: Re: [HACKERS] Roadmap for a Win32 port

Hello,

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

They have had to turn their multi-process model used on Unix into a
fully
multi-threaded one on Win32. I have the feeling that they have had the
same
debate that the one you have.

The CreateProcess() syscall is very costly on Windows. Some improvements
have been done in Windows XP but it is still far more costly than a Unix
fork().

I have been programming with threads on NT for a long time now.
They are quiet robust and efficient. I fear that it is the only
successful
way to port PostgreSQL.

Sorry for this interruption,
Serge

-----Original Message-----
From: Jan Wieck [mailto:JanWieck@Yahoo.com]
Sent: Tuesday, June 18, 2002 16:07
To: Dann Corbit
Cc: Bruce Momjian; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

Dann Corbit wrote:

It will be at least another copy of the postmaster (dot exe).

Yea, I just liked the idea of the postmaster binary

somehow reporting

the postmaster status. Seems it is in a better position to
do that than
a shell script.

Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers.

It

does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but

a spawn or a

create process to create threaded Postgres instances?

Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.

Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could

be

passed to the Postgres servers on the command line.

exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.

The startup stuff for PostgreSQL is just a few files. It does not

seem

insurmountable to change it. But it is none of my business. If it is

a

major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Tom Lane writes:

Peter Eisentraut <peter_e@gmx.net> writes:

I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.

Like what?

The biggie is that pg_ctl reports the postmaster to have started
successfully without ever checking. And the "wait" option is broken and
not trivial to fix.

Other problems are the matching of the port numbers and the requirement
that admins should be able to enter a password when the server starts (for
SSL).

The luring prerequisite here is that the postmaster would have to be able
to log directly to a file, which now that all communication is guaranteed
to go through elog() should be less complicated, at least compared to
fixing the "wait" option. In fact I'm hoping that the Windows porters
will run into this same requirement just about pretty soon.

--
Peter Eisentraut peter_e@gmx.net

Peter Eisentraut <peter_e@gmx.net> writes:

Like what?

The biggie is that pg_ctl reports the postmaster to have started
successfully without ever checking. And the "wait" option is broken and
not trivial to fix.

Indeed, but how will it help to merge the two executables into one?
I don't think you can simply postpone the fork() until all setup
is complete --- that would mean you don't know the final postmaster
PID until much too late.

regards, tom lane

Le Mardi 18 Juin 2002 18:42, Serge Adda a écrit :

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

Hello,

It seems clear that several teams are working without central point management
and contact:

- W32 port: at least a Japase team, a PostgreSQL hacker team and a company are
working separately. This makes three separate efforts for the most important
project this year.

- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.

- Web site: a www list is working on several issues. Is there a central design
for all PostgreSQL site, like PHP does?

- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?

Personnaly, I agree with someone who wrote "Remember Betamax. It was the best
technical standard, but it died for commercial reasons". I don't want
PostgreSQL to be the next Betamax.

Some projects, like Debian, have a democratic organisation. The team leader is
elected for a year. Why not settle a similar organization? This would help
take decisions ... and not loose time on important issues.

PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.

Just my 2 cents.
Cheers.
Jean-Michel POURE

On Thu, Jun 20, 2002 at 12:01:53PM +0200, Jean-Michel POURE wrote:

Some projects, like Debian, have a democratic organisation. The team leader is
elected for a year. Why not settle a similar organization? This would help

IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
:-)

The PostgreSQL is not project like Debian -- if you want to work on Debian
you must know package system and Debian standards only. But if you want to
work on PostgreSQL you must be at the least average programmer (means several
years experience) and you must know a great many of current PostgreSQL code.

PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.

What is non-democratic now?

Karel

--
Karel Zak <zakkr@zf.jcu.cz>
http://home.zf.jcu.cz/~zakkr/

C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Le Jeudi 20 Juin 2002 13:39, Karel Zak a écrit :

IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors

Dear Karel,

My previous e-mail points out several projects where, IMHO, a leadership would
benefit the community at large :
- replication,
- W32 port,
- marketing (read the post "Read this and puke").

What is non-democratic now?

The current processes are based on discussion, and therefore are democratic.
My proposal does not intend to change discussion processes between
pgsql-hackers.

But, in order to face companies like MySQL AB, Oracle or Micro$oft, the
community needs to take important decisions that will help team work. A
clarified organization would help.

Please note I am not a PostgreSQL hacker myself, as I do not contribute code
to PostgreSQL main sources. But, as an outside spectator, I would only like
to point out that some efforts need coordination.

Debian is a very interesting example of Open-Source organization, as for all
aspects linked to "decision making". Usually, at Debian, when a discussion is
driven, a clear choice arizes after a limited time. Projects are sometimes
slow, but always reach their goals.

As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?

In other words, we should ask ourselves the question of PostgreSQL future
organization. We come to point where PostgreSQL has equal chances to become
the #1 database or die like Betamax.

Best regards to you all,
Jean-Michel

On Thu, Jun 20, 2002 at 02:33:04PM +0200, Jean-Michel POURE wrote:

Le Jeudi 20 Juin 2002 13:39, Karel Zak a �crit :

IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors

My previous e-mail points out several projects where, IMHO, a leadership would
benefit the community at large :
- replication,
- W32 port,
- marketing (read the post "Read this and puke").

I understend you. I saw a lot of project and ideas, but it
_always_ depend on people and their time. You can organize, you can
prepare cool planns, but don't forget -- in finale must somebody
implement it. Who? I don't know method how clone Tom Lane or get
money for others developers who can't full time work on PostgreSQL
now.

As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a

If nobody -- you can test it, make it better and write something
about it. If nobody works on some theme it means this theme is not
important for now. BUT everybody can change it and everybody can
start work on arbitrary TODO item. It seems hard, but it's right :-)

Karel

--
Karel Zak <zakkr@zf.jcu.cz>
http://home.zf.jcu.cz/~zakkr/

C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Jean-Michel POURE wrote:

[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?

I think the first native Win32 port that gets contributet will make it.
So far I have seen a couple of people claiming the "have" a native Win32
port, not using CygWIN. But none of them was willing to contribute their
work.

There is no official "schedule" for any of the features. PostgreSQL is a
100% volunteer project, so people work on it if and when they feel the
need to AND have the time. Even if some of us do work on PostgreSQL at
their job, a few even fulltime, you will not see such commitments.

And why do you think InnoDB is a marketing lie? It bought MySQL a good
number of features, including row level locking, transactions and
limited foreign key support. Actually it stopped MySQL from yelling out
the FUD they told people far too long, like "transactions are useless
overhead" and "referential integrity is bad".

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

On Thu, 20 Jun 2002, Jean-Michel POURE wrote:

Le Jeudi 20 Juin 2002 13:39, Karel Zak a écrit :

IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors

Dear Karel,

My previous e-mail points out several projects where, IMHO, a leadership would
benefit the community at large :
- replication,
- W32 port,
- marketing (read the post "Read this and puke").

What is non-democratic now?

The current processes are based on discussion, and therefore are democratic.
My proposal does not intend to change discussion processes between
pgsql-hackers.

But, in order to face companies like MySQL AB, Oracle or Micro$oft, the
community needs to take important decisions that will help team work. A
clarified organization would help.

Jean,

Why on earth does this matter? Postgres will continue to be a good
database as long as developers and users cut code. It is not a
sufficiently complicated project to warrant too much concern about things
like this. Besides, a significant amount of the code committed to
Postgres is inspired by personal interest not obligation.

Gavin

Jean-Michel POURE <jm.poure@freesurf.fr> writes:

As for current PostgreSQL organization, can someone explain me which
W32 port will make its way to PostgreSQL main source code? Can someone
publish a schedule for replication availability? Who is in charge of
explaining newbees that MySQL InnoDB is just a marketing lie? What is
the current PostgreSQL market share?

And an "elected leader" will make all this stuff clear exactly how?

If someone would like to step up and actually *do* that marketing work,
that'd be great. Complaining that it's not being done is a waste of
list bandwidth. Electing someone isn't going to magically make it
happen, either.

It does concern me that there seem to be several W32 porting efforts
going on without contact with each other or with the wider pghackers
community; but if they want to work that way, I don't think there's
much we can do to force them to come out in the open.

BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.

regards, tom lane

Le Jeudi 20 Juin 2002 15:22, Tom Lane a écrit :

BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.

You are right. Leading by example and consensus is the way things work. If the
committee can publish clear goals, there is no need to elect a leader.

Now I realize my concern is about project management. Do you think the
committee could publish a project web page showing who works on what?

Someone willing to help on a projet would view the page and contact
developpers more easilly. Presently, there is no clear knowledge who is
working on the W32 port or on replication. Using a project page, there would
be less chances to see new forks of particular projects (W32, replication).

This page would be accessible from the to-do-list. Conversly, the to-do-list
would lead to the projects page.

Just my 2 cents.
Thanks to all of you who replied my mail.

Cheers,
Jean-Michel

On Thu, 2002-06-20 at 14:33, Jean-Michel POURE wrote:

Le Jeudi 20 Juin 2002 13:39, Karel Zak a écrit :

IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors

Dear Karel,

My previous e-mail points out several projects where, IMHO, a leadership would
benefit the community at large :
- replication,
- W32 port,
- marketing (read the post "Read this and puke").

What is non-democratic now?

The current processes are based on discussion, and therefore are democratic.
My proposal does not intend to change discussion processes between
pgsql-hackers.

But, in order to face companies like MySQL AB, Oracle or Micro$oft, the
community needs to take important decisions that will help team work. A
clarified organization would help.

In what way ?

Do you really think that if we would elect Tom or Bruce or someone else
"The President of the PostgreSQL Community" then their word would weight
more in mainstream press ?

Please note I am not a PostgreSQL hacker myself, as I do not contribute code
to PostgreSQL main sources. But, as an outside spectator, I would only like
to point out that some efforts need coordination.

Debian is a very interesting example of Open-Source organization, as for all
aspects linked to "decision making". Usually, at Debian, when a discussion is
driven, a clear choice arizes after a limited time. Projects are sometimes
slow, but always reach their goals.

From an "outside spactator" perspective Debian seems to be really,
really slow.

As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code?

There is already one W32 port in main source (the one that uses cygwin).

IMHO the first native port that
a) works
b) does not make *nix version slower/harder to maintain
and
c) is submitted for inclusion will make it into main source.

Can someone publish a schedule for replication availability?

I guess any of the teams working on different ways to replicate could do
it. Another question is if they can stick to it.

Who is in charge of explaining newbees that MySQL InnoDB is just a
marketing lie?

Nobody is "in charge", but everybody is welcome to do it, even without
being "elected" or "nominated ";)

Still, having a "success stories" or "advocacy" section on
www.postgresq.org seems like a good idea.

What is the current PostgreSQL market share?

In other words, we should ask ourselves the question of PostgreSQL future
organization.

The current organization is "a loosely knit team" which seems to work
quite well.

We come to point where PostgreSQL has equal chances to become
the #1 database or die like Betamax.

Open-source will probably work differently, i.e postgres will probably
not die even if it will not be #1.

----------------
Hannu

Jean-Michel POURE wrote:

Le Mardi 18 Juin 2002 18:42, Serge Adda a ?crit :

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

Hello,

It seems clear that several teams are working without central point management
and contact:

- W32 port: at least a Japanese team, a PostgreSQL hacker team and a company are
working separately. This makes three separate efforts for the most important
project this year.

Funny you should mention that. I talked with three people on the phone
just yesterday in an attempt to get them together on the Win32 port. I
have been hesitant to publicly try and join them together because they
haven't publicly stated if they are going to contribute their code
back to the project, or haven't been clear on _when_ they would
contribute it back.

My goal is to take the roadmap I posted on June 5, make a web page, and
get some agreement from the community on how to address each item:

http://candle.pha.pa.us/cgi-bin/pgtodo?win32
.
Then, the groups can know how we prefer to have things done, and if they
contribute back quickly, the other projects can benefit from their work,
and we _don't_ get conflicting Win32 implementations contributed back to
the project.

- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.

I am on the replication mailing list:

http://gborg.postgresql.org/project/pgreplication/projdisplay.php

That project is moving along, and hopefully they can merge their code
into the main tree so others can assist them. Not sure what else we can
do.

- Web site: a www list is working on several issues. Is there a central design
for all PostgreSQL site, like PHP does?

- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?

I skipped commenting on the marketing discussion because I was working
on a patch. Probably our biggest problem compared to MySQL is that
MySQL is a single company behind a single database, while we have >4
companies behind PostgreSQL, and their thrust is promoting their company
rather than PostgreSQL itself.

While we have an excellent development team, we don't have a funded team
to go around to trade shows and write articles promoting PostgreSQL. We
only have a very few people employed full-time with PostgreSQL, and it
usually takes a few full-time people to do just marketing. Great Bridge
was really the only one to push PostgreSQL, and they are gone.

MySQL has such a team, and so does Oracle, and it helps. Linux was in a
similar boat, with multiple companies behind Linux, and every one
promoting its own company rather than Linux itself. We need large
PostgreSQL companies that promote themselves, and PostgreSQL along with
it. Linux is in the same boat, with distributors promoting themselves
and Linux along with it.

Personnaly, I agree with someone who wrote "Remember Betamax. It was the best
technical standard, but it died for commercial reasons". I don't want
PostgreSQL to be the next Betamax.

I thought the problem with Betamax was that Sony controlled the
standard, and other companies didn't like that.

I am always looking for more ideas in these areas.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

...

MySQL has such a team, and so does Oracle, and it helps. Linux was in a
similar boat, with multiple companies behind Linux, and every one
promoting its own company rather than Linux itself. We need large
PostgreSQL companies that promote themselves, and PostgreSQL along with
it. Linux is in the same boat, with distributors promoting themselves
and Linux along with it.

Right. That may have slowed down the trade press recognition of Linux,
but in the end Linux has risen into view by popular demand and acclaim.

PostgreSQL has a similar opportunity and can succeed on a similar path.
We do have a superior technical solution, we have a superior open-source
support system, and we have (imho :) a superior development model and
development team open to new contributions and innovation.

The basis is there for success into the future, and at some point we
will get a lot for a little effort on marketing. Other products
bootstrap themselves by marketing early and often. And to the detriment
of the technical side.

- Thomas

Jan Wieck wrote:

Jean-Michel POURE wrote:

[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?

I think the first native Win32 port that gets contributet will make it.

This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.

Also, I encourage people working on Win32 ports to contribute their work
_as_ the complete each module, rather than as one huge patch. That way,
other Win32 porters can benefit from their work and maybe we can get
this done in 1/2 the time.

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

So, my message to Win32 porters is to communicate what you are doing,
including implementation details, and contribute back as soon as you
can. If you don't, someone else may beat you to it, and your patch may
be rejected, leaving you to either scrap your work or continually port
your changes to every future PostgreSQL release.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Thu, 20 Jun 2002, Jean-Michel POURE wrote:

As for current PostgreSQL organization, can someone explain me which W32
port will make its way to PostgreSQL main source code?

Whichever one actually submits patches for review first that is deemed
acceptable for inclusion ... as its always been ...

Can someone publish a schedule for replication availability?

Its available now, and has been for several months *shrug* There is a
project called PgReplication that is Open Source (hopefully someone from
that camp will pop up) ... and PgSQL, Inc has several deployments of their
commercial replication out there now, with several more pending ...

In fact, all of our work has been based on the rserv code that is in
contrib, and that we released over (almost?) a year ago ... but I've seen
nobody actually try to build on it, altho its what we've extended and are
using sucessfully in production environments ...

Who is in charge of explaining newbees that MySQL InnoDB is just a
marketing lie?

You are ... and anyone else that asks about it / mentions it ...

What is the current PostgreSQL market share?

If you can think of a method of calculating this, as there is no
'commercial licensing' involved, please let us know ... would love to find
out ... there have been several surveys about it, but, quite frankly,
without having some sort of 'licensing' required to use, there is zero way
of getting any *real* numbers on this ...

In other words, we should ask ourselves the question of PostgreSQL
future organization. We come to point where PostgreSQL has equal chances
to become the #1 database or die like Betamax.

At this point in time, the current organization == future organization ...
you are putting a 'marketing effort' as being the responsibility of the
open source project, and using MySQL AB as a comparison ... MySQL AB is a
*commercial company*, as is PostgreSQL, Inc -and- SRA and several other
newcomers, all of whom are doing marketing in their own way, based on
budget and requirements for growth ...

The "developmental organization", which we are, has been successful for
the past 7 years now ... flame wars are minimal, as are disagreements ...
there are some patches that get rejected that those generating them are
disappointed about, but most of *them* just bounce back with improvements
based on what has been told to them as being unacceptable ... about the
only thing that *has* changed over the past 7 years is that our standards
are tightened up as we move from mainly fixing bugs/stability to improving
the server itself ...

On Thu, 20 Jun 2002, Bruce Momjian wrote:

Jan Wieck wrote:

Jean-Michel POURE wrote:

[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?

I think the first native Win32 port that gets contributet will make it.

This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.

1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...

Also, I encourage people working on Win32 ports to contribute their work
_as_ the complete each module, rather than as one huge patch. That way,
other Win32 porters can benefit from their work and maybe we can get
this done in 1/2 the time.

This is what should be done anyway ... I wouldn't even be so much worried
abou t"other Win32 porters" as much as trying to integrate that 'hugh
patch' at a later date, consiedering all the changes that go into our code
:)

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)

So, my message to Win32 porters is to communicate what you are doing,
including implementation details, and contribute back as soon as you
can. If you don't, someone else may beat you to it, and your patch may
be rejected, leaving you to either scrap your work or continually port
your changes to every future PostgreSQL release.

Agreed here ... and this doesn't just apply to Win32 porters ... there
have been *alot* of "big projects" worked on over the years that have been
implemented in pieces so that code-base doesn't diverge too much, making
patching difficult ... the last thing anyone should want is to work for a
few months to create a nice big patch to find out that what they have
"fixed" got yanked out of the code already :)

On Thu, 20 Jun 2002, Jean-Michel POURE wrote:

Le Jeudi 20 Juin 2002 15:22, Tom Lane a ���crit :

BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.

You are right. Leading by example and consensus is the way things work. If the
committee can publish clear goals, there is no need to elect a leader.

Now I realize my concern is about project management. Do you think the
committee could publish a project web page showing who works on what?

No ... cause nobody works on anything specific ... they work on what
appeals to them, as it appeals to them ...

Someone willing to help on a projet would view the page and contact
developpers more easilly. Presently, there is no clear knowledge who is
working on the W32 port or on replication. Using a project page, there
would be less chances to see new forks of particular projects (W32,
replication).

If someone workign on the w32 port isn't willing to announce such on
-hackers, how do you think we'd get that information for a web page? The
easiest way to 'contact developers' is to post to this list, and then you
don't restrict yourself to anyone one person, but everyone involved in teh
project ...

I think you are confusing a project on the scale of an operating system
with PgSQL ... an OS has to have someone at the top to make sure each
sub-package integrates well into the overall OS ... we *are* the
sub-package, and our "leaders" are essentially "our developers" ... we
have a core committee that consists of several of us that brought PgSQL
out of Berkeley and into the real world, but if you saw the archives for
that list, you woudln't see much, since we tend to try and keep all
discussions *on* -hackers ...

On 20 Jun 2002, Hannu Krosing wrote:

Nobody is "in charge", but everybody is welcome to do it, even without
being "elected" or "nominated ";)

Still, having a "success stories" or "advocacy" section on
www.postgresq.org seems like a good idea.

Being worked on ... we are actually working on totally revamping and
cleaning up the web site(s) ...

On Thu, 20 Jun 2002 12:09:35 -0400 (EDT)
"Bruce Momjian" <pgman@candle.pha.pa.us> wrote:

Jean-Michel POURE wrote:

- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.

Replication development is co-ordinated on the pgreplication-general
list which Bruce mentions below, not on -hackers. Any interested
developers should subscribe to it, read the relevant research papers
on Postgres-R, and contribute code.

As for the speed of development, I've started to contribute code recently,
and Darren Johnson (the main replication developer) says he should have some
free time soon -- there are also some other new developers interested in
the project. So while there was a period of inactivity, I think that progress
is now being made.

I am on the replication mailing list:

http://gborg.postgresql.org/project/pgreplication/projdisplay.php

That project is moving along, and hopefully they can merge their code
into the main tree so others can assist them. Not sure what else we can
do.

I can't make any claims about a schedule -- since everyone working on the
project is a volunteer, the only release goals I'd like to set are "when it's
ready". The code right now is pretty unstable, so I don't think it's appropriate
for the main CVS tree until we work it into better shape.

Cheers,

Neil

--
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

"Marc G. Fournier" wrote:

On Thu, 20 Jun 2002, Bruce Momjian wrote:

This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.

1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...

Omitted point taken :-)

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)

The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #

Jan Wieck wrote:

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)

The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!

Yes, but it doesn't prevent discussion. I think open implementation
discussion will help. I am suggesting this to everyone, not just Jan.
I have been in private discussion with others too.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Thursday 20 June 2002 02:57 pm, Jan Wieck wrote:

set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions?

Don Baccus?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Jan Wieck wrote:

"Marc G. Fournier" wrote:

...

IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)

The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!

It would be nice to also have it "fire up" under Windows CE as well ;-)

Mike Mascari
mascarm@mascari.com

On Thu, 20 Jun 2002, Jan Wieck wrote:

"Marc G. Fournier" wrote:

On Thu, 20 Jun 2002, Bruce Momjian wrote:

This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.

1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...

Omitted point taken :-)

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)

The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port

Actually, IIRC Don did the triggers, and I did the utility commands/create
stuff, but the point is still the same. (Made in the point of
historical accuracy since I don't want someone else's work to end up
getting attributed to me since that's unfair to them. :) )

Stephan Szabo wrote:

Actually, IIRC Don did the triggers, and I did the utility commands/create
stuff, but the point is still the same. (Made in the point of
historical accuracy since I don't want someone else's work to end up
getting attributed to me since that's unfair to them. :) )

This is why I love this PostgreSQL Project. The honesty and
humbleness everyone is practicing. Who ever did what, together we
did a good job! Which other open source database has the full set
of referential actions and deferrability?

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being
right. #
# Let's break this rule - forgive
me. #
#==================================================
JanWieck@Yahoo.com #

It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.

However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.
If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.

~Jon Franz

Bruce Momjian wrote:

Jon Franz wrote:

It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.

However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.

I think we have to be involved to prevent chaos when those patches
arrive.

If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.

Actually, don't we have a cygwin mailing list? Seems that would be a
great location, except for the name. Maybe Marc can close the list and
migrate them all to a new 'win32' list.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

On Thu, 20 Jun 2002, Bruce Momjian wrote:

Jon Franz wrote:

It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.

However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.

I think we have to be involved to prevent chaos when those patches
arrive.

If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.

Actually, don't we have a cygwin mailing list? Seems that would be a
great location, except for the name. Maybe Marc can close the list and
migrate them all to a new 'win32' list.

Two different issues here ... a win32 port is different then the cygwin
issues ... also, I had suggested ages back creatin ga seperate list, but
was told that it would frustrate the effort ... so unless those that
disagreed with it in the first place have changed their mind, the
discussion of any porting effort to win32 should remain here ...

Jean-Michel,

It seems clear that several teams are working without central point

management

and contact:

<snip>

- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?

<snip>
ome projects, like Debian, have a democratic organisation. The team leader is

elected for a year. Why not settle a similar organization? This would help
take decisions ... and not loose time on important issues.

PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.

Let me introduce myself. In addition to being a contributor of supplimentary
documentation and the occasional spec to the PostgreSQL project, I am
volunteer marketing lead and the primary motivator for governance overhaul in
the OpenOffice.org project.

And frankly, I think you're way off base here. We have leaders: Tom, Bruce,
Jan, Stephan, Thomas, Marc and Oliver (did I miss anybody?). Frankly, if
OpenOffice.org had the kind of widely trusted, committed, involved in the
community core developers that PostgreSQL already has, I wouldn't be on my
fourth draft of an OpenOffice.org Community Council charter. OpenOffice.org
will have an election process because we are too big and too dispersed for a
simple trust network, not because we want one for its own sake.

PostgreSQL is, quite possibly, the smoothest-running Open Source project with
worldwide adoption. I find myself saying, at least once a week, "if only
project X were as well-organized as PostgreSQL!" It is perhaps not
coincidental that Postgres is one of the 15 or 20 oldest Open Source projects
(older than Linux, I believe).

How would a "democratic election" improve this? And why would we want an
elected person or body who was not a core developer? And if we elected a
core developer, why bother? They aready run things.

Regarding your marketing angle: Feel free to nominate yourself "PostgreSQL
Marketing Czar." Write articles. Contact journalists. Generate press
releases for each new Postgres version. Apply for a dot-org booth at
LinuxWorld. Nobody voted for me (actually, I got stuck with the job by not
protesting hard enough <grin>).

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors. As often as not, the database engine choice is made by the
DBA, and they will choose PostgreSQL on its merits, not because of some
Washington Post article.

OpenOffice.org is a different story, as an end-user application. So we have
a Marketing Project.

--
-Josh Berkus
Porject Lead, OpenOffice.org Marketing

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

regards, tom lane

On Tue, 25 Jun 2002, Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

Actually, I'm not sure how "viable" MySQL *is* in a commercial environment
... personally, I think that they just shot themselves in the foot with
their recent 'law suit' with NuSphere, no? Other then MySQL AB
themselves, how many are going to jump onto that bandwagon if nobody is
allowed *some* sort of competitive advantage?

So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

I don't know ... again, my view of Monty is extended to "if it doesn't get
submitted for review, whether it gets in or not, we'll sue you for breach
of license" ... really gives those considering jumping onto MySQL a warm,
fuzzy feeling, no? :(

On Tue, 2002-06-25 at 07:21, Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

#pragma no-flames-please

When/if MySQL becomes better than PostgreSQL, we'd simply have a still
better open-source/free software database than we have now.

In what way exactly is this bad ?

Perhaps (hypothetically speaking) the "hot breath" of MySQL becoming
hotter and hotter will also induce more ideas and performance/enterprise
optimisations for postgres ...

If MySQL were better than PostgreSQL, you could really do two things :

(1) divert your development effort to MySQL, injecting it with the good
stuff known from the postgres effort
(2) make sure postgres becomes even better !
or
(3) turn away from computers and programming in disgust (not
recommended)

Of course, as things stand now, MySQL has still a long way to run before
it's up to par with PostgreSQL on enterprise-level database features.
But *both* are not yet at 100% !

I do agree we need more publicity for PostgreSQL, I mention it every
time I explain a database backend system to our prospects. I really am
a great fan of postgres and use it exclusively (except to look how
poorly other db's compare with it :-).

[snip]

But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

which in itself is not bad at all.

Look at it this way and perhaps the situation doesn't appear too
negative:

A guesstimate of 90% of all open-source/free software systems and
development is GNU/Linux. Nonetheless, the remaining 10% is *very*
active and are in some ways even ahead of the Linux track. In other
ways they are different. In still other ways, they're behind Linux.
For specific tasks, the *BSD family is vastly superior to Linux.
Everyone should use and support the tools that fit the bill.

For the marketing stuff, what about asking some big company's IT dept
for a statement, sort of "FooBarBank chooses/switches to PostgreSQL open
source database"? Then it's just a matter of making a press release
(wording is very important, anyone proficient in making press releases
here ?) and time them adequately.

I'll ask around here to see whether we can publicize some cases.

Cheers,
Tycho

/* this mail protected by No-Flam(tm) fire retardant asbestos
underwear (owwww itchy itchy) */

--
Tycho Fruru tycho@fruru.com
"Prediction is extremely difficult. Especially about the future."
- Niels Bohr

IMO One of the big reasons that MySQL is viewed as being better is it's
percieved simplicity. It has a large following because of this, and many
of them are not experienced database users, in fact just the opposite.

This large user base is perhaps the best marketing that an open source
project can hope for. So I think that if we want to attract more users
we should try to make postgres easier to use. The hard part is how to do
this without sacrificing the integrity of the project. I think for
starters when evaluating the next feature we want to work on we ask the
following questions:

1) Does it make it easier to use for a non-dba ?
2) Does it facilitate making web-applications easier ( assuming that
this is the largest user base ) ?
3) I'm sure there are others, but at the moment I can't come up with
them.

Then if faced with a choice of implementing something which is going to
make postgres more technically complete or something which is going to
appeal to more users we lean towards more users. Note I said lean!

Dave

On Tue, Jun 25, 2002 at 04:25:24PM +0200, Tycho Fruru wrote:

Everyone should use and support the tools that fit the bill.

I've mentioned before, however, that "the tools that fit the bill" is
partly a function of network effects. The *BSD guys have the same
problem when facing the Linux juggernaut: as one system begins to
dominate the minds of certain types of people who happen to make a
lot of decisions, that system knocks other things out of the running
just by virtue of its PH quotient [1]Pointy-hair quotient: the tendency of a given product name to elicit recognition from a technical manager of dubious technical ability..

For the marketing stuff, what about asking some big company's IT dept
for a statement, sort of "FooBarBank chooses/switches to PostgreSQL open
source database"? Then it's just a matter of making a press release
(wording is very important, anyone proficient in making press releases
here ?) and time them adequately.

Best of luck. Here's the dirty secret about PostgreSQL: _lots_ of
big-ish companies are using it, and using it in important, central
functions of their organisations. But they're not willing to admit
it. What you always get is something like, "Yes, we're using an
enterprise-class system with good ANSI SQL 99 compliance, WAL, hot
backup, triggers, rules, an advanced, extensible datatypes system,
and excellent scalability to high concurrency. The system we're
using, **mumble PostmmuumblehandinfrontofmouthgrSQmllL **mumble**, is
very similar to ORACLE in a lot of respects. We have looked
carefully at ORACLE, and are always aware of the constantly-changing
database marketplace. We have a history of strong relationships with
vendors. . . ." You can substitute your favourite big-name RDBMS.
The point of such utterances seems mostly to be to get the name brand
inserted as often as possible, as though some sort of reflected glory
is the answer.

I don't know why this is. I am, to put it mildly, unbelievably
frustrated (not to say embarrassed) by at least one instance of it.
But it's nevertheless true.

[1]: Pointy-hair quotient: the tendency of a given product name to elicit recognition from a technical manager of dubious technical ability.
elicit recognition from a technical manager of dubious technical
ability.

A

-- 
----
Andrew Sullivan                               87 Mowat Avenue 
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110

OK, I have heard this before, but I would like to know specifically how
is PostgreSQL harder to administer than MySQL. Knowing that will help
us address the issue.

A few things that get to me:

Periodic vacuum and analyze. How often is enough? Whats cron?
Automated garbage collection makes all of the Java people happy ;)

ALTER TABLE / DROP COLUMN
ALTER TABLE / ALTER COLUMN <datatype>
ALTER TABLE / ADD COLUMN (as normal table creation)

Both are useful while putting together a quick system. Especially if
you project the ERD and PGAdmin processes onto a screen if working in a
group for a quick application. Explain while creating.

initdb is a seperate process. I've altered my Solaris startup scripts
to automatically initdb the data directory requested if it doesn't
already exist. Moving this into the postmaster itself, and
automatically generating the space would cut about 50% of the effort out
of the current install process. Install, run. Many installers probably
do this already.

Lastly, and hopefully partially fixed (soon?) is my main problem with --
Dropped that object but something else used it. Now it throws funny
errors.

Upgrades are a bit annoying, but if you've stuck with Postgresql long
enough to get to an upgrade process I'd say you're hooked anyway.

Tom,

project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

We also don't have a couple of other things that MySQL has: A
viciously divided community, a bobby-trapped licensing situation, and a
flagrant disredard for the SQL standard and cumulative wisdom of 25
years of database knowledge. (Nested tables! Sheesh!) These things
handicap the MySQL project quite effectively, and are not likely to be
straightened out in the next year.

BTW, PLEASE DO NOT QUOTE the above. It's ok for the hacker's list,
but I do not want to fuel the MySQL/Postgres "debate" anywhere more
public. This "debate" does not benefit either project.

Also, I am concerned about the focus on MySQL as our "only competitor".
Frankly, I have never regarded MySQL as our primary competitor; that
spot is reserved for Microsoft SQL Server. Especially with the death
of SQL Anywhere, Postgres and MS SQL are the two major databases in the
transaction/vertical application space for the budget-minded business
(although MS SQL is considerably less budget-minded than it was a year
ago).

When we've crushed MS SQL, then it's time to take on Oracle and DB2.

I think there's plenty of room in the RDBMS market for both MySQL and
PostgreSQL. If there's a marketing need, it's to educate DBA's on the
different strengths of the two databases. You think MySQL would
cooperate in this, or do they see themselves as competing head-on with
us?

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

I disagree pretty strongly, Tom. OpenOffice.org Marketing has no
cash, and is an all-volunteer effort. To quote journalist Amy Wohl
"[OpenOffice.org] have managed to put together a better buunch of
volunteer marketers than Sun is able to hire." Frankly, of the various
marketing techniques, only going to trade shows costs money; the rest
is all labor which can be done by volunteers and donors.

Of course, this requires somebody pretty inspired to organize it. I
already have my hands full with OpenOffice.org. Volunteers?

As Great Bridge should have taught you, corporate money for mmarketing
comes with expectations and deadlines attached. Landmark gave you one
shot at "making it", and then yanked the carpet when that didn't pan
out immediately. Other companies are going to be the same. One of
the greatest things about Postgres is that we have been able to outlast
the death of half a dozen companies that supported us, and replace them
with new.

And isn't Red Hat doing anything to promote us?

Finally, thanks to you guys, we are still advancing our project faster
than most commercial software. How many RDBMSs out there have DOMAIN
support? How many have advanced data types that really work? How
many support 5 procedural languages and subselects just abotu
everywhere?

-Josh Berkus

I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.

Suggestion:
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.

The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Here are just a few links that I've come across recently:
How-to for using Open Office and unixODBC
http://www.unixodbc.org/doc/OOoMySQL.pdf

Others are considering MySQL.
http://dba.openoffice.org/proposals/MySQL_OOo.html

James Hubbard

Dave Cramer wrote:

James Hubbard wrote:

I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.

Suggestion:
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.

OK, we just started heavy discussion on this and I believe a few people
are actively working on this. The timeframe is 3-6 months, though we
have the Cygwin solution right now.

The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.

OK, I have heard this before, but I would like to know specifically how
is PostgreSQL harder to administer than MySQL. Knowing that will help
us address the issue.

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Perhaps a config utility that asked you questions and modified template1
and the config files. How about that?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

And, quite frankly, until the BigO loses its grip, I really don't see them
coming out of the closet and admitting to using PgSQL ... why? I don't
know abotu you, but all I can imagine in my head is a horde of O-salesman
descending on the company wondering why they switched and how can they
convince them otherwise, etc, etc ...

I know ... I deal with those salesman all the time, from Oracle to Sun to
Microsoft ...

Yeah, but the lunches are usually pretty good ;)

Anyway, we've kept a trinket system around which covers nearly every big
name required in order to allow marketing to push that we use the
technology (PeopleSoft, Oracle, NT + Sun Clustering, etc.). Honestly,
I'm not sure how essential to the system the CRM is. We'd notice if it
was missing -- but we could live with a couple hours downtime without
any issues.

Fact is, even if we replaced the CRM with another solution an Oracle
based NT box with some application would be running distributed.net in
the corner in order to be able to say we use it when people ask. If
they ask what for, it's always mission critical but very vague.

Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

OK, I want to know, does anyone see MySQL gaining in market share in
comparison to PostgreSQL, or is MySQL gaining against other databases?
Is MySQL gaining sites faster than we are gaining sites?

Every indication I can see is that PostgreSQL is gaining on MySQL.

The Linux/FreeBSD comparison is potent. Does PostgreSQL remain a niche
player? Does *BSD remain a niche player?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Josh Berkus wrote:

Also, I am concerned about the focus on MySQL as our "only competitor".
Frankly, I have never regarded MySQL as our primary competitor; that
spot is reserved for Microsoft SQL Server. Especially with the death
of SQL Anywhere, Postgres and MS SQL are the two major databases in the
transaction/vertical application space for the budget-minded business
(although MS SQL is considerably less budget-minded than it was a year
ago).

When we've crushed MS SQL, then it's time to take on Oracle and DB2.

I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our feature set matches
there's most closely.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

OK, let me make some comments on this. First, Great Bridge had me doing
some marketing stuff while I was with them. This included trade shows,
magazine articles, and interviews. I am available to do all those
again. GB lined up the contacts and got it all started. If people want
me to do more of that, I can find the time.

I am not sure how effective that was. There was a lot more marketing
done by Great Bridge that would take lots of money to do.

Do people want an advocacy article written, like "How to choose a
database?" I could do that.

Basically, I am open to ideas. Would it help to fly me out to meet IT
leaders? More books/articles? What does it take? What do successful
companies and open source projects do that works?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

OK, I have heard this before, but I would like to know specifically how
is PostgreSQL harder to administer than MySQL. Knowing that will help
us address the issue.

I find this statement about pgsql being hard to admin a bit hard to swallow
as well. Maybe it's cause pgsql is all i've _really_ used, and did tech
support
for it for a while.

I've had a few run in's with mysql[ for personal projects] and tried to
approach it with an open mind, but often i find the documentation is
unclear
and hard to follow.

PostgreSQL has GREAT documentation and it feels very straight forward to
run.

The only problem is convincing everyone else that.

Bruce Momjian wrote:

James Hubbard wrote:

I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.

Suggestion:
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.

OK, we just started heavy discussion on this and I believe a few people
are actively working on this. The timeframe is 3-6 months, though we
have the Cygwin solution right now.

I've been watching the activity around this. I have a great deal of hope that
this will produce something. My wife needed a database and web scripting
solution for use on Windows at work and my only suggestion was to use mysql and
php. The binaries are on the website and are easy to download and install.

The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.

OK, I have heard this before, but I would like to know specifically how
is PostgreSQL harder to administer than MySQL. Knowing that will help
us address the issue.

I wasn't really comparing to MySQL here. I meant, in relationship to MS
Access. Start it up and it just works. No worries about configuration files,
etc. I've not used MySQL before except to install it on Windows NT in a VMWare
session. I've been meaning to get back around to playing with to see how well
it functions.

The company that my wife works for is almost exclusively MS. There are a few
file servers that are Novell. I'm sure that the only reason that they are
using MySQL is that it's easy to obtain install and use and use with PHP.

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Perhaps a config utility that asked you questions and modified template1
and the config files. How about that?

I think that would work pretty well. A basic configuration that locks eveything
down with the goal of a single user desktop setting, but also provides the user
with the capability of opening things up so that it could function as a
multiuser system. That then forces the issue of being able to move the
database to a server machine relatively painlessly.

Keep in mind that I was primarily focusing on the potential to include it with
something like OpenOffice. This is why I said that my post was a little far
fetched.

James Hubbard

I'd have to say that personally, given a choice between expending effort
to fix current know bugs and add known needed features, and expending
effort to port to Windows, I'd pick the former, not the latter.

I could personally care less if postgresql ever runs as a native window
application, since I personally don't believe windows is a suitable OS for
hosting a dbms.

Note that the "portablility problems" in postgresql are and were
introduced by Windows deciding to do everything different than every other
OS. Postgresql is quite portable, when one is porting it to OSes that
aren't windows, like VMS, MVS, or all the different flavors of Unix.

Besides, in another 5 years, Windows as a server OS will likely be the
shrinking percentage, while Linux/BSD et. al. will be growing. focus on
the future, and let Windows wither and die (in the server room) as it
should.

On Tue, 25 Jun 2002, James Hubbard wrote:

I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.

Suggestion:
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.

The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Here are just a few links that I've come across recently:
How-to for using Open Office and unixODBC
http://www.unixodbc.org/doc/OOoMySQL.pdf

Others are considering MySQL.
http://dba.openoffice.org/proposals/MySQL_OOo.html

James Hubbard

Dave Cramer wrote:

IMO One of the big reasons that MySQL is viewed as being better is it's
percieved simplicity. It has a large following because of this, and many
of them are not experienced database users, in fact just the opposite.

This large user base is perhaps the best marketing that an open source
project can hope for. So I think that if we want to attract more users
we should try to make postgres easier to use. The hard part is how to do
this without sacrificing the integrity of the project. I think for
starters when evaluating the next feature we want to work on we ask the
following questions:

1) Does it make it easier to use for a non-dba ?
2) Does it facilitate making web-applications easier ( assuming that
this is the largest user base ) ?
3) I'm sure there are others, but at the moment I can't come up with
them.

Then if faced with a choice of implementing something which is going to
make postgres more technically complete or something which is going to
appeal to more users we lean towards more users. Note I said lean!

Dave

On Tue, 2002-06-25 at 01:21, Tom Lane wrote:

Josh Berkus <josh@agliodbs.com> writes:

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.

Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
"Force has no place where there is need of skill.", "Haste in every
business brings failures.", "This is the bitterest pain among men, to have
much knowledge but no power." -- Herodotus

When we've crushed MS SQL, then it's time to take on Oracle and DB2.

I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our feature set matches
there's most closely.

But if we had a native windows port, I think we would here of a lot more MS
SQL Server converts.

Also, I think that expending effort on a windows port will be a net win as it
it will generate a new userbase and with it more developers. Good or bad,
windows has a large marketshare.

On Tue, 2002-06-25 at 22:48, Josh Berkus wrote:

Bruce,

I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our feature set matches
there's most closely.

I disagree, and did as well when you were with Great Bridge. No matter how
Postgres core functionality compares with Oracle, they have nearly a decade
of building tools, accessories, and extra whiz-bang features for their
product.

My (perhaps a little outdated) experience has been that with Oracle
almost anything on the client side sucks bad.

What they have is a solid database and good upward path to really big
iron.

Not to mention a serious reputation as the "ultimate database if
you can afford it."

On PC server class computers we seem to be able to match them with one
exception - prepared statements with good(?) binary fe/be protocol ?

As long as we target Oracle as our "competition", we will remain "the database
to use if you can't afford Oracle, but to be replaced with Oracle as soon as
you can." Heck, look at DB2, which is toe-to-toe with Oracle for feature
set, but is only really sold to companies who use IBM's other tools. We're
not in a position to challenge that reputation.

But if we are seen as challenging it, it is a good marketing point when
selling to MS SQL folks :)

On the other hand, we already outstrip MS SQL Server's feature set, as well as
being more reliable, lower-maintainence, multi-platform, and cheaper.

If only someone were to write Transact SQL lookalike and even better -
if we had pluggable frontend protocols - FreeTDS compatibility on server
side would be a big step even without native Win32.

Frankly, the only thing that MS SQL has over us is easy-but-unreliable GUI
admin tools (backup, user, and database management).

We almost have it in pgAdmin and Tora.

---------------
Hannu

James,

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Personally, I think it's a redundant idea. There are a couple dozen
"lightweight" RDBMSs available off Sourceforge. There is only one
"Heavy-duty" database: Us.

Others are considering MySQL.
http://dba.openoffice.org/proposals/MySQL_OOo.html

Let me nip this in the bud: That proposal was shot down almost immediately,
mostly due to MySQL's poor adherence to the SQL standard and licensing
problems. I also shot down PostgreSQL as a possibility for inclusion with
OpenOffice.org, since Postgres is quite firmly a *server* database, and 70%
of OpenOffice.org installs are on Windows 95/98.

Currently, we are leaning toward HSQLDB as our included database. However,
you can help us decide: join the DBA.openoffice.org project
(http://dba.openoffice.org/).

Something we could really, really use for OpenOffice.org is "native" (SBDC)
drivers for PostgreSQL. Currently, we have to use UnixODBC or MS ODBC, which
brings all sorts of problems with it. Can anyone help with a driver?

Once we get a native driver, OpenOffice.org will be available as an MS
Access-style tool for simple PostgreSQL database management. This should
increase adoption of Postgres somewhat.

-Josh Berkus
OpenOffice.org

Bruce,

I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our feature set matches
there's most closely.

I disagree, and did as well when you were with Great Bridge. No matter how
Postgres core functionality compares with Oracle, they have nearly a decade
of building tools, accessories, and extra whiz-bang features for their
product. Not to mention a serious reputation as the "ultimate database if
you can afford it."

As long as we target Oracle as our "competition", we will remain "the database
to use if you can't afford Oracle, but to be replaced with Oracle as soon as
you can." Heck, look at DB2, which is toe-to-toe with Oracle for feature
set, but is only really sold to companies who use IBM's other tools. We're
not in a position to challenge that reputation.

On the other hand, we already outstrip MS SQL Server's feature set, as well as
being more reliable, lower-maintainence, multi-platform, and cheaper.
Frankly, the only thing that MS SQL has over us is easy-but-unreliable GUI
admin tools (backup, user, and database management).

Let's pick battles we can win. We'll beat Oracle eventually -- but not in the
next few years.

-Josh Berkus

-----Original Message-----
From: Josh Berkus [mailto:josh@agliodbs.com]
Sent: Tuesday, June 25, 2002 10:49 AM
To: Bruce Momjian
Cc: Tom Lane; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Democracy and organisation : let's make a

Bruce,

I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our

feature set matches

there's most closely.

I disagree, and did as well when you were with Great Bridge.
No matter how
Postgres core functionality compares with Oracle, they have
nearly a decade
of building tools, accessories, and extra whiz-bang features
for their
product. Not to mention a serious reputation as the
"ultimate database if
you can afford it."

As long as we target Oracle as our "competition", we will
remain "the database
to use if you can't afford Oracle, but to be replaced with
Oracle as soon as
you can." Heck, look at DB2, which is toe-to-toe with Oracle
for feature
set, but is only really sold to companies who use IBM's other
tools. We're
not in a position to challenge that reputation.

On the other hand, we already outstrip MS SQL Server's
feature set, as well as
being more reliable, lower-maintainence, multi-platform, and
cheaper.
Frankly, the only thing that MS SQL has over us is
easy-but-unreliable GUI
admin tools (backup, user, and database management).

Let's pick battles we can win. We'll beat Oracle eventually
-- but not in the
next few years.

If you want to aim high, aim at DB/2.
;-)

DB/2 is the best database on the market in terms of its features. It's
highly scalable, runs everywhere, and has an add-on for anything you can
imagine. I absolutely love DB/2. I suspect a market share study will
show that DB/2 is eating Oracle alive in the past couple years.

There is a benefit to setting your sights high. Look at the features of
the really rich products. Take the ones that are truly possible to
implement in a feasible time frame and add them to the project goals.
Eventually, you can have every feature of the most advanced DBMS systems
in that way.

I do have a suggestion as to strategic thinking on how to improve
PostgreSQL. Instead of adding a huge array of features on the to-do
list, attack the weak points of the current product. Here are the three
most important aspects of any DBMS system:
1. Speed
2. Reliability
3. Security

Look at places in the current product where these can be improved.
Think about this for a minute --
If your product is faster, it will appeal to a large crowd for that
reason alone.
If your product is more reliable, it will appeal to a large crowd for
that reason alone.
If your product is more secure, it will appeal to a large crowd for that
reason alone.
None is less important than the others. If any of the above 3 features
are seriously lacking, nobody will want to use the tool.

Bells and whistles are nice, but you still want the bicycle to go as
fast as possible while not breaking down and keeping the rider safe.

So my suggestion for project direction would be as follows:
2/3 effort on improving functionality of the core in the above three
areas.
1/3 effort on adding new features.

Now, I have had very little to contribute myself, and therefore my
suggestions carry no weight. The opinions of the contributors are more
important than someone from the outside looking in. But it is worth
mulling over at least.

There is one feature to PostgreSQL that does seem to be missing that is
found in all the commercial systems that I have worked with (and so I
would suggest adding it to the 'new features' list). That is the
ability to perform stored procedures that return one or more row sets.
The PostgreSQL paradigm of a stored procedure seems to me to be really
limited to a function call. Perhaps it is already being worked on.

Just a personal observation here, based on the work we've been doing
lately ... there are *alot* of very large companies out there, one of
which we just did onsite training for that I swear has an article in just
about every trade magazine I read, each month ... the problem isn't
getting companies to adopt/use PgSQL ... the problem is getting them to
acknowledge its usage ...

And, quite frankly, until the BigO loses its grip, I really don't see them
coming out of the closet and admitting to using PgSQL ... why? I don't
know abotu you, but all I can imagine in my head is a horde of O-salesman
descending on the company wondering why they switched and how can they
convince them otherwise, etc, etc ...

I know ... I deal with those salesman all the time, from Oracle to Sun to
Microsoft ...

The problem, as I see it, is everyone moaning cause we aren't the #1
database for the Web ... who cares? How many sites out there don't even
*need* a database backend in the first place? Someone throws MySQL onto
that and thiks its the best thing since sliced bread, even though the
table contains a single record ...

The markets that matter, enterprise databases, we are making inroads into
and quite substantial ones, but due to 'internal politics', you aren't
going to hear about them ...

How many ppl here can honestly say they know of *at least* one company, if
not more, that are using PgSQL, but don't advertise, or let known, that
fact? I can think of a half dozen that we (PgSQL, Inc) have worked with
to convert, and train, so far ... Tom, call it RedHat DB or PgSQL, its the
same code base ... any numbers from that end? Bruce, how about from SRA?

On Tue, 25 Jun 2002, Bruce Momjian wrote:

Tom,

project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

We also don't have a couple of other things that MySQL has: A
viciously divided community, a bobby-trapped licensing situation, and a
flagrant disredard for the SQL standard and cumulative wisdom of 25
years of database knowledge. (Nested tables! Sheesh!) These things
handicap the MySQL project quite effectively, and are not likely to be
straightened out in the next year.

BTW, PLEASE DO NOT QUOTE the above. It's ok for the hacker's list,
but I do not want to fuel the MySQL/Postgres "debate" anywhere more
public. This "debate" does not benefit either project.

Oh, but it's _so_ tempting :-).

Also, I am concerned about the focus on MySQL as our "only competitor".
Frankly, I have never regarded MySQL as our primary competitor; that
spot is reserved for Microsoft SQL Server. Especially with the death
of SQL Anywhere, Postgres and MS SQL are the two major databases in the
transaction/vertical application space for the budget-minded business
(although MS SQL is considerably less budget-minded than it was a year
ago).

When we've crushed MS SQL, then it's time to take on Oracle and DB2.

Take on SQL Server, and establish a sizable useful niche. The notion that
PostgreSQL is _necessarily_ supposed to be all things to all people promotes
the danger of getting over-arrogant and over-ambitious.

I think there's plenty of room in the RDBMS market for both MySQL and
PostgreSQL. If there's a marketing need, it's to educate DBA's on the
different strengths of the two databases. You think MySQL would
cooperate in this, or do they see themselves as competing head-on with
us?

Why _should_ they want to cooperate?

Their advantage in the marketplace is largely based on the notion that
"MySQL isn't quite as good as Oracle, but it's a lot cheaper!"

For them to say, "and by the way, PostgreSQL, SAPDB, and Firebird are all
basically the same that way" would be shooting themselves in the foot.

Their model is rather like that of Microsoft Access: It's not all that great,
but it gets used a lot, despite its limitations, because everyone has a copy
of it as part of MS Office.

For them to "cooperate" would mean compromising on what's most important to
their ongoing marketing strategy:
"Use MySQL because it's the most popular database!"

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

I disagree pretty strongly, Tom. OpenOffice.org Marketing has no
cash, and is an all-volunteer effort. To quote journalist Amy Wohl
"[OpenOffice.org] have managed to put together a better buunch of
volunteer marketers than Sun is able to hire." Frankly, of the various
marketing techniques, only going to trade shows costs money; the rest
is all labor which can be done by volunteers and donors.

Of course, this requires somebody pretty inspired to organize it. I
already have my hands full with OpenOffice.org. Volunteers?

The _crucial_ marketing that would need to take place is NOT to the public.
It would be to:
a) ISPs
b) Vendors of ISP support software.

The sort of thing that has allowed MySQL to get really popular is the fact
that there are tools like cPanel <http://www.cpanel.net/&gt; that provide a
"friendly" front end to manage web site 'stuff,' including managing MySQL.

And isn't Red Hat doing anything to promote us?

They ought to be...

Finally, thanks to you guys, we are still advancing our project faster
than most commercial software. How many RDBMSs out there have DOMAIN
support? How many have advanced data types that really work? How
many support 5 procedural languages and subselects just abotu
everywhere?

... And this is what is the plausible strategy for making PostgreSQL
increasingly popular. _Improve it_ and people will come.

MySQL won the "basic DBMS for web-hosting" battle, and there's no real way to
overcome that _marketing_ advantage. MySQL got there the "fustest with the
mostest," with things like cPanel allowing ISPs and web hosters to offer a
free DBMS.

PostgreSQL can offer "the same thing;" to evict MySQL, it will have to offer
_really compelling_ advantages. Price _isn't_ a compelling advantage.
PostgreSQL may be more powerful, but people are successfully using MySQL, so
apparently it's _usable enough_ for a lot of purposes.

The other thing that can make PostgreSQL an increasingly preferable option is
for there to be an increasing set of _applications_ that prefer PostgreSQL.

For instance, GnuCash has an SQL interface, or, to be more precise, a
PostgreSQL interface. The makers of GnuCash found they preferred PostgreSQL's
capabilities, and are uninterested in supporting a bunch of DBMSes. Somewhat
similar, SQL-Ledger is compatible with PostgreSQL (and Oracle), but NOT MySQL.

The thing that will make PostgreSQL the "killer app" that needs to be around
is there being _applications_ that "prefer PostgreSQL." THAT is the best
marketing.
--
(reverse (concatenate 'string "ac.notelrac.teneerf@" "454aa"))
http://cbbrowne.com/info/lsf.html
Everyone has a photographic memory, some don't have film.

Marc,

Just a personal observation here, based on the work we've been doing
lately ... there are *alot* of very large companies out there, one of
which we just did onsite training for that I swear has an article in just
about every trade magazine I read, each month ... the problem isn't
getting companies to adopt/use PgSQL ... the problem is getting them to
acknowledge its usage ...

Yeah. I know one database-backed application, used by about 40% of the pople
in this city, which runs on PostgreSQL. However, the company that built
that application won't let me publicize their usage because they are worried
about getting political flack from Oracle and Microsoft's lobbyists at City
Hall.

--
-Josh Berkus

On Tue, Jun 25, 2002 at 02:34:59PM -0400, cbbrowne@cbbrowne.com wrote:

The _crucial_ marketing that would need to take place is NOT to the public.
It would be to:
a) ISPs
b) Vendors of ISP support software.

The sort of thing that has allowed MySQL to get really popular is the fact
that there are tools like cPanel <http://www.cpanel.net/&gt; that provide a
"friendly" front end to manage web site 'stuff,' including managing MySQL.

One consideration is that prior to 7.3, PostgreSQL's permissions scheme
made it difficult or impossible use in a shared-hosting environment (or
at least, that's what I've heard from several different people -- I
don't have any personal experience).

I'm aware that there are people offering PostgreSQL hosting, but the
*perception* among the hosting techies I've talked to is that MySQL's
feature set is better suited for a shared hosting environment. With
schemas and improved permissions in 7.3, that may be a thing of the
past (at which point, ISPs might be a prime area for marketing).

Cheers,

Neil

--
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

Folks,

Both are useful while putting together a quick system. Especially if
you project the ERD and PGAdmin processes onto a screen if working in a
group for a quick application. Explain while creating.

BTW, as nice as PG-Admin is, I do not consider it a solution to our DB
management GUI desires. It only runs on Windows, and cannot be ported to
*nix. :-(

Weren't we ressurecting PGAccess?

--
-Josh Berkus

Hi

-*- Hannu Krosing <hannu@tm.ee> [ 2002-06-25 21:34 ]:

On Tue, 2002-06-25 at 22:48, Josh Berkus wrote:

Frankly, the only thing that MS SQL has over us is easy-but-unreliable GUI
admin tools (backup, user, and database management).

We almost have it in pgAdmin and Tora.

Well, pgAdmin has come a long way -- some of my fellow admins that have Windows workstations use it, and does the developers at my company. Don't know Tora thought, must try it out sometime.

However, I think some sort of scheduling like MS SQL offers would be very helpful for newcomers. Scheduling stuff like backup (now only dumps), vacuum's and such with cron is unfortunately not straight-forward enough.

Merging configuration files is good. However, things like access control should IMO be configurable with SQL commands -- which would also help in development of better administration tools.

Just my two cents.

Regards,
Tolli,
tolli@tol.li

Hi James,

James Hubbard wrote:

<snip>

Keep in mind that I was primarily focusing on the potential to include it with
something like OpenOffice. This is why I said that my post was a little far
fetched.

My understanding of this is that the OpenOffice.org guys don't want
either PostgreSQL nor MySQL as their inbuilt database, but are instead
looking at an altervative Open Source database (HSQL I think, don't
remember for sure).

Doing just what you proposed (getting a Win32 version of PostgreSQL and
offering to the OpenOffice.org people) was suggested to NuSphere a few
months ago, after Jan joined them. For some reason (not sure why) it
wasn't something which they decided to pursue.

Good suggestion though James. :-)

Regards and best wishes,

Justin Clift

Ok, a few comments on various messages that have appeared in this thread.

From: James Hubbard <jhubbard@mcs.uvawise.edu>

I wasn't really comparing to MySQL here. I meant, in relationship
to MS Access. Start it up and it just works.

Yeah, a point-and-drool installation wizard for postgres under windows
would be great. I think, from looking at PGAdminII, that we've already
got great admin tools; it seems just as good as SQL Server Enterprise
Manager to me.

I think that would work pretty well. A basic configuration that
locks eveything down with the goal of a single user desktop setting,
but also provides the user with the capability of opening things up
so that it could function as a multiuser system.

I don't understand this. What's the difference between a "single
user desktop setting" and a low-end multi-user system? I don't see
what would change.

If you're talking more than a twenty or thirty active connections and
a couple of gig of data, yeah, then you need to change stuff. But then
you need a real admin and some planning, and no point-and-click tool is
going to help with that.

Keep in mind that I was primarily focusing on the potential to include
it with something like OpenOffice. This is why I said that my post was
a little far fetched.

That sounds like a great idea to me.

From: Scott Marlowe <scott.marlowe@ihs.com>

I could personally care less if postgresql ever runs as a native window
application, since I personally don't believe windows is a suitable OS for
hosting a dbms.

Well, windows is fine for hosting a DBMS if you're talking about the
facilities the OS offers a DBMS, and effeciency. Administrating windows
boxes sucks, but cygwin can help fix that. (Not that I'd care to go back
to a database running under Windows, but it is practical, if unpleasant.)

Postgresql is quite portable, when one is porting it to OSes that
aren't windows, like VMS, MVS, or all the different flavors of Unix.

I'm not sure what's up with this. Windows does offer POSIX compatability,
after all.

From: Josh Berkus <josh@agliodbs.com>

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Personally, I think it's a redundant idea. There are a couple dozen
"lightweight" RDBMSs available off Sourceforge. There is only one
"Heavy-duty" database: Us.

And what on earth is the advantage of "PostgreSQL Lite"? I don't see how
it would be easier to use in any way. The install dificulties could be
worked around with an install wizard, and PGAdminII seems already to be
a good admin interface.

I also shot down PostgreSQL as a possibility for inclusion with
OpenOffice.org, since Postgres is quite firmly a *server* database, and 70%
of OpenOffice.org installs are on Windows 95/98.

Again, I don't see the problem. Server, schmerver; there's nothing wrong
with running postgres for "non-server" tasks. Unless it's completely
impossible to port to Win98, but is that really the case?

From: Josh Berkus <josh@agliodbs.com>

On the other hand, we already outstrip MS SQL Server's feature set,
as well as being more reliable, lower-maintainence, multi-platform,
and cheaper. Frankly, the only thing that MS SQL has over us is
easy-but-unreliable GUI admin tools (backup, user, and database
management).

Uh..."no way." I've found MS SQL Server is consistently faster when it
comes to the crunch, due to things like writing a heck of a lot less
to the log files, significantly less table overhead, having clustered
indexes, and so on. (Probably more efficient buffer management also
helps a bit.) Other areas where postgres can't compare is backup and
restore, ability to do transaction log shipping, replication, access
rights, disk allocation (i.e., being able to determine on which disk
you're going to put a given table), and so on. SQL Server's optimizer
also seems to me to be better, though I could be wrong there.

cjs
--
Curt Sampson <cjs@cynic.net> +81 90 7737 2974 http://www.netbsd.org
Don't you know, in this new Dark Age, we're all light. --XTC

BTW, as nice as PG-Admin is, I do not consider it a solution to our DB
management GUI desires. It only runs on Windows, and cannot be ported to
*nix. :-(

Weren't we ressurecting PGAccess?

What other development options do we have for soemthing that is GUI and
portable to all platforms that postgresql runs on? Java? wxWindows? Qt?
Gtk? I would think that Gtk is probably the most portable, and it has
bindings to many languages, but we would probalby want to use C.