WIN32 native ... lets start?!?

Actually, take a look at the thread starting at:

http://archives.postgresql.org/pgsql-hackers/2002-05/msg00665.php

Right now, IMHO, the big show stopper is passing global variables to the
child processes in Windows ... the above thread talks about a method of
pulling together the global variables *cleanly* that Tom seems to feel
wouldn't add much in the way of long term maintenance headaches ... *and*,
as I understand it, would provide us with a means to use threading in
future developments if deemed appropriate ...

From what I read by those 'in the know' about Windows programming, if we
could centralize the global variables somewhat, using CreateProcess in
Windows shouldn't be a big deal, eliminiating the whole fork() headache
...

On Thu, 16 May 2002, Joerg Hessdoerfer wrote:

On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:

So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.

Getting it to compile and then "somehow" run on MinGW seems a good first
step on road to full native Win32 PG.

----------
Hannu

On Thu, 2002-05-16 at 13:47, Joerg Hessdoerfer wrote:

So, my route would be to get it to run *somehow* without paying
attention to speed and not to change much of the existing code,
THEN see how we could get rid of fork() on windows.

What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying the memory allocations.
How fast that would be regarding the context switches, i don't know, but i'm willing to experiment some to see how feesible this is...

Anyone tried this before?

Magnus

On Thursday 16 May 2002 22:10, you wrote:
[...]

What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying
the memory allocations. How fast that would be regarding the context
switches, i don't know, but i'm willing to experiment some to see how
feesible this is...

Anyone tried this before?

Magnus

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on windows, it's
creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg.hessdoerfer@sea-gmbh.com
WWW: http://www.sea-gmbh.com

Joerg Hessdoerfer <Joerg.Hessdoerfer@sea-gmbh.com> wrote:
[snip]

The problem is not the fork() call itself, this has been done (MinGW
and cygwin I know of, possibly others) but the speed of fork() on
windows, it's creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

I'm not so familiar with the win32 kernel mode stuff.
But i've seen programs using .vxd (kernelmode, ring X ?) helpers for getting more privileges, maybe cross process ones.
Well, i'll look into this sometime if it's possible to reduce the context switches by going vxd.
There must be some way to read protection of the pages and map them as COW or RO in the new process to get rid of much of the copy, but then again, we're talking microsoft here :)
I once did a .exe loader that used the MapViewOfFile (win32 mmap) of the .exe itself to accomplish shared loadable modules that worked on x86 linux and win32 without recompile (might be something like the XFree86 binary gfx card drivers).

Good luck on the IPC work!

Magnus

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Chris

On Fri, 17 May 2002, Christopher Kings-Lynne wrote:

Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

"Marc G. Fournier" <scrappy@hub.org> writes:

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

On Friday 17 May 2002 22:16, you wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?

Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

Yes, indeed. I would also like to discuss matters on this list, as one get's
a 'heads up' from people in the know much easier.

BTW, I'm in the process of doing the 'really only what is necessary for pg'
ipc-stuff, and was wondering if anybody already did some configuration of the
source tree towards MinGW?? How should we go about that? I would rather like
not using cygwin's sh for that ;-), and we have no 'ln' !!

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: joerg.hessdoerfer@sea-gmbh.com
WWW: http://www.sea-gmbh.com

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Tuesday, June 04, 2002 9:34 PM
To: PostgreSQL-development
Subject: [HACKERS] Roadmap for a Win32 port

OK, I think I am now caught up on the Win32/cygwin
discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies
working on Win32
ports of PostgreSQL. We always said there would not be
PostgreSQL forks
if we were doing our job to meet user needs. Well,
obviously, a number
of groups see a need for a better Win32 port and we aren't
meeting that
need, so they are. I believe this is one of the few cases
where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users.
We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

Nothing else. It is better than any commercial tools in current use.
An excellent piece of work.

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

Do not try to do a fork() on Win32. The one at PW32 is better, but
still awful. Win32 just does not have fascilities for fork().

If you use Cygwin, it will kill the project for commercial use (at least
for many institutions). That's fine, but it will become an academic
exercise instead of a viable commercial tool. If they are comfortable
in that [Cygwin] environment, it makes no sense to use Cygwin instead of
Redhat. The Redhat version will fork() 100 times faster. After all, if
they are going to use unix tools in a unix interface with Unix scripts
you might as well use UNIX. And Cygwin requires a license for
commercial use.
http://cygwin.com/licensing.html

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

CreateProcess() works well for Win32. That is the approach that we used
and also the approach used by the Japanese team.
It is very simple. Simply do a create process call and then perform the
same operations that were done up to that point. It isn't difficult.
Threading is another possibility. I think create process is better,
because you can clone the rights of the one who attaches for the spawned
server (if you want to do that).

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

We wrote our own from scratch. Cygwin will kill it. If there is a
MinGW version it might be OK, but if MinGW is GPL, that will kill it.
Have a look at ACE:
http://www.cs.wustl.edu/~schmidt/ACE.html
Their license is on the same level as a BSD license. Now, they use C++,
but you can always write:
extern "C" {
}
wrappers for stuff and keep PostgreSQL itself in pure, vanilla C. GCC
does come with a C++ compiler, so it isn't going to cut anyone off.

ENVIRONMENT

Lots of our code requires a Unix shell and utilities. Will
we continue
using cygwin for this?

We wrote our own utilities from scratch (e.g. initdb). The Japanese
group that did the port did the same thing.

--------------------------------------------------------------
-------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

If you want a roadmap, the Japanese group laid it out for you. They
did the exact same steps as we did. Now, I don't know if we will be
able to contribute or not (it is very much up in the air). And we had
to do a lot of hacking of the source, so you might not want it if we
volunteered.

Suggestion:
Ask the Japanese group if they would like to post their changes back or
expose them so that the programming team can get ideas form it.

I actually like what they did better than what we did (A giant DLL and
all the binaries are microscopic -- it was how I suggested to do it here
but it was vetoed).

Anyway, here is a roadmap laid out for you exactly. Just do what it
says and you will be fine:
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html

Look at where it says "Gists for patch" and do that.

On Wed, 2002-06-05 at 16:33, Bruce Momjian wrote:

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.

-- 
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 853-3000
+  If your life is a hard drive,     |  830 Blythe Avenue
+  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

regards

Mark

Dan,

The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.

On Tue, Jun 04, 2002 at 10:02:14PM -0700, Dann Corbit wrote:

And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html

The above is not necessarily true:

Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.

Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.

Jason

Jason Tishler wrote:

Dan,

The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.

On Tue, Jun 04, 2002 at 10:02:14PM -0700, Dann Corbit wrote:

And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html

The above is not necessarily true:

Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.

Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.

So apps written using client libraries are BSD, while server-side
changes would have to release source. Makes sense, though we have never
had this distinction before. I assume plpgsql stored procedures would
have be open source, but of course those are stored in plaintext on the
server so that isn't a problem. If companies created custom C stored
procedures, those would have to be open source if using cygwin.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Mark kirkwood wrote:

Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

Absolutely worth seeing how MySQL does it. They use cygwin, and I
assume they aren't seeing the fork() issue because they are threaded,
and perhaps they don't use SysV IPC like we do.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

-- igor

----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "PostgreSQL-development" <pgsql-hackers@postgresql.org>
Sent: Tuesday, June 04, 2002 11:33 PM
Subject: [HACKERS] Roadmap for a Win32 port

OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

--------------------------------------------------------------------------

-

Igor Kovalenko wrote:

I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

...

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.

I would think that we would build on our strengths of having a fork/exec
model for separate clients. A threaded model *could* benefit individual
clients who are doing queries on multiprocessor servers, and I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well supported
targets. If it came for free then great, but if it came with a high cost
then the choice is not as obvious. It is also not a *requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

- Thomas

-----Original Message-----
From: Thomas Lockhart [mailto:thomas@fourpalms.org]
Sent: Wednesday, June 05, 2002 3:03 PM
To: Bruce Momjian
Cc: Igor Kovalenko; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port

...

Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a

web backend

where many sessions are a single query, people may want to

give up the

stability of fork() and go with threads, even on Unix.

I would think that we would build on our strengths of having
a fork/exec
model for separate clients. A threaded model *could* benefit
individual
clients who are doing queries on multiprocessor servers, and
I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well
supported
targets. If it came for free then great, but if it came with
a high cost
then the choice is not as obvious. It is also not a
*requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

Notion:
Have one version do both. Your server can fork(), and your sever can
thread. It can fork() and thread, it can fork() or thread.

That gives the best of all worlds. One client who has his attachments
to a database all setup might want to do a bunch of similar queries.
Hence a threaded model is nice.

A server may be set up to clone the rights of the attaching process for
security reasons. Then you launch a new server with fork().