HEADS UP: Win32/OS2/BeOS native ports

"Marc G. Fournier" wrote:

Morning all ...

Just a heads up that over the next little while, I'm planning on
making a bunch of commits in order to work on making the code able to work
natively in the above environments ... my work will mostly focus on Win32
(since I have no OS2/BeOS installs), but alot of the changes will be such
that it will benefit the others as well ...

The initial changes will be to just wrapper all our shared memory
code, so that I can make use of Apache's libapr libraries *if* they are
installed ... if not, it will just fall back to "the current code" ...

If you want any assistance, drop me an email. I spent a long time (> decade)
doing Windows applications and drivers and know a good number of the cool
tricks.

On Fri, 3 May 2002, mlw wrote:

"Marc G. Fournier" wrote:

Morning all ...

Just a heads up that over the next little while, I'm planning on
making a bunch of commits in order to work on making the code able to work
natively in the above environments ... my work will mostly focus on Win32
(since I have no OS2/BeOS installs), but alot of the changes will be such
that it will benefit the others as well ...

The initial changes will be to just wrapper all our shared memory
code, so that I can make use of Apache's libapr libraries *if* they are
installed ... if not, it will just fall back to "the current code" ...

If you want any assistance, drop me an email. I spent a long time (> decade)
doing Windows applications and drivers and know a good number of the cool
tricks.

hrmmmm ... do you have a working Windows development environment? I'm
running WinXP at home, but don't have any of the compilers or anything
yet, so all my work for the first part is going to be done under Unix ...

but someone that knows something about building makefiles for Windows, and
compiling under it, will definitely be a major asset ;)

Will there really be a need for a BeOS development with the sale of Be to
Palm? Is BeOS even still available? It might not be worth the time to
develop for BeOS until you see what Palm decides to do with the software.

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of Marc G. Fournier
Sent: Friday, May 03, 2002 9:48 AM
To: mlw
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports

On Fri, 3 May 2002, mlw wrote:

"Marc G. Fournier" wrote:

Morning all ...

Just a heads up that over the next little while, I'm planning

on

making a bunch of commits in order to work on making the code able to

work

natively in the above environments ... my work will mostly focus on

Win32

(since I have no OS2/BeOS installs), but alot of the changes will be

such

that it will benefit the others as well ...

The initial changes will be to just wrapper all our shared

memory

code, so that I can make use of Apache's libapr libraries *if* they

are

installed ... if not, it will just fall back to "the current code" ...

If you want any assistance, drop me an email. I spent a long time (>

decade)

doing Windows applications and drivers and know a good number of the

cool

tricks.

hrmmmm ... do you have a working Windows development environment? I'm
running WinXP at home, but don't have any of the compilers or anything
yet, so all my work for the first part is going to be done under Unix ...

but someone that knows something about building makefiles for Windows, and
compiling under it, will definitely be a major asset ;)

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

On Fri, 3 May 2002, Travis Hoyt wrote:

Will there really be a need for a BeOS development with the sale of Be to
Palm? Is BeOS even still available? It might not be worth the time to
develop for BeOS until you see what Palm decides to do with the software.

Note that the changes I'm making are to make use of what is available
through the libapr API that the Apache group has developed ... so, as long
as they have the hooks in for BeOS, we will ... doesn't mean PgSQL will
actually have makefiles for, and will compile under it, unless someone
*with* BeOS steps forward, but alot of the core functionality that has
held back native ports should work ...

"Marc G. Fournier" <scrappy@hub.org> writes:

The initial changes will be to just wrapper all our shared memory
code, so that I can make use of Apache's libapr libraries *if* they are
installed ... if not, it will just fall back to "the current code" ...

I think we should redesign the shared memory API (and even more so the
semaphore API), not just put a wrapper layer on it. A lot of the
internal API is unnecessarily dependent on SysV shmem/sem behavior.

Note however that there are some things you will break if you are not
very careful. We are depending on shmem/sem behavior to catch a number
of multiple-postmaster conflict situations. If there's not a more or
less SysV-ish kernel underneath us, those situations will have to be
rethought and some other interlock invented.

In short, I want to see a design review first, not a bunch of
off-the-cuff commits.

regards, tom lane

Hi Marc,

How about using Dev-C++?

It's a Windows IDE with a GCC backend, and has a nice rep (and a Linux
port):

http://sourceforge.net/projects/dev-cpp/

It's always in SF.net's "Top 10" most worked on projects too, with about
roughly 7,000 downloads per day. It can generate mingwin code too.

:-)

Regards and best wishes,

Justin Clift

"Marc G. Fournier" wrote:

On Fri, 3 May 2002, mlw wrote:

"Marc G. Fournier" wrote:

Morning all ...

Just a heads up that over the next little while, I'm planning on
making a bunch of commits in order to work on making the code able to work
natively in the above environments ... my work will mostly focus on Win32
(since I have no OS2/BeOS installs), but alot of the changes will be such
that it will benefit the others as well ...

The initial changes will be to just wrapper all our shared memory
code, so that I can make use of Apache's libapr libraries *if* they are
installed ... if not, it will just fall back to "the current code" ...

If you want any assistance, drop me an email. I spent a long time (> decade)
doing Windows applications and drivers and know a good number of the cool
tricks.

hrmmmm ... do you have a working Windows development environment? I'm
running WinXP at home, but don't have any of the compilers or anything
yet, so all my work for the first part is going to be done under Unix ...

but someone that knows something about building makefiles for Windows, and
compiling under it, will definitely be a major asset ;)

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

On Fri, 3 May 2002, Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

The initial changes will be to just wrapper all our shared memory
code, so that I can make use of Apache's libapr libraries *if* they are
installed ... if not, it will just fall back to "the current code" ...

I think we should redesign the shared memory API (and even more so the
semaphore API), not just put a wrapper layer on it. A lot of the
internal API is unnecessarily dependent on SysV shmem/sem behavior.

Note however that there are some things you will break if you are not
very careful. We are depending on shmem/sem behavior to catch a number
of multiple-postmaster conflict situations. If there's not a more or
less SysV-ish kernel underneath us, those situations will have to be
rethought and some other interlock invented.

In short, I want to see a design review first, not a bunch of
off-the-cuff commits.

All I'm planning on doing is changing the appropriate shm_* functions iwth
pg_shm_* functions ... if !(libapr), all those pg_shm_* functions will
have in them is the original call we've always used ... there will even be
a --disable-libapr configure option so that if someone already has Apache2
installed, but doesn't wanna use libapr for PgSQL, they don't have to ...

Basically, all I'm looking at is allowing PgSQL to use a different library
for its shared memory calls then the standard one, nothing else ...

"Marc G. Fournier" <scrappy@hub.org> writes:

All I'm planning on doing is changing the appropriate shm_* functions iwth
pg_shm_* functions ... if !(libapr), all those pg_shm_* functions will
have in them is the original call we've always used ... there will even be
a --disable-libapr configure option so that if someone already has Apache2
installed, but doesn't wanna use libapr for PgSQL, they don't have to ...

Basically, all I'm looking at is allowing PgSQL to use a different library
for its shared memory calls then the standard one, nothing else ...

Oh. I guess my next question is how closely that Apache library
emulates the SysV shmem semantics. In particular, can you reliably
tell how many processes are attached to a shmem block? (Cf
SharedMemoryIsInUse() in storage/ipc/ipc.c) Without that feature we
have an interlock problem.

regards, tom lane

On Fri, 3 May 2002, Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

All I'm planning on doing is changing the appropriate shm_* functions iwth
pg_shm_* functions ... if !(libapr), all those pg_shm_* functions will
have in them is the original call we've always used ... there will even be
a --disable-libapr configure option so that if someone already has Apache2
installed, but doesn't wanna use libapr for PgSQL, they don't have to ...

Basically, all I'm looking at is allowing PgSQL to use a different library
for its shared memory calls then the standard one, nothing else ...

Oh. I guess my next question is how closely that Apache library
emulates the SysV shmem semantics. In particular, can you reliably
tell how many processes are attached to a shmem block? (Cf
SharedMemoryIsInUse() in storage/ipc/ipc.c) Without that feature we
have an interlock problem.

Will investigate this ... my immediate goal is to just get it so that an
alternate library can be used ... default behaviour will be to stick with
our current function calls ... to use libapr, you will/would have to use a
configure option for it (sorry, meant --enable above, not --disable) ...

The only '#ifdef's I'm planning on for this will be in a central shmem.*
file(s), so there isn't going to be a string of those all over the place
or anything stupid like that ...

Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

All I'm planning on doing is changing the appropriate shm_* functions iwth
pg_shm_* functions ... if !(libapr), all those pg_shm_* functions will
have in them is the original call we've always used ... there will even be
a --disable-libapr configure option so that if someone already has Apache2
installed, but doesn't wanna use libapr for PgSQL, they don't have to ...

Basically, all I'm looking at is allowing PgSQL to use a different library
for its shared memory calls then the standard one, nothing else ...

Oh. I guess my next question is how closely that Apache library
emulates the SysV shmem semantics. In particular, can you reliably
tell how many processes are attached to a shmem block? (Cf
SharedMemoryIsInUse() in storage/ipc/ipc.c) Without that feature we
have an interlock problem.

I am not familiar with the Apache code, but I see no reason why all the
features in SysV SHM should not be implementable in a Windows modules. IMHO
that's what should be done.

"Marc G. Fournier" wrote:

On Fri, 3 May 2002, Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

All I'm planning on doing is changing the appropriate shm_* functions iwth
pg_shm_* functions ... if !(libapr), all those pg_shm_* functions will
have in them is the original call we've always used ... there will even be
a --disable-libapr configure option so that if someone already has Apache2
installed, but doesn't wanna use libapr for PgSQL, they don't have to ...

Basically, all I'm looking at is allowing PgSQL to use a different library
for its shared memory calls then the standard one, nothing else ...

Oh. I guess my next question is how closely that Apache library
emulates the SysV shmem semantics. In particular, can you reliably
tell how many processes are attached to a shmem block? (Cf
SharedMemoryIsInUse() in storage/ipc/ipc.c) Without that feature we
have an interlock problem.

Will investigate this ... my immediate goal is to just get it so that an
alternate library can be used ... default behaviour will be to stick with
our current function calls ... to use libapr, you will/would have to use a
configure option for it (sorry, meant --enable above, not --disable) ...

The only '#ifdef's I'm planning on for this will be in a central shmem.*
file(s), so there isn't going to be a string of those all over the place
or anything stupid like that ...

I think that you should create a verbatim implementation of the SysV shared
memory API in native Win32. It may have to be a pgsysvshm.dll or something like
it, but I think it is the best possible approach.

Let me look at it, I may be able to have something pretty quick.

Tom Lane wrote:

mlw <markw@mohawksoft.com> writes:

I think that you should create a verbatim implementation of the SysV
shared memory API in native Win32. It may have to be a pgsysvshm.dll
or something like it, but I think it is the best possible approach.

Let me look at it, I may be able to have something pretty quick.

The notion of redesigning the internal API shouldn't be forgotten,
though. I'm not so dissatisfied with the shmem API (mainly because
it's only relevant at startup; once we've created and attached the
shmem segment, we're done worrying about it). But the SysV semaphore
API is really kind of ugly, and the ugliness doesn't buy anything except
porting difficulty. Moreover, putting a cleaner API layer there would
make it easier to experiment with cheaper semaphore primitives, such
as POSIX mutexes.

There was a thread last fall concerning redesigning that code --- I've
forgotten the guy's name, but IIRC he wanted to make a port to QNX6,
and the sema code was getting in the way. We put the work on hold
because we were getting close to 7.2 release (or thought we were,
anyway) but the project ought to be taken up again.

I will commit to writing a windows version of what ever shm/semaphore/mutex
code you guys specify.

mlw <markw@mohawksoft.com> writes:

I think that you should create a verbatim implementation of the SysV
shared memory API in native Win32. It may have to be a pgsysvshm.dll
or something like it, but I think it is the best possible approach.

Let me look at it, I may be able to have something pretty quick.

The notion of redesigning the internal API shouldn't be forgotten,
though. I'm not so dissatisfied with the shmem API (mainly because
it's only relevant at startup; once we've created and attached the
shmem segment, we're done worrying about it). But the SysV semaphore
API is really kind of ugly, and the ugliness doesn't buy anything except
porting difficulty. Moreover, putting a cleaner API layer there would
make it easier to experiment with cheaper semaphore primitives, such
as POSIX mutexes.

There was a thread last fall concerning redesigning that code --- I've
forgotten the guy's name, but IIRC he wanted to make a port to QNX6,
and the sema code was getting in the way. We put the work on hold
because we were getting close to 7.2 release (or thought we were,
anyway) but the project ought to be taken up again.

regards, tom lane

sysv shm/sem

I am writing a Win32 DLL implementation of :

int semget(key_t key, int nsems, int semflg);
int semctl(int semid, int semnum, int cmd, union semun arg);
int semop(int semid, struct sembuf * sops, unsigned nsops);
int shmctl(int shmid, int cmd, struct shmid_ds *buf);
int shmget(key_t key, int size, int shmflg);
void * shmat(int shmid, const void *shmaddr, int shmfl);
int shmdt(const void *shmaddr);

I will donate it do PostgreSQL.

UNIX permissions will be ignored, i.e. uig/gid will be 0
Do you see any need for the msgxxx calls?
Is the function ipc() ever used?

mlw <markw@mohawksoft.com> writes:

UNIX permissions will be ignored, i.e. uig/gid will be 0

Win32 has no security anyway, right? ;-)

Do you see any need for the msgxxx calls?
Is the function ipc() ever used?

Nope, and nope.

regards, tom lane

mlw <markw@mohawksoft.com> writes:

I think that you should create a verbatim implementation of the SysV
shared memory API in native Win32. It may have to be a pgsysvshm.dll
or something like it, but I think it is the best possible approach.

Let me look at it, I may be able to have something pretty quick.

The notion of redesigning the internal API shouldn't be forgotten,
though. I'm not so dissatisfied with the shmem API (mainly because
it's only relevant at startup; once we've created and attached the
shmem segment, we're done worrying about it). But the SysV semaphore
API is really kind of ugly, and the ugliness doesn't buy anything except
porting difficulty. Moreover, putting a cleaner API layer there would
make it easier to experiment with cheaper semaphore primitives, such
as POSIX mutexes.

There was a thread last fall concerning redesigning that code --- I've
forgotten the guy's name, but IIRC he wanted to make a port to QNX6,

That would be me.

and the sema code was getting in the way. We put the work on hold
because we were getting close to 7.2 release (or thought we were,
anyway) but the project ought to be taken up again.

Yes, I am intended to give it another spin soon. I think it is bad idea to
impose SysV ugliness on systems which have better solutions. Main problem
with SysV primitives is that they are 'sticky' (i.e., not cleaned up if
process dies/exits by the system). So Postgres has to deal with issues like
discovering leftovers, finding unused IPC keys, etc. It is inelegant and
takes up lot of code. POSIX primitives are anonymous and cleaned up
automatically. So you just say 'give me a semaphore' and you get it, nothing
gets into your way.

Performance of POSIX mutexes and semaphores (on platforms where they are
implemented properly) is also better than SysV semaphores. Unfortunately
some systems have rather lame POSIX support, for example semaphores and
mutexes can't be shared across processes on Linux. That's basically the
reason why people keep sticking to SysV.

What really need to be done is new abstraction layer which would cover SysV
API, POSIX and whatever native APIs are better for BeOS/OS2/Win32. I almost
did it last time...

-- igor

mlw <markw@mohawksoft.com> writes:

I am writing a Win32 DLL implementation of :

int semget(key_t key, int nsems, int semflg);
int semctl(int semid, int semnum, int cmd, union semun arg);
int semop(int semid, struct sembuf * sops, unsigned nsops);

Rather than propagating the SysV semaphore API still further, why don't
we kill it now? (I'm willing to keep the shmem API, however.)

After looking over the uses of these functions, I believe that we could
easily develop a non-SysV-centric internal API. Here's a first cut:

1. Define a struct type PGSemaphore that has implementation-specific
contents (the generic code will never look inside it). Operations on
semaphores will take "PGSemaphore *" arguments. When implementing atop
SysV semaphores, PGSemaphore will contain two fields, the semaphore id
and semaphore number. In other cases the contents could be different.

2. All PGSemaphore structs will be physically stored in shared memory.
This doesn't matter for SysV support, where the id/number are constants
anyway; but it will allow implementations based on mutexes.

3. The operations needed are

* Reserve semaphores. This will be told the number of semaphores
needed. On SysV it will do the necessary semget()s, but on some
implementations it might be a no-op. This should also be prepared
to clean up after a failed postmaster, if it is possible for sema
resources to outlive the creating postmaster.

* Create semaphore. Given a pointer to an uninitialized PGSemaphore
struct, initialize it to a new semaphore with count 1. (On SysV this
would hand out the individual semas previously allocated by Reserve.)
Note that this is not responsible for allocating the memory occupied
by the PGSemaphore struct --- I envision the structs being part of
larger objects such as PROC structures.

* Release semaphores. Release all resources allocated by previous
Reserve and Create operations. This is called when shutting down
or when resetting shared memory after a backend crash.

* Reset semaphore. Reset an existing PGSemaphore to count zero.

* Lock semaphore. Identical to current IpcSemaphoreLock(), except
parameter is a PGSemaphore *. See code of that routine for detailed
semantics.

* Unlock semaphore. Identical to current IpcSemaphoreUnlock(), except
parameter is a PGSemaphore *.

* Conditional lock semaphore. Identical to current
IpcSemaphoreTryLock(), except parameter is a PGSemaphore *.

Reserve/create/release would all be called in the postmaster process,
so they could communicate via malloc'd private memory (eg, an array
of semaphore IDs would be needed in the SysV case). The remaining
operations would be invokable by any backend.

Comments?

I'd be willing to work on refactoring the existing SysV-based code
to meet this spec.

regards, tom lane

"Igor Kovalenko" <Igor.Kovalenko@motorola.com> writes:

What really need to be done is new abstraction layer which would cover SysV
API, POSIX and whatever native APIs are better for BeOS/OS2/Win32. I almost
did it last time...

Yes. I just sent off a proposal for a cleaner semaphore API --- please
comment on it.

My inclination is to stick with the SysV API for shared memory, however.
The "stickiness" is actually not a bad thing for us in the shared memory
case, because it allows a new postmaster to detect the situation where
old backends are still running: it can see that there is an old shmem
segment still present with attached processes. Without that, we have no
good defense against the scenario where an old postmaster dumped core
leaving backends still running. The backends are fine as long as they
are left to finish out their operations, or even killed with whatever
degree of prejudice the admin wants. But what we must *not* do is allow
a new postmaster to start while the old backends are still running;
that would mean two sets of backends running without contact with each
other, which would be fatal for data integrity. The SysV API lets us
detect that case, but I don't see any equally good way to do it if we
are using anonymous shared memory.

regards, tom lane

Like I told Marc, I don't care. You spec out what you want and I'll write it
for Windows.

That being said, a SysV IPC interface for native Windows would be kind of cool
to have.

Tom Lane wrote: