darwin pgsql patches

Peter Bierman <bierman@apple.com> writes:

/* placeholder file for a self-test program I plan to write. pmb-001128 */

I think there is a semaphore-testing program in the QNX port; have you
looked at that?

+#define darwin

Could we make this symbol __darwin__, or something else a little less
likely to conflict with plain names in the code? (For that matter,
doesn't the compiler predefine some symbol along this line?)

I'd make the latter change myself, except I have no ability to test it.
Please change the patch appropriately and resubmit...

regards, tom lane

Peter Bierman writes:

Here are patches to CVS top of tree that allow darwin to completely
build and install. Unfortunately, the sem...() shim does not function
correctly yet, so it's not stable enough to use.

We've been through this before. Newer Darwin kernels have semaphore
support, which means that you'll at least have to compensate for that. I
believe we agreed that semaphore support for older kernels would be
available as a patch.

#include "postgres.h"
#include "storage/ipc.h"
#include "storage/proc.h"
-#include <sys/sem.h>
+//#include <sys/sem.h>
+#include "sem.h"

No C++ comments.

-void *pg_dlopen(const char *filename)
+void *pg_dlopen(char *filename)

Why?

+#if defined(darwin) && defined(__ppc__)
+static void
+tas_dummy()
+{
+       __asm__("               \n\
+               .globl  tas     \n\
+               .globl  _tas    \n\
+_tas:                          \n\
+tas:                           \n\
+               lwarx   r5,0,r3 \n\
+               cmpwi   r5,0    \n\
+               bne     fail    \n\
+               addi    r5,r5,1 \n\
+               stwcx.  r5,0,r3 \n\
+               beq     success \n\
+fail:          li      r3,1    \n\
+               blr             \n\
+success:                       \n\
+               li r3,0         \n\
+               blr             \n\
+	");
+}
+
+#endif  /* __ppc__ && darwin */

Here the question was why the existing powerpc code doesn't work. There
are only syntax differences. Any idea?

+#if defined(darwin)
+#include "../backend/port/darwin/sem.h"
+#endif

This doesn't work. There's no guarantee that the source tree is the same
as the build tree.

4c6,7
< CFLAGS=-O0
---

#CFLAGS=-O2

No commented out things. Either you take -02 or -O0 or you set CFLAGS to
something else, but it has to be set to something.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

At 11:05 AM -0500 11/29/00, Tom Lane wrote:

Peter Bierman <bierman@apple.com> writes:

/* placeholder file for a self-test program I plan to write. pmb-001128 */

I think there is a semaphore-testing program in the QNX port; have you
looked at that?

Yes, and it allowed the user to manually toggle semaphores, but I want to write something that will do more of a full regression sweep and produce a pass or fail result. (Which might be unnecessary if latest darwin kernels actually implement <sys/sem.h> completely, see below.)

+#define darwin

Could we make this symbol __darwin__, or something else a little less
likely to conflict with plain names in the code? (For that matter,
doesn't the compiler predefine some symbol along this line?)

Sure.
The only symbols darwin gcc defines are:
#define __MACH__ 1
#define __NATURAL_ALIGNMENT__ 1
#define __APPLE__ 1
#define __GNUC_MINOR__ 95
#define __ppc__ 1
#define __GNUC__ 2
#define __STDC__ 1
#define __DYNAMIC__ 1
#define __APPLE_CC__ 912
#define __BIG_ENDIAN__ 1
So defining one in PG via configure seems best.

At 5:30 PM +0100 11/29/00, Peter Eisentraut wrote:

Peter Bierman writes:

Here are patches to CVS top of tree that allow darwin to completely
build and install. Unfortunately, the sem...() shim does not function
correctly yet, so it's not stable enough to use.

We've been through this before. Newer Darwin kernels have semaphore
support, which means that you'll at least have to compensate for that. I
believe we agreed that semaphore support for older kernels would be
available as a patch.

Can you point me to an archive of that discussion? I'd love for it to be true.
I did the patch on a Mac OS X Public Beta machine, and gave it a quick test on the most recent Mac OS X internal build, but I didn't try it without the shim on the latest build. I'll try that.

+//#include <sys/sem.h>

No C++ comments.

Good to know.

-void *pg_dlopen(const char *filename)
+void *pg_dlopen(char *filename)

Why?

All of the other ports do not use the const. The code is called from the main sourcebase assuming it's not const. I chose the quicker fix of having the darwin code match the rest of the source instead of trying to push the more correct const definition back into the rest of the code.

+#if defined(darwin) && defined(__ppc__)
+static void
+tas_dummy()
+{
+       __asm__("               \n\

Here the question was why the existing powerpc code doesn't work. There
are only syntax differences. Any idea?

I think it has more to do with Apple's forked gcc than anything else. I don't have a LinuxPPC machine to check if adding the rX syntax there would continue to work. Also, the darwin gcc doesn't create the _tas symbol automatically, so it has to be declared in the asm routine.

+#if defined(darwin)
+#include "../backend/port/darwin/sem.h"
+#endif

This doesn't work. There's no guarantee that the source tree is the same
as the build tree.

Fair enough, what would be the right way to include this file then?

4c6,7
< CFLAGS=-O0
---

#CFLAGS=-O2

No commented out things. Either you take -02 or -O0 or you set CFLAGS to
something else, but it has to be set to something.

Ok. I don't see any reason why -O2 shouldn't work, So I'll just pull the line.

Thanks for the feedback! I'm doing the port for personal reasons, but I think it will be a great feature if Mac OS X GM works with PG.

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

Peter Bierman <bierman@apple.com> writes:

Here the question was why the existing powerpc code doesn't work. There
are only syntax differences. Any idea?

I think it has more to do with Apple's forked gcc than anything
else. I don't have a LinuxPPC machine to check if adding the rX syntax
there would continue to work.

I do (have a LinuxPPC machine), so if you'd identify exactly what I
should try, I'll be glad to check this point. Seems like it'd be smart
to minimize the number of PPC variants of this routine.

regards, tom lane

Tom Lane writes:

I think it has more to do with Apple's forked gcc than anything
else. I don't have a LinuxPPC machine to check if adding the rX syntax
there would continue to work.

I do (have a LinuxPPC machine), so if you'd identify exactly what I
should try, I'll be glad to check this point. Seems like it'd be smart
to minimize the number of PPC variants of this routine.

Minimize the differences between

+#if defined(darwin) && defined(__ppc__)
+static void
+tas_dummy()
+{
+       __asm__("               \n\
+               .globl  tas     \n\
+               .globl  _tas    \n\
+_tas:                          \n\
+tas:                           \n\
+               lwarx   r5,0,r3 \n\
+               cmpwi   r5,0    \n\
+               bne     fail    \n\
+               addi    r5,r5,1 \n\
+               stwcx.  r5,0,r3 \n\
+               beq     success \n\
+fail:          li      r3,1    \n\
+               blr             \n\
+success:                       \n\
+               li r3,0         \n\
+               blr             \n\
+       ");
+}
+
+#endif  /* __ppc__ && darwin */

and

[src/backend/storage/buffer/s_lock.c]
#if defined(__powerpc__)
/* Note: need a nice gcc constrained asm version so it can be inlined */
static void
tas_dummy()
{
__asm__(" \n\
.global tas \n\
tas: \n\
lwarx 5,0,3 \n\
cmpwi 5,0 \n\
bne fail \n\
addi 5,5,1 \n\
stwcx. 5,0,3 \n\
beq success \n\
fail: li 3,1 \n\
blr \n\
success: \n\
li 3,0 \n\
blr \n\
");
}

#endif /* __powerpc__ */

under the constraint that the first version is probably not negotiable.
I'd like to wish that the GNU assembler is a little more forgiving.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

I think there is a semaphore-testing program in the QNX port; have you
looked at that?

Yes, and it allowed the user to manually toggle semaphores, but I want to write something that will do more of a full regression sweep and produce a pass or fail result. (Which might be unnecessary if latest darwin kernels actually implement <sys/sem.h> completely, see below.)

+#define darwin

Could we make this symbol __darwin__, or something else a little less
likely to conflict with plain names in the code? (For that matter,
doesn't the compiler predefine some symbol along this line?)

The only symbols darwin gcc defines are:
#define __MACH__ 1
#define __NATURAL_ALIGNMENT__ 1
#define __APPLE__ 1
#define __GNUC_MINOR__ 95
#define __ppc__ 1
#define __GNUC__ 2
#define __STDC__ 1
#define __DYNAMIC__ 1
#define __APPLE_CC__ 912
#define __BIG_ENDIAN__ 1
So defining one in PG via configure seems best.

Why not use "__APPLE__"? There probably isn't much hope of running
anything on a Mac with MacOS which isn't also darwin, right? Presumably
the gcc for LinuxPPC does not define __APPLE__??

- Thomas

Thomas Lockhart writes:

Why not use "__APPLE__"? There probably isn't much hope of running
anything on a Mac with MacOS which isn't also darwin, right?

What about Rhapsody?

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

Why not use "__APPLE__"? There probably isn't much hope of running
anything on a Mac with MacOS which isn't also darwin, right?

What about Rhapsody?

Yeah, what about it? Are there any production machines which might get
PostgreSQL installed on them? Not so far, otherwise we would have seen a
port, right?

Peter B, comments?

- Thomas

Peter B, comments?

fwiw (in case I'm coming across as *just* in a bad mood ;), it seems
like the right time to help Apple understand that they should be
building their compiler with predefined symbols so that the machine and
OS type can be identified automatically. In general, one expects that
from the compiler, though I'm sure we have exceptions in our supported
platforms.

- Thomas

At 6:47 PM +0000 11/30/00, Thomas Lockhart wrote:

Why not use "__APPLE__"? There probably isn't much hope of running
anything on a Mac with MacOS which isn't also darwin, right?

What about Rhapsody?

Yeah, what about it? Are there any production machines which might get
PostgreSQL installed on them? Not so far, otherwise we would have seen a
port, right?

Peter B, comments?

Mac OS X exists in two forks at the moment. One of them is Mac OS X server, which is based on mach 2.5, and is a snapshot of the Mac OS X development process from 1999. There are enough customers of this product that Apple has continued to bring that fork forward onto the latest machines, so that they can smoothly migrate them back onto the top of tree when it ships as a server product. These people are either WebObjects developers who are probably deploying on Solaris or HPUX, or Schools and Workgroups who need a high end AppleShareIP server.

Mac OS X top of tree is Mac OS X client, currently available as Mac OS X Public Beta. (Darwin, which is the name for all of the Open Source parts, everything below the window server, is actually slightly more current.) When it's stable enough, the server solution software will be sold on top of this codebase.

While there are probably several Mac OS X Server customers that would be interested in PG, I think you can safely concentrate on Mac OS X (client, Public Beta, darwin) with the knowledge that they will eventually come to you.

fwiw (in case I'm coming across as *just* in a bad mood ;), it seems
like the right time to help Apple understand that they should be
building their compiler with predefined symbols so that the machine and
OS type can be identified automatically. In general, one expects that
from the compiler, though I'm sure we have exceptions in our supported
platforms.

The list I sent is actually the list that was decided on about a year ago, after trimming a much longer list of stuff that the compiler used to export. For that reason, I don't think you'll see it grow.

But I don't actually think it should. It's the prupose of configure, not the compiler, to determine the system that's running. Why should the compiler care what varriant of an OS it's running on? It might not even be compiling code for that OS. Is there some cost of #define __darwin__ in src/template/darwin that I'm not seeing? (not trying to be inflamatory here. :-)

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

At 5:30 PM +0100 11/29/00, Peter Eisentraut wrote:

Peter Bierman writes:

Here are patches to CVS top of tree that allow darwin to completely
build and install. Unfortunately, the sem...() shim does not function
correctly yet, so it's not stable enough to use.

We've been through this before. Newer Darwin kernels have semaphore
support, which means that you'll at least have to compensate for that. I
believe we agreed that semaphore support for older kernels would be
available as a patch.

I just talked to the kernel engineer, and semctl & friends will not be in darwin anytime soon.

The current Darwin kernel supports the sem_* functions, but not sem_init and sem_destroy (unnamed BSD semaphores). sem_open and sem_close work though. I plan to change the qnx shim code to use named semaphores, and with luck, we'll be off to the races. ;-)

I still don't quite know what to do about:

+#if defined(darwin)
+#include "../backend/port/darwin/sem.h"
+#endif

This doesn't work. There's no guarantee that the source tree is the same
as the build tree.

I'm guessing qnx4 doesn't have this problem because they have a <sys/sem.h>, just no implementation.

I could put the contents in os.h, though that feels wrong.
I could add a CFLAGS+=-Isrc/backend/port/darwin/ either to the affected file's parent directory makefiles, or in template/darwin.

Suggestions?

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

Why not use "__APPLE__"? There probably isn't much hope of running
anything on a Mac with MacOS which isn't also darwin, right?

What about Rhapsody?

Mac OS X exists in two forks at the moment. One of them is Mac OS X server, which is based on mach 2.5, and is a snapshot of the Mac OS X development process from 1999. There are enough customers of this product that Apple has continued to bring that fork forward onto the latest machines, so that they can smoothly migrate them back onto the top of tree when it ships as a server product. These people are either WebObjects developers who are probably deploying on Solaris or HPUX, or Schools and Workgroups who need a high end AppleShareIP server.
Mac OS X top of tree is Mac OS X client, currently available as Mac OS X Public Beta. (Darwin, which is the name for all of the Open Source parts, everything below the window server, is actually slightly more current.) When it's stable enough, the server solution software will be sold on top of this codebase.
While there are probably several Mac OS X Server customers that would be interested in PG, I think you can safely concentrate on Mac OS X (client, Public Beta, darwin) with the knowledge that they will eventually come to you.

fwiw (in case I'm coming across as *just* in a bad mood ;), it seems
like the right time to help Apple understand that they should be
building their compiler with predefined symbols so that the machine and
OS type can be identified automatically. In general, one expects that
from the compiler, though I'm sure we have exceptions in our supported
platforms.

The list I sent is actually the list that was decided on about a year ago, after trimming a much longer list of stuff that the compiler used to export. For that reason, I don't think you'll see it grow.
But I don't actually think it should. It's the prupose of configure, not the compiler, to determine the system that's running. Why should the compiler care what varriant of an OS it's running on? It might not even be compiling code for that OS. Is there some cost of #define __darwin__ in src/template/darwin that I'm not seeing? (not trying to be inflamatory here. :-)

No, no particular cost, other than in trying to distill the *actual*
properties of our code base down to the fundamental necessities.

In this case, ./configure can test for general properties of the system
as well as simply testing for the signiture used to set up things like
the ports/ files. If there is a testable property, we can write a test
for configure to try. If there is a signiture based on the results of
uname etc then configure can help select files for inclusion. But the
inline code which is #ifdef'd for specific platforms may as well stand
on its own if possible. So, for example, on Linux systems the "#ifdef
__linux__" was provided by the compiler, not by ./configure, though if
we had to we could try to come up with tests for determining that or we
could hardcode the definition into a ports/ file.

In your summary of MacOS-X, you indicated that we could assume a single
supported flavor, since the two existing code forks will eventually
merge. If there are not *other* Mac platforms which define "__APPLE__"
and which could also run PostgreSQL, then that symbol would likely be
the best choice for our inline tests. All imho of course. Peter E. has
been doing extensive work on the configure system recently, and may have
some more or different suggestions.

Regards.

- Thomas

At 5:50 AM +0000 12/1/00, Thomas Lockhart wrote:

In your summary of MacOS-X, you indicated that we could assume a single
supported flavor, since the two existing code forks will eventually
merge. If there are not *other* Mac platforms which define "__APPLE__"
and which could also run PostgreSQL, then that symbol would likely be
the best choice for our inline tests.

Well, it's hard for me to say definitively. But I don't think there are other POSIX-like OS implementations that define __APPLE__

On a different note, the same kernel engineer wrote me back a few minutes ago to tell me that named semaphores actually started working with *today's* kernel. So it looks like running it on Mac OS X Public Beta will tickle a kernel bug in that release.

Any opinions on the best way to check for this and warn the user?

I'm thinking of creating a darwin_startup() called from backend/main/main.c that would check the system version and abort the startup if the system version is too old. I don't want to make it a compile time check, because I'm sure there will be people grabbing binary distributions after X-GM is available and trying them on X-PB.

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

Thomas Lockhart <lockhart@alumni.caltech.edu> writes:

I'm thinking of creating a darwin_startup() called from
backend/main/main.c that would check the system version and abort the
startup if the system version is too old.

... this seems to be overkill.

I agree with Thomas here: platform-specific warts with a short life
expectancy are a bad idea. We have more than enough warts already.
We don't need to hard-wire a check that tells people running on a
beta OS release that that beta has bugs.

regards, tom lane

Well, it's hard for me to say definitively. But I don't think there are other POSIX-like OS implementations that define __APPLE__

OK, that sounds like a winner for now...

On a different note, the same kernel engineer wrote me back a few minutes ago to tell me that named semaphores actually started working with *today's* kernel. So it looks like running it on Mac OS X Public Beta will tickle a kernel bug in that release.
Any opinions on the best way to check for this and warn the user?

Hmm. MacOS X Public Beta. PostgreSQL-7.1beta. Lots of betas, and no
releases (yet). So what is to warn about? You will possibly be testing
on upcoming snapshots and betas and full releases, so we will have some
feedback on whether things look good for PostgreSQL-7.1 and MacOS-X
released versions. And we'd be happy to have a MacOS-X FAQ in our
sources and binary distros, so at that point the users can either help
with development or read the FAQ and stay away.

I'm sure some of your constituancy is not used to this style of "user
beware" system, so you'll want to help them as much as possible, but...

I'm thinking of creating a darwin_startup() called from backend/main/main.c that would check the system version and abort the startup if the system version is too old. I don't want to make it a compile time check, because I'm sure there will be people grabbing binary distributions after X-GM is available and trying them on X-PB.

... this seems to be overkill. Hardcoding some defensive checks in the
server backend to guard against improper installations of this nature
seems to be the wrong direction to take. For Linux, one could put these
kinds of checks in the RPM installation script, and RPM will decline to
proceed if the checks fail. What sort of installation environment does
MacOS-X have? What will binary distros look like? If there are, say,
installation scripts then checks could be put there. Or if the distro
will have a startup script then the checks could be put there. But
compiling that into the backend? We haven't had to do that for other
platforms, and it would be nice to avoid a Mac-only appendage in the
compiled source code.

Again, all imho and not meant to be annoying or unhelpful...

- Thomas

At 2:01 AM -0500 12/1/00, Tom Lane wrote:

Thomas Lockhart <lockhart@alumni.caltech.edu> writes:

I'm thinking of creating a darwin_startup() called from
backend/main/main.c that would check the system version and abort the
startup if the system version is too old.

... this seems to be overkill.

I agree with Thomas here: platform-specific warts with a short life
expectancy are a bad idea. We have more than enough warts already.
We don't need to hard-wire a check that tells people running on a
beta OS release that that beta has bugs.

I was going to say I disagree, but then I remembered that the beta has a (well documented) timebomb, so it will indeed be a moot issue eventually.

What sort of installation environment does
MacOS-X have? What will binary distros look like?

Funny you should ask, since my real job is to work on the Installer.

PG is not the kind of thing we'd make a GUI install package for, at least not yet. The PG user is going to need to use the command line to interact with it, so they might as well use it to install it. As such, binary dists would probably just be a tarball.

In other news, I just (litterally a minute ago) fixed the last issues with my port (hurray!), and the regression test passes every test except geometry, where the diffs differ in the 15th decimal digit for the last coordinate pair. Rounding error somewhere?

Do the regression tests adequately stress semaphores? Are semephores used interprocess? Do the regression tests cause this sort of use to be tested?

The only thing I'm worried about is that the regression tests simulate an artifically sequential load.

And any suggestions on how to include my sem.h?
I can send in the revised patch as soon as that's resolved.

Thanks!

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

and the regression test passes every test except geometry, where the diffs
differ in the 15th decimal digit for the last coordinate pair. Rounding
error somewhere?

Do the regression tests adequately stress semaphores? Are semephores used
interprocess? Do the regression tests cause this sort of use to be tested?

The only thing I'm worried about is that the regression tests simulate an
artifically sequential load.

And I'm dumb, and should RTFM before asking these questions. (Answers to above: no, probably, yes, & yes.)

Would you believe I've yet to write a single line of SQL? Gotta start somewhere.

-pmb

--
bierman@apple.com

"4 out of 5 people with the wrong hardware want to run Mac OS X because..."
http://www.newertech.com/oscompatibility/osxinfo.html

Peter Bierman <bierman@apple.com> writes:

In other news, I just (litterally a minute ago) fixed the last issues
with my port (hurray!), and the regression test passes every test
except geometry, where the diffs differ in the 15th decimal digit for
the last coordinate pair. Rounding error somewhere?

Do the results match either of the existing expected-files for powerpc
systems?

Do the regression tests adequately stress semaphores? Are semephores
used interprocess? Do the regression tests cause this sort of use to
be tested?

The parallel format ("make check") provides at least some confidence.
The regress tests don't really exercise IPC very thoroughly though :-(

regards, tom lane

Peter Bierman <bierman@apple.com> writes:

But I don't actually think it should. It's the prupose of configure,
not the compiler, to determine the system that's running. Why should
the compiler care what varriant of an OS it's running on? It might not
even be compiling code for that OS.

Yeah, but header files care, and predefined symbols are the only way
that the C compiler can pass what it knows about the target environment
(which it *must* know about, in some fashion) to header files.
To take one example, what if a header file needs to know whether it's
being compiled under Rhapsody or Darwin?

Is there some cost of #define __darwin__ in src/template/darwin that
I'm not seeing? (not trying to be inflamatory here. :-)

I was just reminded of another reason why we like compiler-provided
symbols of this kind better than hacking one up in the port's header
file: you don't have to be very careful about whether such a symbol
is defined yet or not. For example, our config.h has

/*
* Define this if your operating system supports AF_UNIX family sockets.
*/
#if !defined(__CYGWIN__) && !defined(__QNX__) && !defined(__BEOS__)
# define HAVE_UNIX_SOCKETS 1
#endif

which works fine, but only because those are all compiler-predefined
symbols --- the port-specific os.h file hasn't been included yet.
("Rearrange the code" isn't a very good retort, because that just
introduces other ordering problems.)

You might want to think about having the template/darwin file add
-D__darwin__ to CPPFLAGS, instead of putting a #define in port/darwin.h.
And have another go at convincing Apple that they're wrong not to have
their compiler provide such a symbol automatically.

regards, tom lane