improving PL/Python builds on OS X
The PL/Python build on OS X is currently hardcoded to use the system
Python install. If you try to override this when running configure, you
get a mysterious mix-and-match build. If you want to build against your
own Python build, or MacPorts or Homebrew, you can't.
This is straightforward to fix. In configure, besides checking Python
include and library paths, we can also check whether it's a framework
build and the right parameters for that. The attached patch does that
and does the job for me. Please test it.
One constraint, which is explained in the comment in
src/pl/plpython/Makefile is that in Python <2.5, there is no official
way to detect either framework builds or shared libpython builds, so we
can't support those versions on OS X, at least without more hardcoding
of things. I'd rather phase some of that out, but if someone needs to
continue to use Python 2.4 or earlier on OS X, let me know. (Or more
proper fix would be to split DLSUFFIX into two variables, but that seems
more work than it's worth right now.)
Attachments:
pg-python-framework.patchtext/plain; charset=UTF-8; name=pg-python-framework.patch; x-mac-creator=0; x-mac-type=0Download
diff --git a/config/python.m4 b/config/python.m4
index 663ccf9..af4d8d7 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -48,7 +48,6 @@ AC_MSG_RESULT([$python_includespec])
AC_SUBST(python_majorversion)[]dnl
AC_SUBST(python_version)[]dnl
-AC_SUBST(python_configdir)[]dnl
AC_SUBST(python_includespec)[]dnl
])# _PGAC_CHECK_PYTHON_DIRS
@@ -69,8 +68,14 @@ python_libdir=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(N
python_ldlibrary=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LDLIBRARY'))))"`
python_so=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('SO'))))"`
ldlibrary=`echo "${python_ldlibrary}" | sed "s/${python_so}$//"`
+python_framework=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('PYTHONFRAMEWORK'))))"`
+python_enable_shared=`${PYTHON} -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_vars().get('Py_ENABLE_SHARED',0))"`
-if test x"${python_libdir}" != x"" -a x"${python_ldlibrary}" != x"" -a x"${python_ldlibrary}" != x"${ldlibrary}"
+if test -n "$python_framework"; then
+ python_frameworkprefix=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('PYTHONFRAMEWORKPREFIX'))))"`
+ python_libspec="-F $python_frameworkprefix -framework $python_framework"
+ python_enable_shared=1
+elif test x"${python_libdir}" != x"" -a x"${python_ldlibrary}" != x"" -a x"${python_ldlibrary}" != x"${ldlibrary}"
then
# New way: use the official shared library
ldlibrary=`echo "${ldlibrary}" | sed "s/^lib//"`
@@ -86,13 +91,16 @@ else
python_libspec="-L${python_libdir} -lpython${python_ldversion}"
fi
-python_additional_libs=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LIBS','LIBC','LIBM','BASEMODLIBS'))))"`
+if test -z "$python_framework"; then
+ python_additional_libs=`${PYTHON} -c "import distutils.sysconfig; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LIBS','LIBC','LIBM','BASEMODLIBS'))))"`
+fi
AC_MSG_RESULT([${python_libspec} ${python_additional_libs}])
AC_SUBST(python_libdir)[]dnl
AC_SUBST(python_libspec)[]dnl
AC_SUBST(python_additional_libs)[]dnl
+AC_SUBST(python_enable_shared)[]dnl
# threaded python is not supported on OpenBSD
AC_MSG_CHECKING(whether Python is compiled with thread support)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 9cc14da..ecfb801 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -175,11 +175,11 @@ enable_dtrace = @enable_dtrace@
enable_coverage = @enable_coverage@
enable_thread_safety = @enable_thread_safety@
+python_enable_shared = @python_enable_shared@
python_includespec = @python_includespec@
python_libdir = @python_libdir@
python_libspec = @python_libspec@
python_additional_libs = @python_additional_libs@
-python_configdir = @python_configdir@
python_majorversion = @python_majorversion@
python_version = @python_version@
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index afd8dea..e9b5e3c 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -5,13 +5,20 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-# On some platforms we can only build PL/Python if libpython is a
-# shared library. Since there is no official way to determine this
-# (at least not in pre-2.3 Python), we see if there is a file that is
-# named like a shared library.
+# We need libpython as a shared library. In Python >=2.5, configure
+# asks Python directly. But because this has been broken in Debian
+# for a long time (http://bugs.debian.org/695979), and to support
+# older Python versions, we see if there is a file that is named like
+# a shared library as a fallback. (Note that this is wrong on OS X,
+# where DLSUFFIX is .so, but libpython is a .dylib. Python <2.5 is
+# therefore not supported on OS X.)
+ifeq (1,$(python_enable_shared))
+shared_libpython = yes
+else
ifneq (,$(wildcard $(python_libdir)/libpython*$(DLSUFFIX)*))
shared_libpython = yes
endif
+endif
# Windows needs to convert backslashed paths to normal slashes,
# and we have to remove -lpython from the link since we are building our own
@@ -21,13 +28,6 @@ python_includespec := $(subst \,/,$(python_includespec))
override python_libspec =
endif
-# Darwin (OS X) has its own ideas about how to do this.
-ifeq ($(PORTNAME), darwin)
-shared_libpython = yes
-override python_libspec = -framework Python
-override python_additional_libs =
-endif
-
# If we don't have a shared library, we have to skip it.
ifeq ($(shared_libpython),yes)
On Fri, Dec 21, 2012 at 5:45 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
The PL/Python build on OS X is currently hardcoded to use the system
Python install. If you try to override this when running configure, you
get a mysterious mix-and-match build. If you want to build against your
own Python build, or MacPorts or Homebrew, you can't.This is straightforward to fix. In configure, besides checking Python
include and library paths, we can also check whether it's a framework
build and the right parameters for that. The attached patch does that
and does the job for me. Please test it.One constraint, which is explained in the comment in
src/pl/plpython/Makefile is that in Python <2.5, there is no official
way to detect either framework builds or shared libpython builds, so we
can't support those versions on OS X, at least without more hardcoding
of things. I'd rather phase some of that out, but if someone needs to
continue to use Python 2.4 or earlier on OS X, let me know. (Or more
proper fix would be to split DLSUFFIX into two variables, but that seems
more work than it's worth right now.)
This patch seems to have broken the build for our installers for 9.3.
Because we need a consistent build of the PL interpretors on all the
platforms we support, we use the ActiveState distributions of Perl,
Python and TCL (we can't rely on vendor supplied packages, because
their versions vary between different Linux distros and different OS X
versions). However, ActivePython doesn't include a shared library,
which this change seems to require.
Can that requirement be reverted?
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5/8/13 11:51 AM, Dave Page wrote:
This patch seems to have broken the build for our installers for 9.3.
Because we need a consistent build of the PL interpretors on all the
platforms we support, we use the ActiveState distributions of Perl,
Python and TCL (we can't rely on vendor supplied packages, because
their versions vary between different Linux distros and different OS X
versions). However, ActivePython doesn't include a shared library,
which this change seems to require.Can that requirement be reverted?
There was no change in this regard. A shared library was always
required on OS X.
ActivePython does include a shared library. I just tried it and it
builds fine.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, May 8, 2013 at 5:34 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 5/8/13 11:51 AM, Dave Page wrote:
This patch seems to have broken the build for our installers for 9.3.
Because we need a consistent build of the PL interpretors on all the
platforms we support, we use the ActiveState distributions of Perl,
Python and TCL (we can't rely on vendor supplied packages, because
their versions vary between different Linux distros and different OS X
versions). However, ActivePython doesn't include a shared library,
which this change seems to require.Can that requirement be reverted?
There was no change in this regard. A shared library was always
required on OS X.
It's failing on Linux. Even worse, it configures fine and then builds
without error. There is a message spewed out by configure, but it
doesn't contain the words warning or error. Given that I explicitly
said I wanted Python support when I ran configure, it should certainly
fail with an error at configure time. We only noticed this was a
problem when the QA guys started diving in to more detailed tested, as
we don't watch for every message in the 50+ MB of logs our automated
build systems generate.
ActivePython does include a shared library. I just tried it and it
builds fine.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
It's failing on Linux. Even worse, it configures fine and then builds
without error. There is a message spewed out by configure, but it
doesn't contain the words warning or error. Given that I explicitly
said I wanted Python support when I ran configure, it should certainly
fail with an error at configure time. We only noticed this was a
problem when the QA guys started diving in to more detailed tested, as
we don't watch for every message in the 50+ MB of logs our automated
build systems generate.
It worked before because we used to allow linking shared libraries
against static libraries on some platforms. But that was more or less a
lie because it doesn't work on 64-bit platforms.
ActiveState Python contains a static library with PIC files. There is
no obvious way to detect that, which is why we don't support it
directly. You can sort it out yourself by building with
make shared_libpython=yes
In the long term, find a way to detect whether the library is usable.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 9, 2013 at 2:26 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
It's failing on Linux. Even worse, it configures fine and then builds
without error. There is a message spewed out by configure, but it
doesn't contain the words warning or error. Given that I explicitly
said I wanted Python support when I ran configure, it should certainly
fail with an error at configure time. We only noticed this was a
problem when the QA guys started diving in to more detailed tested, as
we don't watch for every message in the 50+ MB of logs our automated
build systems generate.It worked before because we used to allow linking shared libraries
against static libraries on some platforms. But that was more or less a
lie because it doesn't work on 64-bit platforms.ActiveState Python contains a static library with PIC files. There is
no obvious way to detect that, which is why we don't support it
directly. You can sort it out yourself by building withmake shared_libpython=yes
OK, thanks - I'll try that.
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 9, 2013 at 8:09 AM, Dave Page <dpage@pgadmin.org> wrote:
On Thu, May 9, 2013 at 2:26 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
On Wed, 2013-05-08 at 18:24 +0100, Dave Page wrote:
It's failing on Linux. Even worse, it configures fine and then builds
without error. There is a message spewed out by configure, but it
doesn't contain the words warning or error. Given that I explicitly
said I wanted Python support when I ran configure, it should certainly
fail with an error at configure time. We only noticed this was a
problem when the QA guys started diving in to more detailed tested, as
we don't watch for every message in the 50+ MB of logs our automated
build systems generate.It worked before because we used to allow linking shared libraries
against static libraries on some platforms. But that was more or less a
lie because it doesn't work on 64-bit platforms.ActiveState Python contains a static library with PIC files. There is
no obvious way to detect that, which is why we don't support it
directly. You can sort it out yourself by building withmake shared_libpython=yes
OK, thanks - I'll try that.
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?
BTW - it's always worked fine for us on 64 bit machines with the past
few major releases of both PG and Python - are you saying that's pure
chance? For example:
postgres=# CREATE LANGUAGE plpython3u;
CREATE LANGUAGE
postgres=# CREATE FUNCTION pyversion() RETURNS text AS
postgres-# $$
postgres$# import sys
postgres$# return sys.version
postgres$# $$ LANGUAGE 'plpython3u';
CREATE FUNCTION
postgres=# SELECT pyversion();
pyversion
-----------------------------------------
3.2.2 (default, Sep 8 2011, 12:20:18) +
[GCC 4.0.2 20051125 (Red Hat 4.0.2-8)]
(1 row)
postgres=# SELECT version();
version
---------------------------------------------------------------------------------------------------------------
PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC)
4.1.2 20080704 (Red Hat 4.1.2-52), 64-bit
(1 row)
postgres=#
(that's with fresh installations of ActivePython 3.2.2.3 and EDB's
PostgreSQL 9.2.4 package, on CentOS 6).
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5/9/13 3:09 AM, Dave Page wrote:
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?
That is ancient behavior, which I'm not fond of, but now is not the time
to change that.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5/9/13 3:25 AM, Dave Page wrote:
BTW - it's always worked fine for us on 64 bit machines with the past
few major releases of both PG and Python - are you saying that's pure
chance?
It works because ActiveState Python has PIC inside a static library.
But we have no straightforward way of knowing that (AFAIK), other than
observing whether the build result crashes or not.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 5/9/13 3:09 AM, Dave Page wrote:
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?That is ancient behavior, which I'm not fond of, but now is not the time
to change that.
Eh? Beta is precisely the time to find and fix bugs like this.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5/9/13 10:52 AM, Dave Page wrote:
On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 5/9/13 3:09 AM, Dave Page wrote:
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?That is ancient behavior, which I'm not fond of, but now is not the time
to change that.Eh? Beta is precisely the time to find and fix bugs like this.
It's not a bug, it's working as designed (not by me).
It's certainly worth reexamining, but there are so many variations in
this area (and Perl and Tcl are set up the same way) that I don't want
to mess around in it now.
A better approach for now would be to look for a configuration item in
Python that tells us whether libpython contains PIC.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, May 9, 2013 at 10:06 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 5/9/13 10:52 AM, Dave Page wrote:
On Thu, May 9, 2013 at 3:09 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
On 5/9/13 3:09 AM, Dave Page wrote:
I assume you'll fix the configure script so it actually errors out if
plpython cannot be built, but is explicitly requested?That is ancient behavior, which I'm not fond of, but now is not the time
to change that.Eh? Beta is precisely the time to find and fix bugs like this.
It's not a bug, it's working as designed (not by me).
Well, design bug then. It's clearly a nasty POLA violation at minimum.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, 2013-05-09 at 10:11 -0400, Peter Eisentraut wrote:
On 5/9/13 3:25 AM, Dave Page wrote:
BTW - it's always worked fine for us on 64 bit machines with the past
few major releases of both PG and Python - are you saying that's pure
chance?It works because ActiveState Python has PIC inside a static library.
But we have no straightforward way of knowing that (AFAIK), other than
observing whether the build result crashes or not.
After further digging, it seems to me that their build is not a standard
build. They must be patching in compiler options through the backdoor
somehow. Their config/Makefile has
BASECFLAGS= -fno-strict-aliasing -fPIC
meaning that they compile *everything* with those options. But that's
not something that the standard Python configure script can produce.
Again, reliably detecting that might be difficult.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, May 10, 2013 at 2:47 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
On Thu, 2013-05-09 at 10:11 -0400, Peter Eisentraut wrote:
On 5/9/13 3:25 AM, Dave Page wrote:
BTW - it's always worked fine for us on 64 bit machines with the past
few major releases of both PG and Python - are you saying that's pure
chance?It works because ActiveState Python has PIC inside a static library.
But we have no straightforward way of knowing that (AFAIK), other than
observing whether the build result crashes or not.After further digging, it seems to me that their build is not a standard
build. They must be patching in compiler options through the backdoor
somehow. Their config/Makefile hasBASECFLAGS= -fno-strict-aliasing -fPIC
meaning that they compile *everything* with those options. But that's
not something that the standard Python configure script can produce.
Huh, interesting. I used to have a contact at ActiveState - I'll see
if I can get some more details.
Again, reliably detecting that might be difficult.
Yeah, that's understandable. As long as I can force it though (and
your suggestion on that front seems to work fine), I'm happy.
Thanks.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers