pgsql: Add transforms feature

Started by Peter Eisentrautalmost 11 years ago34 messageshackers
Jump to latest
#1Peter Eisentraut
peter_e@gmx.net

Add transforms feature

This provides a mechanism for specifying conversions between SQL data
types and procedural languages. As examples, there are transforms
for hstore and ltree for PL/Perl and PL/Python.

reviews by Pavel Stěhule and Andres Freund

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/cac76582053ef8ea07df65fed0757f352da23705

Modified Files
--------------
contrib/Makefile | 12 +
contrib/hstore_plperl/.gitignore | 4 +
contrib/hstore_plperl/Makefile | 24 +
.../hstore_plperl/expected/create_transform.out | 74 +
contrib/hstore_plperl/expected/hstore_plperl.out | 213 +
contrib/hstore_plperl/hstore_plperl--1.0.sql | 17 +
contrib/hstore_plperl/hstore_plperl.c | 90 +
contrib/hstore_plperl/hstore_plperl.control | 6 +
contrib/hstore_plperl/hstore_plperlu--1.0.sql | 17 +
contrib/hstore_plperl/hstore_plperlu.control | 6 +
contrib/hstore_plperl/sql/create_transform.sql | 47 +
contrib/hstore_plperl/sql/hstore_plperl.sql | 148 +
contrib/hstore_plpython/.gitignore | 6 +
contrib/hstore_plpython/Makefile | 31 +
.../hstore_plpython/expected/hstore_plpython.out | 132 +
contrib/hstore_plpython/hstore_plpython.c | 116 +
contrib/hstore_plpython/hstore_plpython2u--1.0.sql | 19 +
contrib/hstore_plpython/hstore_plpython2u.control | 6 +
contrib/hstore_plpython/hstore_plpython3u--1.0.sql | 19 +
contrib/hstore_plpython/hstore_plpython3u.control | 6 +
contrib/hstore_plpython/hstore_plpythonu--1.0.sql | 19 +
contrib/hstore_plpython/hstore_plpythonu.control | 6 +
contrib/hstore_plpython/sql/hstore_plpython.sql | 103 +
contrib/ltree_plpython/.gitignore | 6 +
contrib/ltree_plpython/Makefile | 31 +
contrib/ltree_plpython/expected/ltree_plpython.out | 45 +
contrib/ltree_plpython/ltree_plpython.c | 32 +
contrib/ltree_plpython/ltree_plpython2u--1.0.sql | 12 +
contrib/ltree_plpython/ltree_plpython2u.control | 6 +
contrib/ltree_plpython/ltree_plpython3u--1.0.sql | 12 +
contrib/ltree_plpython/ltree_plpython3u.control | 6 +
contrib/ltree_plpython/ltree_plpythonu--1.0.sql | 12 +
contrib/ltree_plpython/ltree_plpythonu.control | 6 +
contrib/ltree_plpython/sql/ltree_plpython.sql | 37 +
doc/src/sgml/catalogs.sgml | 82 +
doc/src/sgml/hstore.sgml | 19 +
doc/src/sgml/information_schema.sgml | 85 +
doc/src/sgml/ltree.sgml | 15 +
doc/src/sgml/ref/allfiles.sgml | 2 +
doc/src/sgml/ref/alter_extension.sgml | 21 +
doc/src/sgml/ref/comment.sgml | 22 +
doc/src/sgml/ref/create_function.sgml | 18 +
doc/src/sgml/ref/create_transform.sgml | 207 +
doc/src/sgml/ref/drop_transform.sgml | 123 +
doc/src/sgml/reference.sgml | 2 +
src/Makefile.shlib | 2 +-
src/backend/catalog/Makefile | 1 +
src/backend/catalog/dependency.c | 8 +
src/backend/catalog/information_schema.sql | 34 +-
src/backend/catalog/objectaddress.c | 65 +-
src/backend/catalog/pg_aggregate.c | 1 +
src/backend/catalog/pg_proc.c | 46 +-
src/backend/commands/dropcmds.c | 8 +
src/backend/commands/event_trigger.c | 3 +
src/backend/commands/functioncmds.c | 343 +-
src/backend/commands/proclang.c | 3 +
src/backend/commands/typecmds.c | 1 +
src/backend/nodes/copyfuncs.c | 17 +
src/backend/nodes/equalfuncs.c | 15 +
src/backend/parser/gram.y | 90 +-
src/backend/tcop/utility.c | 16 +
src/backend/utils/adt/ruleutils.c | 39 +-
src/backend/utils/cache/lsyscache.c | 71 +
src/backend/utils/cache/syscache.c | 23 +
src/backend/utils/fmgr/funcapi.c | 44 +
src/bin/pg_dump/common.c | 5 +
src/bin/pg_dump/pg_dump.c | 273 +-
src/bin/pg_dump/pg_dump.h | 11 +
src/bin/pg_dump/pg_dump_sort.c | 11 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/dependency.h | 1 +
src/include/catalog/indexing.h | 5 +
src/include/catalog/pg_class.h | 2 +-
src/include/catalog/pg_proc.h | 5382 ++++++++++----------
src/include/catalog/pg_proc_fn.h | 3 +
src/include/catalog/pg_transform.h | 47 +
src/include/commands/defrem.h | 3 +
src/include/funcapi.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 15 +
src/include/parser/kwlist.h | 2 +
src/include/utils/lsyscache.h | 3 +
src/include/utils/syscache.h | 2 +
src/interfaces/ecpg/preproc/ecpg.tokens | 2 +-
src/interfaces/ecpg/preproc/ecpg.trailer | 5 +-
src/interfaces/ecpg/preproc/ecpg_keywords.c | 2 -
src/pl/plperl/GNUmakefile | 4 +-
src/pl/plperl/plperl.c | 47 +-
src/pl/plperl/plperl_helpers.h | 2 +
src/pl/plpython/Makefile | 40 +-
src/pl/plpython/plpy_main.c | 1 +
src/pl/plpython/plpy_procedure.c | 43 +-
src/pl/plpython/plpy_procedure.h | 2 +
src/pl/plpython/plpy_spi.c | 3 +-
src/pl/plpython/plpy_typeio.c | 159 +-
src/pl/plpython/plpy_typeio.h | 9 +-
src/pl/plpython/plpy_util.c | 21 +-
src/pl/plpython/plpy_util.h | 1 +
src/pl/plpython/plpython.h | 1 +
src/pl/plpython/regress-python3-mangle.mk | 35 +
src/test/regress/expected/sanity_check.out | 1 +
101 files changed, 6061 insertions(+), 2838 deletions(-)

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#1)
Re: pgsql: Add transforms feature

Peter Eisentraut <peter_e@gmx.net> writes:

Add transforms feature

I don't know why this patch is fooling around with compile/link flags,
but it's broken at least prairiedog and some of the Windows critters.

regards, tom lane

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#3Michael Paquier
michael@paquier.xyz
In reply to: Tom Lane (#2)
Re: pgsql: Add transforms feature

On Mon, Apr 27, 2015 at 1:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Add transforms feature

I don't know why this patch is fooling around with compile/link flags,
but it's broken at least prairiedog and some of the Windows critters.

It breaks as well on all the Windows machines using MS or VC toolchains...
--
Michael

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#4Peter Eisentraut
peter_e@gmx.net
In reply to: Michael Paquier (#3)
Re: pgsql: Add transforms feature

On 4/27/15 2:23 AM, Michael Paquier wrote:

On Mon, Apr 27, 2015 at 1:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Add transforms feature

I don't know why this patch is fooling around with compile/link flags,
but it's broken at least prairiedog and some of the Windows critters.

It breaks as well on all the Windows machines using MS or VC toolchains...

The failure on bowerbird (Windows msvc) is because the build system
needs to be told to add additional include directories to the compilation.

The failure on frogmouth (Windows gcc) is because of some conflict with
perl headers. I must have missed a trick that was already in use for
plperl.

Help in either case, but especially the first one, is welcome.

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#5Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Peter Eisentraut (#1)
Re: pgsql: Add transforms feature

Peter Eisentraut wrote:

Add transforms feature

This provides a mechanism for specifying conversions between SQL data
types and procedural languages. As examples, there are transforms
for hstore and ltree for PL/Perl and PL/Python.

BF member olinguito is also failing the contrib test, because plpython
is not compiled in that platform.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#6Andrew Dunstan
andrew@dunslane.net
In reply to: Peter Eisentraut (#4)
Re: pgsql: Add transforms feature

On 04/27/2015 08:21 AM, Peter Eisentraut wrote:

On 4/27/15 2:23 AM, Michael Paquier wrote:

On Mon, Apr 27, 2015 at 1:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Add transforms feature

I don't know why this patch is fooling around with compile/link flags,
but it's broken at least prairiedog and some of the Windows critters.

It breaks as well on all the Windows machines using MS or VC toolchains...

The failure on bowerbird (Windows msvc) is because the build system
needs to be told to add additional include directories to the compilation.

The failure on frogmouth (Windows gcc) is because of some conflict with
perl headers. I must have missed a trick that was already in use for
plperl.

Help in either case, but especially the first one, is welcome.

I haven't looked at the MSVC case yet, but one thing I notice on jacana
(Windows 8.1, gcc) is that the order of include directories is
different. When compiling plperl, the perl CORE directory is last in the
include list, whereas for compiling hstore_plperl it's second from the
beginning. I have no idea yet if that makes any difference, but would
not be at all surprised it if did. We should probably be mimicking what
the plperl makefile does.

Incidentally, I notice that hstore_plperl.c doesn't have a copyright
notice nor a single comment, not even at the start of the public
functions. That seems less than good.

cheers

andrew

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#7Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#6)
Re: pgsql: Add transforms feature

On 04/27/2015 07:54 PM, Andrew Dunstan wrote:

On 04/27/2015 08:21 AM, Peter Eisentraut wrote:

On 4/27/15 2:23 AM, Michael Paquier wrote:

On Mon, Apr 27, 2015 at 1:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Add transforms feature

I don't know why this patch is fooling around with compile/link flags,
but it's broken at least prairiedog and some of the Windows critters.

It breaks as well on all the Windows machines using MS or VC
toolchains...

The failure on bowerbird (Windows msvc) is because the build system
needs to be told to add additional include directories to the
compilation.

The failure on frogmouth (Windows gcc) is because of some conflict with
perl headers. I must have missed a trick that was already in use for
plperl.

Help in either case, but especially the first one, is welcome.

I haven't looked at the MSVC case yet, but one thing I notice on
jacana (Windows 8.1, gcc) is that the order of include directories is
different. When compiling plperl, the perl CORE directory is last in
the include list, whereas for compiling hstore_plperl it's second from
the beginning. I have no idea yet if that makes any difference, but
would not be at all surprised it if did. We should probably be
mimicking what the plperl makefile does.

Incidentally, I notice that hstore_plperl.c doesn't have a copyright
notice nor a single comment, not even at the start of the public
functions. That seems less than good.

w.r.t MSVC builds, it looks like we need entries in
$contrib_extraincludes in src/tools/msvc/Mkvcbuild.pm at the very least.

cheers

andrew

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#8Michael Paquier
michael@paquier.xyz
In reply to: Andrew Dunstan (#7)
Re: pgsql: Add transforms feature

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in $contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.
--
Michael

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#9Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#5)
Re: pgsql: Add transforms feature

Alvaro Herrera <alvherre@2ndquadrant.com> writes:

Peter Eisentraut wrote:

Add transforms feature

BF member olinguito is also failing the contrib test, because plpython
is not compiled in that platform.

Yeah. Even more specifically, olinguito does have --with-python in its
configure flags, but then the plpython Makefile skips the build because
libpython isn't available as a shared library. But the contrib test is
(I assume, haven't looked) conditional only on the configure flag.

I'm not real sure now why we felt that was a good approach. The general
project policy is that if you ask for a feature in the configure flags,
we'll build it or die trying; how come this specific Python issue gets
special treatment contrary to that policy?

So I'd vote for changing that to put the shared-library test in configure,
or at least make it a build failure later on, not "silently don't build
what was asked for". That would mean olinguito's configure flags would
have to be adjusted.

Plan B would require propagating the shared-libpython test into the
contrib makefiles, which seems pretty unpalatable even if you're willing
to defend the status quo otherwise.

regards, tom lane

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#10Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#8)
Re: pgsql: Add transforms feature

On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in $contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.

OK, attached are two patches to put back the buildfarm nodes using MSVC to green
- 0001 adds support for build and installation of the new transform
modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
this patch is enough to put back the buildfarm to a green state for
MSVC, but it disables the regression tests for those modules,
something addressed in the next patch...
- 0002 adds support for regression tests for the new modules. The
thing is that we need to check the major version version of Python
available in configuration and choose what are the extensions to
preload before the tests run. It is a little bit hacky... But it has
the merit to work, and I am not sure we could have a cleaner solution
by looking at the Makefiles of the transform modules that use
currently conditions based on $(python_majorversion).
Regards,
--
Michael

Attachments:

0001-Fix-MSVC-build-for-transform-modules.patchapplication/x-patch; name=0001-Fix-MSVC-build-for-transform-modules.patchDownload+212-125
0002-Enable-version-specific-tests-of-plpython-transforms.patchapplication/x-patch; name=0002-Enable-version-specific-tests-of-plpython-transforms.patchDownload+27-3
#11Andrew Dunstan
andrew@dunslane.net
In reply to: Michael Paquier (#10)
Re: pgsql: Add transforms feature

On 04/28/2015 09:04 AM, Michael Paquier wrote:

On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in $contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.

OK, attached are two patches to put back the buildfarm nodes using MSVC to green
- 0001 adds support for build and installation of the new transform
modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
this patch is enough to put back the buildfarm to a green state for
MSVC, but it disables the regression tests for those modules,
something addressed in the next patch...
- 0002 adds support for regression tests for the new modules. The
thing is that we need to check the major version version of Python
available in configuration and choose what are the extensions to
preload before the tests run. It is a little bit hacky... But it has
the merit to work, and I am not sure we could have a cleaner solution
by looking at the Makefiles of the transform modules that use
currently conditions based on $(python_majorversion).
Regards,

I have pushed the first of these with some tweaks.

I'm looking at the second.

cheers

andrew

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#12Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#11)
Re: pgsql: Add transforms feature

[switching to -hackers]

On 04/28/2015 11:51 AM, Andrew Dunstan wrote:

On 04/28/2015 09:04 AM, Michael Paquier wrote:

On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in
$contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.

OK, attached are two patches to put back the buildfarm nodes using
MSVC to green
- 0001 adds support for build and installation of the new transform
modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
this patch is enough to put back the buildfarm to a green state for
MSVC, but it disables the regression tests for those modules,
something addressed in the next patch...
- 0002 adds support for regression tests for the new modules. The
thing is that we need to check the major version version of Python
available in configuration and choose what are the extensions to
preload before the tests run. It is a little bit hacky... But it has
the merit to work, and I am not sure we could have a cleaner solution
by looking at the Makefiles of the transform modules that use
currently conditions based on $(python_majorversion).
Regards,

I have pushed the first of these with some tweaks.

I'm looking at the second.

Regarding this second patch - apart from the buggy python version test
which I just fixed in the first patch, I see this:

    +       if ($pyver eq "2")
    +       {
    +           push @opts, "--load-extension=plpythonu";
    +           push @opts, '--load-extension=' . $module . 'u';
    +       }

But what do we do for Python3?

Do we actually have a Windows machine building with Python3?

cheers

andrew

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

#13Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#12)
Re: [COMMITTERS] pgsql: Add transforms feature

Oops, thought I'd changed the address line.

[switching to -hackers]

On 04/28/2015 11:51 AM, Andrew Dunstan wrote:

On 04/28/2015 09:04 AM, Michael Paquier wrote:

On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in
$contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.

OK, attached are two patches to put back the buildfarm nodes using
MSVC to green
- 0001 adds support for build and installation of the new transform
modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
this patch is enough to put back the buildfarm to a green state for
MSVC, but it disables the regression tests for those modules,
something addressed in the next patch...
- 0002 adds support for regression tests for the new modules. The
thing is that we need to check the major version version of Python
available in configuration and choose what are the extensions to
preload before the tests run. It is a little bit hacky... But it has
the merit to work, and I am not sure we could have a cleaner solution
by looking at the Makefiles of the transform modules that use
currently conditions based on $(python_majorversion).
Regards,

I have pushed the first of these with some tweaks.

I'm looking at the second.

Regarding this second patch - apart from the buggy python version test
which I just fixed in the first patch, I see this:

    +       if ($pyver eq "2")
    +       {
    +           push @opts, "--load-extension=plpythonu";
    +           push @opts, '--load-extension=' . $module . 'u';
    +       }

But what do we do for Python3?

Do we actually have a Windows machine building with Python3?

cheers

andrew

--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#12)
Re: [COMMITTERS] pgsql: Add transforms feature

On 04/28/2015 12:03 PM, Andrew Dunstan wrote:

[switching to -hackers]

On 04/28/2015 11:51 AM, Andrew Dunstan wrote:

On 04/28/2015 09:04 AM, Michael Paquier wrote:

On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:

w.r.t MSVC builds, it looks like we need entries in
$contrib_extraincludes
in src/tools/msvc/Mkvcbuild.pm at the very least.

If our goal is to put back to green the Windows nodes as quick as
possible, we could bypass their build this way , and we'll
additionally need to update the install script and
vcregress.pl/contribcheck to bypass those modules accordingly. Now I
don't think that we should make the things produced inconsistent.

OK, attached are two patches to put back the buildfarm nodes using
MSVC to green
- 0001 adds support for build and installation of the new transform
modules: hstore_plperl, hstore_plpython and ltree_plpython. Note that
this patch is enough to put back the buildfarm to a green state for
MSVC, but it disables the regression tests for those modules,
something addressed in the next patch...
- 0002 adds support for regression tests for the new modules. The
thing is that we need to check the major version version of Python
available in configuration and choose what are the extensions to
preload before the tests run. It is a little bit hacky... But it has
the merit to work, and I am not sure we could have a cleaner solution
by looking at the Makefiles of the transform modules that use
currently conditions based on $(python_majorversion).
Regards,

I have pushed the first of these with some tweaks.

I'm looking at the second.

Regarding this second patch - apart from the buggy python version test
which I just fixed in the first patch, I see this:

+       if ($pyver eq "2")
+       {
+           push @opts, "--load-extension=plpythonu";
+           push @opts, '--load-extension=' . $module . 'u';
+       }

But what do we do for Python3?

Do we actually have a Windows machine building with Python3?

The answer seems to be "probably not". When I tried enabling this with
bowerbird I got a ton of errors like these:

plpy_cursorobject.obj : error LNK2001: unresolved external symbol
PyObject_SelfIter [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
plpy_cursorobject.obj : error LNK2019: unresolved external symbol
__imp_PyType_Ready referenced in function PLy_cursor_init_type
[C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]

Something else to fix I guess.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#15Peter Eisentraut
peter_e@gmx.net
In reply to: Andrew Dunstan (#14)
Re: [COMMITTERS] pgsql: Add transforms feature

On 4/28/15 4:10 PM, Andrew Dunstan wrote:

Do we actually have a Windows machine building with Python3?

The answer seems to be "probably not". When I tried enabling this with
bowerbird I got a ton of errors like these:

plpy_cursorobject.obj : error LNK2001: unresolved external symbol
PyObject_SelfIter [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
plpy_cursorobject.obj : error LNK2019: unresolved external symbol
__imp_PyType_Ready referenced in function PLy_cursor_init_type
[C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]

Something else to fix I guess.

I think at least at one point the community Windows binaries built by
EnterpriseDB were built against Python 3 (which upset some users). But
they might not be using the msvc toolchain.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#9)
shared_libperl, shared_libpython

On 4/28/15 12:05 AM, Tom Lane wrote:

Yeah. Even more specifically, olinguito does have --with-python in its
configure flags, but then the plpython Makefile skips the build because
libpython isn't available as a shared library. But the contrib test is
(I assume, haven't looked) conditional only on the configure flag.

I'm not real sure now why we felt that was a good approach. The general
project policy is that if you ask for a feature in the configure flags,
we'll build it or die trying; how come this specific Python issue gets
special treatment contrary to that policy?

So I'd vote for changing that to put the shared-library test in configure,
or at least make it a build failure later on, not "silently don't build
what was asked for". That would mean olinguito's configure flags would
have to be adjusted.

Plan B would require propagating the shared-libpython test into the
contrib makefiles, which seems pretty unpalatable even if you're willing
to defend the status quo otherwise.

I had tried plan B prime, moving the shared_libpython etc. detection
into Makefile.global. But that doesn't work because contrib/pgxs
makefiles require setting all variables *before* including the global
makefiles. So you can't decide whether you want to build something
before you have told it everything you want to build.

The reason for the current setup is actually that when plperl and later
plpython was added, we still had Perl and Python client modules in our
tree (Pg.pm and pygresql), and configure --with-perl and --with-python
were meant to activate their build primarily. Also, in those days,
having a shared libperl or libpython was rare. But we didn't want to
fail the frontend interface builds because of that. So we arrived at
the current workaround.

My preference would be to rip all that out and let the compiler or
linker decide when it doesn't want to link something.

The alternative would be creating a configure check that mirrors the
current logic. Arguably, however, the current logic is quite unworthy
of a configure check, because it's just a collection of
on-this-platform-do-that, instead of a real test. Then again, a real
test would require building and loading a shared library in configure,
and we are not set up for that.

Preferences?

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#17Andrew Dunstan
andrew@dunslane.net
In reply to: Peter Eisentraut (#16)
Re: shared_libperl, shared_libpython

On 04/28/2015 04:31 PM, Peter Eisentraut wrote:

On 4/28/15 12:05 AM, Tom Lane wrote:

Yeah. Even more specifically, olinguito does have --with-python in its
configure flags, but then the plpython Makefile skips the build because
libpython isn't available as a shared library. But the contrib test is
(I assume, haven't looked) conditional only on the configure flag.

I'm not real sure now why we felt that was a good approach. The general
project policy is that if you ask for a feature in the configure flags,
we'll build it or die trying; how come this specific Python issue gets
special treatment contrary to that policy?

So I'd vote for changing that to put the shared-library test in configure,
or at least make it a build failure later on, not "silently don't build
what was asked for". That would mean olinguito's configure flags would
have to be adjusted.

Plan B would require propagating the shared-libpython test into the
contrib makefiles, which seems pretty unpalatable even if you're willing
to defend the status quo otherwise.

I had tried plan B prime, moving the shared_libpython etc. detection
into Makefile.global. But that doesn't work because contrib/pgxs
makefiles require setting all variables *before* including the global
makefiles. So you can't decide whether you want to build something
before you have told it everything you want to build.

The reason for the current setup is actually that when plperl and later
plpython was added, we still had Perl and Python client modules in our
tree (Pg.pm and pygresql), and configure --with-perl and --with-python
were meant to activate their build primarily. Also, in those days,
having a shared libperl or libpython was rare. But we didn't want to
fail the frontend interface builds because of that. So we arrived at
the current workaround.

My preference would be to rip all that out and let the compiler or
linker decide when it doesn't want to link something.

The alternative would be creating a configure check that mirrors the
current logic. Arguably, however, the current logic is quite unworthy
of a configure check, because it's just a collection of
on-this-platform-do-that, instead of a real test. Then again, a real
test would require building and loading a shared library in configure,
and we are not set up for that.

Preferences?

In general I prefer things not being available to be tested at configure
time. After all, we check for libxml2, for example. Certainly silent
failure is a bad thing.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#16)
Re: shared_libperl, shared_libpython

Peter Eisentraut <peter_e@gmx.net> writes:

On 4/28/15 12:05 AM, Tom Lane wrote:

Yeah. Even more specifically, olinguito does have --with-python in its
configure flags, but then the plpython Makefile skips the build because
libpython isn't available as a shared library. But the contrib test is
(I assume, haven't looked) conditional only on the configure flag.

I'm not real sure now why we felt that was a good approach. The general
project policy is that if you ask for a feature in the configure flags,
we'll build it or die trying; how come this specific Python issue gets
special treatment contrary to that policy?

The reason for the current setup is actually that when plperl and later
plpython was added, we still had Perl and Python client modules in our
tree (Pg.pm and pygresql), and configure --with-perl and --with-python
were meant to activate their build primarily. Also, in those days,
having a shared libperl or libpython was rare. But we didn't want to
fail the frontend interface builds because of that. So we arrived at
the current workaround.

Ah. I'm glad you remember, because I didn't.

My preference would be to rip all that out and let the compiler or
linker decide when it doesn't want to link something.

Works for me, assuming that we get an understandable failure message and
not, say, a plperl.so that mysteriously doesn't work.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#19Michael Paquier
michael@paquier.xyz
In reply to: Peter Eisentraut (#15)
Re: [COMMITTERS] pgsql: Add transforms feature

On Wed, Apr 29, 2015 at 5:15 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

On 4/28/15 4:10 PM, Andrew Dunstan wrote:

Do we actually have a Windows machine building with Python3?

The answer seems to be "probably not". When I tried enabling this with
bowerbird I got a ton of errors like these:

plpy_cursorobject.obj : error LNK2001: unresolved external symbol
PyObject_SelfIter [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
plpy_cursorobject.obj : error LNK2019: unresolved external symbol
__imp_PyType_Ready referenced in function PLy_cursor_init_type
[C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]

Something else to fix I guess.

I think at least at one point the community Windows binaries built by
EnterpriseDB were built against Python 3 (which upset some users). But
they might not be using the msvc toolchain.

Er, aren't you simply trying to link with 32b libraries while building
in a 64b environment? I am able to make the build work with python 3.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#20Michael Paquier
michael@paquier.xyz
In reply to: Tom Lane (#18)
Re: shared_libperl, shared_libpython

On Wed, Apr 29, 2015 at 12:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

On 4/28/15 12:05 AM, Tom Lane wrote:

Yeah. Even more specifically, olinguito does have --with-python in its
configure flags, but then the plpython Makefile skips the build because
libpython isn't available as a shared library. But the contrib test is
(I assume, haven't looked) conditional only on the configure flag.

I'm not real sure now why we felt that was a good approach. The general
project policy is that if you ask for a feature in the configure flags,
we'll build it or die trying; how come this specific Python issue gets
special treatment contrary to that policy?

The reason for the current setup is actually that when plperl and later
plpython was added, we still had Perl and Python client modules in our
tree (Pg.pm and pygresql), and configure --with-perl and --with-python
were meant to activate their build primarily. Also, in those days,
having a shared libperl or libpython was rare. But we didn't want to
fail the frontend interface builds because of that. So we arrived at
the current workaround.

Ah. I'm glad you remember, because I didn't.

Interesting, those are pieces of history:
commit: f10a9033bf308f9dde0aa77caad6503e233489d1
author: Peter Eisentraut <peter_e@gmx.net>
date: Mon, 1 Sep 2003 23:01:49 +0000
Clean up after pygresql removal: adjust/remove documentation and remove
unneeded configure work.

commit: 9a0b4d7f847469544798133391e221481548e1b9
author: Marc G. Fournier <scrappy@hub.org>
date: Fri, 30 Aug 2002 13:06:22 +0000

perl5 interface moved to gborg

My preference would be to rip all that out and let the compiler or
linker decide when it doesn't want to link something.

Works for me, assuming that we get an understandable failure message and
not, say, a plperl.so that mysteriously doesn't work.

+1.
-- 
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#21Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#14)
#22Michael Paquier
michael@paquier.xyz
In reply to: Andrew Dunstan (#21)
#23Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#18)
#24Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#2)
#25Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#24)
#26Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#24)
#27Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#26)
#28Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#27)
#29Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#28)
#30Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Dunstan (#29)
#31Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#30)
#32Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#25)
#33Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#32)
#34Noah Misch
noah@leadboat.com
In reply to: Peter Eisentraut (#32)