pl/perl extension fails on Windows

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:

I compiled PG 10 beta1/beta2 with "--with-perl" option on Windows and the
extension crashes the database.
*src/pl/plperl/Util.c: loadable library and perl binaries are mismatched
(got handshake key 0A900080, needed 0AC80080)*

This is seen with Perl 5.24 but not with 5.20, 5.16. What I found is that
the handshake function is added in Perl 5.21.x and probably that is why we
don't see this issue in earlier versions.

Well, we have various buildfarm machines running perls newer than that,
eg, crake, with 5.24.1. So I'd say there is something busted about your
perl installation. Perhaps leftover bits of an older version somewhere?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jul 12, 2017 at 4:35 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:

I compiled PG 10 beta1/beta2 with "--with-perl" option on Windows and the
extension crashes the database.
*src/pl/plperl/Util.c: loadable library and perl binaries are mismatched
(got handshake key 0A900080, needed 0AC80080)*

This is seen with Perl 5.24 but not with 5.20, 5.16. What I found is that
the handshake function is added in Perl 5.21.x and probably that is why

we

don't see this issue in earlier versions.

Well, we have various buildfarm machines running perls newer than that,
eg, crake, with 5.24.1. So I'd say there is something busted about your
perl installation. Perhaps leftover bits of an older version somewhere?

Well crake is a Fedora box - and we have no problems on Linux, only on
Windows.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 07/12/2017 11:49 AM, Dave Page wrote:

On Wed, Jul 12, 2017 at 4:35 PM, Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com
<mailto:sandeep.thakkar@enterprisedb.com>> writes:

I compiled PG 10 beta1/beta2 with "--with-perl" option on

Windows and the

extension crashes the database.
*src/pl/plperl/Util.c: loadable library and perl binaries are

mismatched

(got handshake key 0A900080, needed 0AC80080)*

This is seen with Perl 5.24 but not with 5.20, 5.16. What I

found is that

the handshake function is added in Perl 5.21.x and probably that

is why we

don't see this issue in earlier versions.

Well, we have various buildfarm machines running perls newer than
that,
eg, crake, with 5.24.1. So I'd say there is something busted
about your
perl installation. Perhaps leftover bits of an older version
somewhere?

Well crake is a Fedora box - and we have no problems on Linux, only on
Windows.

Yeah, I have this on one of my Windows boxes, and haven't had time to
get to the bottom of it yet ;-(

Latest versions of ActivePerl don't ship with library descriptor files,
either, which is unpleasant.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jul 13, 2017 at 12:01 AM, Andrew Dunstan
<andrew.dunstan@2ndquadrant.com> wrote:

On 07/12/2017 11:49 AM, Dave Page wrote:

Well crake is a Fedora box - and we have no problems on Linux, only on
Windows.

Yeah, I have this on one of my Windows boxes, and haven't had time to
get to the bottom of it yet ;-(

I could also see this problem in my Windows machine and I have spent
some time to analyse it. Here are my findings:

The stacktrace where the crash happens is:

perl524.dll!Perl_xs_handshake(const unsigned long key, void
*v_my_perl, const char * file, ...) Line 5569 C
plperl.dll!boot_PostgreSQL__InServer__Util(interpreter * my_perl, cv
* cv) Line 490 + 0x22 bytes C
perl524.dll!Perl_pp_entersub(interpreter * my_perl) Line 3987 + 0xc bytes C
perl524.dll!Perl_runops_standard(interpreter * my_perl) Line 41 + 0x6 bytes C
perl524.dll!S_run_body(interpreter * my_perl, long oldscope) Line 2485 C
perl524.dll!perl_run(interpreter * my_perl) Line 2406 + 0xc bytes C
plperl.dll!plperl_init_interp() Line 829 + 0xb bytes C
plperl.dll!_PG_init() Line 470 + 0x5 bytes C

I couldn't get much out of above bt, but, one thing i could notice is
that the input key passed to 'Perl_xs_handshake()' function is not
matching with the key being generated inside 'Perl_xs_handshake()',
hence, the handshaking is failing.

Please have a look into the following code snippet from 'perl 5.24'
and 'Util.c' file in plperl respectively,

Perl-5.24:
=======
got = INT2PTR(void*, (UV)(key & HSm_KEY_MATCH));
need = (void *)(HS_KEY(FALSE, FALSE, "", "") & HSm_KEY_MATCH);
if (UNLIKELY(got != need))
goto bad_handshake;

Util.c in plperl:
==========
485 XS_EXTERNAL(boot_PostgreSQL__InServer__Util)
486 {
487 #if PERL_VERSION_LE(5, 21, 5)
488 dVAR; dXSARGS;
489 #else
490 dVAR; dXSBOOTARGSAPIVERCHK;

Actually the macro 'dXSBOOTARGSAPIVERCHK' in line #490 gets expanded to,

#define XS_APIVERSION_SETXSUBFN_POPMARK_BOOTCHECK
\
Perl_xs_handshake(HS_KEY(TRUE, TRUE, "v" PERL_API_VERSION_STRING,
""), \
HS_CXT, __FILE__, "v" PERL_API_VERSION_STRING)

And the point to be noted is, the way, the key is being generated in
above macro and in Perl_xs_handshake(). As shown above,
'XS_APIVERSION_SETXSUBFN_POPMARK_BOOTCHECK' macro passes
'PERL_API_VERSION_STRING' as input to HS_KEY() to generate the key and
this key is passed to Perl_xs_handshake function whereas inside
Perl_xs_handshake(), there is no such version string being used to
generate the key. Hence, the key mismatch is seen thereby resulting in
a bad_handshake error.

After doing some study, I could understand that Util.c is generated
from Util.xs by xsubpp compiler at build time. This is being done in
Mkvcbuild.pm file in postgres. If I manually replace
'dXSBOOTARGSAPIVERCHK' macro with 'dXSBOOTARGSNOVERCHK' macro in
src/pl/plperl/Util.c, the things work fine. The diff is as follows,

XS_EXTERNAL(boot_PostgreSQL__InServer__Util)
{
#if PERL_VERSION_LE(5, 21, 5)
dVAR; dXSARGS;
#else
- dVAR; dXSBOOTARGSAPIVERCHK;
+ dVAR; dXSBOOTARGSNOVERCHK;

I need to further investigate, but let me know if you have any ideas.

Latest versions of ActivePerl don't ship with library descriptor files,
either, which is unpleasant.

I too had similar observation and surprisingly, I am seeing
'libperl*.a' file instead of 'perl*.lib' file in most of the
ActiverPerl versions for Windows.

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 07/13/2017 08:08 AM, Ashutosh Sharma wrote:

After doing some study, I could understand that Util.c is generated
from Util.xs by xsubpp compiler at build time. This is being done in
Mkvcbuild.pm file in postgres. If I manually replace
'dXSBOOTARGSAPIVERCHK' macro with 'dXSBOOTARGSNOVERCHK' macro in
src/pl/plperl/Util.c, the things work fine. The diff is as follows,

XS_EXTERNAL(boot_PostgreSQL__InServer__Util)
{
#if PERL_VERSION_LE(5, 21, 5)
dVAR; dXSARGS;
#else
- dVAR; dXSBOOTARGSAPIVERCHK;
+ dVAR; dXSBOOTARGSNOVERCHK;

I need to further investigate, but let me know if you have any ideas.

Good job hunting this down!

One suggestion I saw in a little googling was that we add this to the XS
file after the inclusion of XSUB.h:

#undef dXSBOOTARGSAPIVERCHK
#define dXSBOOTARGSAPIVERCHK dXSBOOTARGSNOVERCHK

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

On 07/13/2017 08:08 AM, Ashutosh Sharma wrote:

-    dVAR; dXSBOOTARGSAPIVERCHK;
+    dVAR; dXSBOOTARGSNOVERCHK;

Good job hunting this down!
One suggestion I saw in a little googling was that we add this to the XS
file after the inclusion of XSUB.h:
#undef dXSBOOTARGSAPIVERCHK
#define dXSBOOTARGSAPIVERCHK dXSBOOTARGSNOVERCHK

I don't see anything even vaguely like that in the Util.c file generated
by Perl 5.10.1, which is what I've got on my RHEL machine.

What I do notice is this in Util.xs:

VERSIONCHECK: DISABLE

which leads immediately to two questions:

1. Why is your version of xsubpp apparently ignoring this directive
and generating a version check anyway?

2. Why do we have this directive in the first place? It does not seem
to me like a terribly great idea to ignore low-level version mismatches.

In the same vein, I'm suspicious of proposals to "fix" this problem
by removing the version check, which seems to be where Ashutosh
is headed. In the long run that seems certain to cause huge headaches.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 07/13/2017 10:36 AM, Tom Lane wrote:

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

On 07/13/2017 08:08 AM, Ashutosh Sharma wrote:

-    dVAR; dXSBOOTARGSAPIVERCHK;
+    dVAR; dXSBOOTARGSNOVERCHK;

Good job hunting this down!
One suggestion I saw in a little googling was that we add this to the XS
file after the inclusion of XSUB.h:
#undef dXSBOOTARGSAPIVERCHK
#define dXSBOOTARGSAPIVERCHK dXSBOOTARGSNOVERCHK

I don't see anything even vaguely like that in the Util.c file generated
by Perl 5.10.1, which is what I've got on my RHEL machine.

This is all fairly modern, so it's hardly surprising that it doesn't
happen with the ancient perl 5.10.

here's a snippet from the generated Util.c on crake (Fedora 25, perl 5.24):

XS_EXTERNAL(boot_PostgreSQL__InServer__Util); /* prototype to pass
-Wmissing-prototypes */
XS_EXTERNAL(boot_PostgreSQL__InServer__Util)
{
#if PERL_VERSION_LE(5, 21, 5)
dVAR; dXSARGS;
#else
dVAR; dXSBOOTARGSAPIVERCHK;
#endif

What I do notice is this in Util.xs:

VERSIONCHECK: DISABLE

which leads immediately to two questions:

1. Why is your version of xsubpp apparently ignoring this directive
and generating a version check anyway?

2. Why do we have this directive in the first place? It does not seem
to me like a terribly great idea to ignore low-level version mismatches.

In the same vein, I'm suspicious of proposals to "fix" this problem
by removing the version check, which seems to be where Ashutosh
is headed. In the long run that seems certain to cause huge headaches.

That is a different version check. It's the equivalent of xsubpp's
--noversioncheck flag. The versions it would check are the object file
and the corresponding pm file.

In fact the usage I suggested seems to be blessed in XSUB.h in this comment:

/* dXSBOOTARGSNOVERCHK has no API in xsubpp to choose it so do
#undef dXSBOOTARGSXSAPIVERCHK
#define dXSBOOTARGSXSAPIVERCHK dXSBOOTARGSNOVERCHK */

It would be nice to get to the bottom of why we're getting a version
mismatch on Windows, since we're clearly not getting one on Linux. But
since we've got on happily all these years without the API version check
we might well survive a few more going on as we are.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

It would be nice to get to the bottom of why we're getting a version
mismatch on Windows, since we're clearly not getting one on Linux.

Yeah, that's what's bothering me: as long as that remains unexplained,
I don't have any confidence that we're fixing the right thing.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: Dave Page 2017-07-12 <CA+OCxox=h5xftcnyk1AfA_QKSQSruD=Er0qweZakiCEg3U4rcg@mail.gmail.com>

Well, we have various buildfarm machines running perls newer than that,
eg, crake, with 5.24.1. So I'd say there is something busted about your
perl installation. Perhaps leftover bits of an older version somewhere?

Well crake is a Fedora box - and we have no problems on Linux, only on
Windows.

The plperl segfault on Debian's kfreebsd port I reported back in 2013
is also still present:

/messages/by-id/20130515064201.GC704@msgid.df7cb.de

https://buildd.debian.org/status/fetch.php?pkg=postgresql-10&amp;arch=kfreebsd-amd64&amp;ver=10~beta2-1&amp;stamp=1499947011&amp;raw=0

(Arguably, this is a toy architecture, so we can just leave it unfixed
there without any harm...)

Christoph

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jul 13, 2017 at 6:04 PM, Andrew Dunstan
<andrew.dunstan@2ndquadrant.com> wrote:

On 07/13/2017 08:08 AM, Ashutosh Sharma wrote:

After doing some study, I could understand that Util.c is generated
from Util.xs by xsubpp compiler at build time. This is being done in
Mkvcbuild.pm file in postgres. If I manually replace
'dXSBOOTARGSAPIVERCHK' macro with 'dXSBOOTARGSNOVERCHK' macro in
src/pl/plperl/Util.c, the things work fine. The diff is as follows,

XS_EXTERNAL(boot_PostgreSQL__InServer__Util)
{
#if PERL_VERSION_LE(5, 21, 5)
dVAR; dXSARGS;
#else
- dVAR; dXSBOOTARGSAPIVERCHK;
+ dVAR; dXSBOOTARGSNOVERCHK;

I need to further investigate, but let me know if you have any ideas.

Good job hunting this down!

One suggestion I saw in a little googling was that we add this to the XS
file after the inclusion of XSUB.h:

#undef dXSBOOTARGSAPIVERCHK
#define dXSBOOTARGSAPIVERCHK dXSBOOTARGSNOVERCHK

Thanks for that suggestion. It was really helpful. I think, the point
is, in XSUB.h, the macro 'dXSBOOTARGSXSAPIVERCHK' has been redefined
for novercheck but surprisingly it hasn't been done for
'dXSBOOTARGSAPIVERCHK' macro and that could be the reason why we want
to redefine it in the XS file after including XSUB.h file. Attached is
the patch that redefines 'dXSBOOTARGSAPIVERCHK' in Util.xs and SPI.xs
files. Please have a look into the attached patch and let me know your
comments. Thanks.

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com

On Thu, Jul 13, 2017 at 10:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

It would be nice to get to the bottom of why we're getting a version
mismatch on Windows, since we're clearly not getting one on Linux.

Yeah, that's what's bothering me: as long as that remains unexplained,
I don't have any confidence that we're fixing the right thing.

Okay, I tried to explore on this a bit and my findings are as follows.

On Linux, the handshake key being generated in plperl code i.e. inside
XS_EXTERNAL() in Util.c (generated from Util.xs during build time) and perl
code (inside Perl_xs_handshake function) are same which means the following
condition inside Perl_xs_handshake() becomes true and therefore, there is
no mismatch error.

got = INT2PTR(void*, (UV)(key & HSm_KEY_MATCH));
need = (void *)(HS_KEY(FALSE, FALSE, "", "") & HSm_KEY_MATCH);
if (UNLIKELY(got != need))
goto bad_handshake;

However, on Windows, the handshake key generated in plperl code inside
XS_EXTERNAL() and in perl code i.e. inside Perl_xs_handshake() are
different thereby resulting in a mismatch error.

Actually the function used for generation of handshake Key i.e HS_KEYp()
considers 'sizeof(PerlInterpreter)' to generate the key and somehow the
sizeof PerlInterpreter is not uniform in plperl and perl modules incase of
Windows but on Linux it remains same in both the modules.

This is how PerlInterpreter is defined in Perl source,

*typedef struct interpreter PerlInterpreter;struct interpreter {# include
"intrpvar.h"};*

where intrpvar.h has different variables defined inside it and most of the
variables definition are protected with various macros. And there are some
macros that are just defined in perl but not in plperl module which means
the sizeof(PerlInterpreter) on the two modules are going to be different
and thereby resulting in a different key. But, then the point is, why no
such difference is observed on Linux. Well, as of now, i haven't found the
reason behind it and i am still investigating on it.

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Actually the function used for generation of handshake Key i.e HS_KEYp()
considers 'sizeof(PerlInterpreter)' to generate the key and somehow the
sizeof PerlInterpreter is not uniform in plperl and perl modules incase of
Windows but on Linux it remains same in both the modules.

Yipes. So actually, this is catching a live ABI problem, which presumably
we've escaped seeing bad effects from only through sheer good luck.
I suppose that the discrepancies in the struct contents only occur after
the last field that plperl happens to touch directly --- but that is
unlikely to be true forever.

*typedef struct interpreter PerlInterpreter;struct interpreter {# include
"intrpvar.h"};*
where intrpvar.h has different variables defined inside it and most of the
variables definition are protected with various macros. And there are some
macros that are just defined in perl but not in plperl module which means
the sizeof(PerlInterpreter) on the two modules are going to be different
and thereby resulting in a different key.

I imagine the route to a solution is to fix things so that the relevant
macros are all defined correctly in both cases. But why they aren't
already is certainly an interesting question. Have you identified just
which fields are added or missing relative to what libperl thinks?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jul 19, 2017 at 12:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Actually the function used for generation of handshake Key i.e HS_KEYp()
considers 'sizeof(PerlInterpreter)' to generate the key and somehow the
sizeof PerlInterpreter is not uniform in plperl and perl modules incase of
Windows but on Linux it remains same in both the modules.

Yipes.

+1 for "yipes". It sounds like we should really try to fix the
underlying problem, rather than just working around the check.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jul 19, 2017 at 9:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Actually the function used for generation of handshake Key i.e HS_KEYp()
considers 'sizeof(PerlInterpreter)' to generate the key and somehow the
sizeof PerlInterpreter is not uniform in plperl and perl modules incase of
Windows but on Linux it remains same in both the modules.

Yipes. So actually, this is catching a live ABI problem, which presumably
we've escaped seeing bad effects from only through sheer good luck.
I suppose that the discrepancies in the struct contents only occur after
the last field that plperl happens to touch directly --- but that is
unlikely to be true forever.

*typedef struct interpreter PerlInterpreter;struct interpreter {# include
"intrpvar.h"};*
where intrpvar.h has different variables defined inside it and most of the
variables definition are protected with various macros. And there are some
macros that are just defined in perl but not in plperl module which means
the sizeof(PerlInterpreter) on the two modules are going to be different
and thereby resulting in a different key.

I imagine the route to a solution is to fix things so that the relevant
macros are all defined correctly in both cases. But why they aren't
already is certainly an interesting question. Have you identified just
which fields are added or missing relative to what libperl thinks?

Here are the list of macros and variables from 'intrpvar.h' file that
are just defined in perl module but not in plperl on Windows,

#ifdef PERL_USES_PL_PIDSTATUS
PERLVAR(I, pidstatus, HV *) /* pid-to-status mappings for waitpid */
#endif

#ifdef PERL_SAWAMPERSAND
PERLVAR(I, sawampersand, U8) /* must save all match strings */
#endif

#ifdef FCRYPT
PERLVARI(I, cryptseen, bool, FALSE) /* has fast crypt() been initialized? */
#else
/* One byte hole in the interpreter structure. */
#endif

#ifdef USE_REENTRANT_API
PERLVAR(I, reentrant_buffer, REENTR *) /* here we store the _r buffers */
#endif

#ifdef PERL_GLOBAL_STRUCT_PRIVATE
PERLVARI(I, my_cxt_keys, const char **, NULL) /* per-module array of
pointers to MY_CXT_KEY constants */
# endif

#ifdef DEBUG_LEAKING_SCALARS_FORK_DUMP
/* File descriptor to talk to the child which dumps scalars. */
PERLVARI(I, dumper_fd, int, -1)
#endif

#ifdef DEBUG_LEAKING_SCALARS
PERLVARI(I, sv_serial, U32, 0) /* SV serial number, used in sv.c */
#endif

#ifdef PERL_TRACE_OPS
PERLVARA(I, op_exec_cnt, OP_max+2, UV)
#endif

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

On Wed, Jul 19, 2017 at 9:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I imagine the route to a solution is to fix things so that the relevant
macros are all defined correctly in both cases. But why they aren't
already is certainly an interesting question. Have you identified just
which fields are added or missing relative to what libperl thinks?

Here are the list of macros and variables from 'intrpvar.h' file that
are just defined in perl module but not in plperl on Windows,

#ifdef PERL_USES_PL_PIDSTATUS
PERLVAR(I, pidstatus, HV *) /* pid-to-status mappings for waitpid */
#endif

#ifdef PERL_SAWAMPERSAND
PERLVAR(I, sawampersand, U8) /* must save all match strings */
#endif

#ifdef FCRYPT
PERLVARI(I, cryptseen, bool, FALSE) /* has fast crypt() been initialized? */
#else
/* One byte hole in the interpreter structure. */
#endif

#ifdef USE_REENTRANT_API
PERLVAR(I, reentrant_buffer, REENTR *) /* here we store the _r buffers */
#endif

#ifdef PERL_GLOBAL_STRUCT_PRIVATE
PERLVARI(I, my_cxt_keys, const char **, NULL) /* per-module array of
pointers to MY_CXT_KEY constants */
# endif

#ifdef DEBUG_LEAKING_SCALARS_FORK_DUMP
/* File descriptor to talk to the child which dumps scalars. */
PERLVARI(I, dumper_fd, int, -1)
#endif

#ifdef DEBUG_LEAKING_SCALARS
PERLVARI(I, sv_serial, U32, 0) /* SV serial number, used in sv.c */
#endif

#ifdef PERL_TRACE_OPS
PERLVARA(I, op_exec_cnt, OP_max+2, UV)
#endif

Huh. So those seem like symbols that ought to be exposed somewhere in
Perl's headers. Perhaps we're failing to #include some "perl_config.h" or
equivalent file that records these ABI options?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Here are the list of macros and variables from 'intrpvar.h' file that
are just defined in perl module but not in plperl on Windows,

#ifdef PERL_USES_PL_PIDSTATUS
PERLVAR(I, pidstatus, HV *) /* pid-to-status mappings for waitpid */
#endif

#ifdef PERL_SAWAMPERSAND
PERLVAR(I, sawampersand, U8) /* must save all match strings */
#endif

I am really suspicious that this means your libperl was built in an unsafe
fashion, that is, by injecting configuration choices as random -D switches
in the build process rather than making sure the choices were recorded in
perl's config.h. As an example, looking at the perl 5.24.1 headers on
a Fedora box, it looks to me like PERL_SAWAMPERSAND could only get defined
if PERL_COPY_ON_WRITE were not defined, and the only way that that can
happen is if PERL_NO_COW is defined, and there are no references to the
latter anyplace except in this particular #if defined test in perl.h.

Where did your perl installation come from, anyway? Are you sure the .h
files match up with the executables?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jul 19, 2017 at 05:01:31PM -0400, Tom Lane wrote:

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Here are the list of macros and variables from 'intrpvar.h' file that
are just defined in perl module but not in plperl on Windows,

#ifdef PERL_USES_PL_PIDSTATUS
PERLVAR(I, pidstatus, HV *) /* pid-to-status mappings for waitpid */
#endif

#ifdef PERL_SAWAMPERSAND
PERLVAR(I, sawampersand, U8) /* must save all match strings */
#endif

I am really suspicious that this means your libperl was built in an unsafe
fashion, that is, by injecting configuration choices as random -D switches
in the build process rather than making sure the choices were recorded in
perl's config.h. As an example, looking at the perl 5.24.1 headers on
a Fedora box, it looks to me like PERL_SAWAMPERSAND could only get defined
if PERL_COPY_ON_WRITE were not defined, and the only way that that can
happen is if PERL_NO_COW is defined, and there are no references to the
latter anyplace except in this particular #if defined test in perl.h.

Where did your perl installation come from, anyway? Are you sure the .h
files match up with the executables?

I see corresponding symptoms with the following Perl distributions:

strawberry-perl-5.26.0.1-64bit.msi:
src/pl/plperl/Util.c: loadable library and perl binaries are mismatched (got handshake key 0000000011800080, needed 0000000011c00080)
ActivePerl-5.24.1.2402-MSWin32-x64-401627.exe:
src/pl/plperl/Util.c: loadable library and perl binaries are mismatched (got handshake key 0000000011500080, needed 0000000011900080)

So, this affects each of the two prominent families of Perl Windows binaries.
Notes for anyone trying to reproduce:

- Both of those Perl distributions require the hacks described here:
/messages/by-id/CABcP5fjEjgOsh097cWnQrsK9yCswo4DZxp-V47DKCH-MxY9Gig@mail.gmail.com
- Add PERL_USE_UNSAFE_INC=1 to the environment until we update things to cope
with this Perl 5.26 change:
http://search.cpan.org/~xsawyerx/perl-5.26.0/pod/perldelta.pod#Removal_of_the_current_directory_(%22.%22)_from_@INC

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Jul 24, 2017 at 11:58 AM, Noah Misch <noah@leadboat.com> wrote:

On Wed, Jul 19, 2017 at 05:01:31PM -0400, Tom Lane wrote:

Ashutosh Sharma <ashu.coek88@gmail.com> writes:

Here are the list of macros and variables from 'intrpvar.h' file that
are just defined in perl module but not in plperl on Windows,

#ifdef PERL_USES_PL_PIDSTATUS
PERLVAR(I, pidstatus, HV *) /* pid-to-status mappings for waitpid */
#endif

#ifdef PERL_SAWAMPERSAND
PERLVAR(I, sawampersand, U8) /* must save all match strings */
#endif

I am really suspicious that this means your libperl was built in an unsafe
fashion, that is, by injecting configuration choices as random -D switches
in the build process rather than making sure the choices were recorded in
perl's config.h. As an example, looking at the perl 5.24.1 headers on
a Fedora box, it looks to me like PERL_SAWAMPERSAND could only get defined
if PERL_COPY_ON_WRITE were not defined, and the only way that that can
happen is if PERL_NO_COW is defined, and there are no references to the
latter anyplace except in this particular #if defined test in perl.h.

Where did your perl installation come from, anyway? Are you sure the .h
files match up with the executables?

I see corresponding symptoms with the following Perl distributions:

strawberry-perl-5.26.0.1-64bit.msi:
src/pl/plperl/Util.c: loadable library and perl binaries are mismatched (got handshake key 0000000011800080, needed 0000000011c00080)
ActivePerl-5.24.1.2402-MSWin32-x64-401627.exe:
src/pl/plperl/Util.c: loadable library and perl binaries are mismatched (got handshake key 0000000011500080, needed 0000000011900080)

So, this affects each of the two prominent families of Perl Windows binaries.

I think the real question is where do we go from here. Ashutosh has
proposed a patch up-thread based on a suggestion from Andrew, but it
is not clear if we want to go there as that seems to be bypassing
handshake mechanism. The other tests and analysis seem to indicate
that the new version Perl binaries on Windows are getting built with
flags that are incompatible with what we use for plperl and it is not
clear why perl is using those flags. Do you think we can do something
at our end to make it work or someone should check with Perl community
about it?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jul 19, 2017 at 5:01 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I am really suspicious that this means your libperl was built in an unsafe
fashion, that is, by injecting configuration choices as random -D switches
in the build process rather than making sure the choices were recorded in
perl's config.h. As an example, looking at the perl 5.24.1 headers on
a Fedora box, it looks to me like PERL_SAWAMPERSAND could only get defined
if PERL_COPY_ON_WRITE were not defined, and the only way that that can
happen is if PERL_NO_COW is defined, and there are no references to the
latter anyplace except in this particular #if defined test in perl.h.

Hmm, it might not be so random as all that. Have a look at this
commit log entry:

commit 1a904fc88069e249a4bd0ef196a3f1a7f549e0fe
Author: Father Chrysostomos <sprout@cpan.org>
Date: Sun Nov 25 12:57:04 2012 -0800

Disable PL_sawampersand

PL_sawampersand actually causes bugs (e.g., perl #4289), because the
behaviour changes. eval '$&' after a match will produce different
results depending on whether $& was seen before the match.

Using copy-on-write for the pre-match copy (preceding patches do that)
alleviates the slowdown caused by mentioning $&. The copy doesn’t
happen unless the string is modified after the match. It’s now a
post- match copy. So we no longer need to do things differently
depending on whether $& has been seen.

PL_sawampersand is now #defined to be equal to what it would be if
every program began with $',$&,$`.

I left the PL_sawampersand code in place, in case this commit proves
immature. Running Configure with -Accflags=PERL_SAWAMPERSAND will
reënable the PL_sawampersand mechanism.

Based on a bit of experimentation, that last bit contains a typo: it
should say -Accflags=-DPERL_SAWAMPERSAND; as written, the -D is
missing.[1]Arguably, the umlaut over "reenable" is also a typo, but that's a sort of in a different category. Anyway, the point is that at least in this case, there
seems to have been some idea that somebody might want to reenable this
in their own build even after it was disabled by default.

Perl also has a mechanism for flags added to Configure to be passed
along when building loadable modules; if it didn't, not just plperl
but every Perl module written in C would have this issue if any such
flags where used. Normally, you compile perl modules by running "perl
Makefile.PL" to generate a makefile, and then building from the
makefile. If you do that, then the Makefile ends up with a section in
it that looks like this:

# --- MakeMaker cflags section:

CCFLAGS = -fno-strict-aliasing -pipe -fstack-protector-strong
-I/opt/local/include -DPERL_USE_SAFE_PUTENV -DPERL_SAWAMPERSAND -Wall
-Werror=declaration-after-statement -Wextra -Wc++-compat
-Wwrite-strings
OPTIMIZE = -O3
PERLTYPE =
MPOLLUTE =

...and lo-and-behold, the -DPERL_SAWAMPERSAND flag which I passed to
Configure is there. After a bit of time deciphering how MakeMaker
actually works, I figured out that it gets the value for CFLAGS by
doing "use Config;" and then referencing $Config::Config{'ccflags'};
an alternative way to get it, from the shell, is to run perl
-V:ccflags.

While I'm not sure of the details, I suspect that we need to use one
of those methods to get the CCFLAGS used to build perl, and include
those when SPI.o, Util.o, and plperl.o in src/pl/plperl. Or at least
the -D switches from those CCFLAGS. Here's about the simplest thing
that seems like it might work on Linux; Windows would need something
equivalent:

override CPPFLAGS += $(shell $(PERL) -MConfig -e 'print
$$Config::Config{"ccflags"};')

On my MacBook Pro, with the built-in switches, that produces:

-fno-common -DPERL_DARWIN -mmacosx-version-min=10.12 -pipe -Os
-fno-strict-aliasing -fstack-protector-strong -I/opt/local/include
-DPERL_USE_SAFE_PUTENV

Or we could try to extract just the -D switches:

override CPPFLAGS += $(shell $(PERL) -MConfig -e 'print join " ", grep
{ /^-D/ } split /\s+/, $$Config::Config{"ccflags"};')

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[1]: Arguably, the umlaut over "reenable" is also a typo, but that's a sort of in a different category.
sort of in a different category.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers