meson vs windows perl
meson.build has this code
ldopts = run_command(perl, '-MExtUtils::Embed', '-e', 'ldopts',
check: true).stdout().strip() undesired =
run_command(perl_conf_cmd, 'ccdlflags', check:
true).stdout().split() undesired += run_command(perl_conf_cmd,
'ldflags', check: true).stdout().split() perl_ldopts = []
foreach ldopt : ldopts.split(' ') if ldopt == '' or ldopt in
undesired continue endif perl_ldopts +=
ldopt.strip('"') endforeach message('LDFLAGS recommended by
perl: "@0@"'.format(ldopts)) message('LDFLAGS for embedding
perl: "@0@"'.format(' '.join(perl_ldopts)))
This code is seriously broken if perl reports items including spaces,
when a) removing the quotes is quite wrong, and b) splitting on spaces
is also wrong.
Here's an example from one of my colleagues:
C:\Program Files\Microsoft Visual Studio\2022\Professional>perl.EXE -MExtUtils::Embed -e ldopts
-nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"C:\edb\languagepack\v4\Perl-5.38\lib\CORE"
-machine:AMD64 -subsystem:console,"5.02" "C:\edb\languagepack\v4\Perl-5.38\lib\CORE\perl538.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\oldnames.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\kernel32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\user32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\gdi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\winspool.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\comdlg32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\advapi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\shell32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\ole32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\oleaut32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\netapi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\uuid.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\ws2_32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\mpr.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\winmm.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\version.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\odbc32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\odbccp32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\comctl32.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\msvcrt.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\vcruntime.lib"
"C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64\ucrt.lib"
And with that we get errors like
cl : Command line warning D9024 : unrecognized source file type 'C:\Program', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Files\Microsoft', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Visual', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'C:\Program', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Files', object file assumed
cl : Command line warning D9024 : unrecognized source file type '(x86)\Windows', object file assumed
It looks like we need to get smarter about how we process the ldopts and strip out the ccdlflags and ldflags
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On 2024-04-02 Tu 09:34, Andrew Dunstan wrote:
meson.build has this code
ldopts = run_command(perl, '-MExtUtils::Embed', '-e',
'ldopts', check: true).stdout().strip() undesired =
run_command(perl_conf_cmd, 'ccdlflags', check:
true).stdout().split() undesired += run_command(perl_conf_cmd,
'ldflags', check: true).stdout().split() perl_ldopts = []
foreach ldopt : ldopts.split(' ') if ldopt == '' or ldopt in
undesired continue endif perl_ldopts +=
ldopt.strip('"') endforeach message('LDFLAGS recommended
by perl: "@0@"'.format(ldopts)) message('LDFLAGS for embedding
perl: "@0@"'.format(' '.join(perl_ldopts)))This code is seriously broken if perl reports items including spaces,
when a) removing the quotes is quite wrong, and b) splitting on spaces
is also wrong.Here's an example from one of my colleagues:
C:\Program Files\Microsoft Visual Studio\2022\Professional>perl.EXE -MExtUtils::Embed -e ldopts
-nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"C:\edb\languagepack\v4\Perl-5.38\lib\CORE"
-machine:AMD64 -subsystem:console,"5.02" "C:\edb\languagepack\v4\Perl-5.38\lib\CORE\perl538.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\oldnames.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\kernel32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\user32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\gdi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\winspool.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\comdlg32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\advapi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\shell32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\ole32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\oleaut32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\netapi32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\uuid.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\ws2_32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\mpr.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\winmm.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\version.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\odbc32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\odbccp32.lib"
"C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64\comctl32.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\msvcrt.lib"
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\lib\x64\vcruntime.lib"
"C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64\ucrt.lib"And with that we get errors like
cl : Command line warning D9024 : unrecognized source file type 'C:\Program', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Files\Microsoft', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Visual', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'C:\Program', object file assumed
cl : Command line warning D9024 : unrecognized source file type 'Files', object file assumed
cl : Command line warning D9024 : unrecognized source file type '(x86)\Windows', object file assumedIt looks like we need to get smarter about how we process the ldopts and strip out the ccdlflags and ldflags
Here is an attempt to fix all that. It's ugly, but I think it's more
principled.
First, instead of getting the ldopts and then trying to filter out the
ldflags and ccdlflags, it tells perl not to include those in the first
place, by overriding a couple of routines in ExtUtils::Embed. And
second, it's smarter about splitting what's left, so that it doesn't
split on a space that's in a quoted item. The perl that's used to do
that second bit is not pretty, but it has been tested on the system
where the problem arose and apparently cures the problem. (No doubt some
perl guru could improve it.) It also works on my Ubuntu system, so I
don't think we'll be breaking anything (famous last words).
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Attachments:
ldopts2.difftext/x-patch; charset=UTF-8; name=ldopts2.diffDownload
diff --git a/meson.build b/meson.build
index 87437960bc..91e87055d6 100644
--- a/meson.build
+++ b/meson.build
@@ -993,22 +993,11 @@ if not perlopt.disabled()
# Config's ccdlflags and ldflags. (Those are the choices of those who
# built the Perl installation, which are not necessarily appropriate
# for building PostgreSQL.)
- ldopts = run_command(perl, '-MExtUtils::Embed', '-e', 'ldopts', check: true).stdout().strip()
- undesired = run_command(perl_conf_cmd, 'ccdlflags', check: true).stdout().split()
- undesired += run_command(perl_conf_cmd, 'ldflags', check: true).stdout().split()
-
- perl_ldopts = []
- foreach ldopt : ldopts.split(' ')
- if ldopt == '' or ldopt in undesired
- continue
- endif
-
- perl_ldopts += ldopt.strip('"')
- endforeach
-
- message('LDFLAGS recommended by perl: "@0@"'.format(ldopts))
- message('LDFLAGS for embedding perl: "@0@"'.format(' '.join(perl_ldopts)))
-
+ ldopts = run_command(perl, '-MExtUtils::Embed', '-e', '*ExtUtils::Embed::_ldflags = *Extutils::Embed::_ccdlflags = sub { return q[]; }; ldopts', check: true).stdout().strip()
+ message('LDFLAGS for embedding perl: "@0@"'.format(ldopts))
+ # avoid use of " char in this perl mini-program to avoid possibly
+ # confusing the Windows command processor
+ perl_ldopts = run_command(perl, '-MEnglish', '-e', 'my $arg = shift; while ($arg =~ /\S/) { if ($arg =~ /^\s*([^\042 ]+)(?![\042])/) { print qq[$1\n]; $arg = $POSTMATCH;} elsif ($arg =~ /^\s*\042([^\042]+)\042/) { print qq[$1\n]; $arg = $POSTMATCH;} }', '--', ldopts, check: true).stdout().split('\n')
perl_dep_int = declare_dependency(
compile_args: perl_ccflags,
link_args: perl_ldopts,
On 2024-04-05 Fr 08:25, Andrew Dunstan wrote:
Here is an attempt to fix all that. It's ugly, but I think it's more
principled.First, instead of getting the ldopts and then trying to filter out the
ldflags and ccdlflags, it tells perl not to include those in the first
place, by overriding a couple of routines in ExtUtils::Embed. And
second, it's smarter about splitting what's left, so that it doesn't
split on a space that's in a quoted item. The perl that's used to do
that second bit is not pretty, but it has been tested on the system
where the problem arose and apparently cures the problem. (No doubt
some perl guru could improve it.) It also works on my Ubuntu system,
so I don't think we'll be breaking anything (famous last words).
Apparently I spoke too soon. Please ignore the above for now.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On 2024-04-05 Fr 10:12, Andrew Dunstan wrote:
On 2024-04-05 Fr 08:25, Andrew Dunstan wrote:
Here is an attempt to fix all that. It's ugly, but I think it's more
principled.First, instead of getting the ldopts and then trying to filter out
the ldflags and ccdlflags, it tells perl not to include those in the
first place, by overriding a couple of routines in ExtUtils::Embed.
And second, it's smarter about splitting what's left, so that it
doesn't split on a space that's in a quoted item. The perl that's
used to do that second bit is not pretty, but it has been tested on
the system where the problem arose and apparently cures the problem.
(No doubt some perl guru could improve it.) It also works on my
Ubuntu system, so I don't think we'll be breaking anything (famous
last words).Apparently I spoke too soon. Please ignore the above for now.
OK, this has been fixed and checked. The attached is what I propose.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
ldopts3.patchtext/x-patch; charset=UTF-8; name=ldopts3.patchDownload
diff --git a/meson.build b/meson.build
index 87437960bc..be87ef6950 100644
--- a/meson.build
+++ b/meson.build
@@ -993,21 +993,15 @@ if not perlopt.disabled()
# Config's ccdlflags and ldflags. (Those are the choices of those who
# built the Perl installation, which are not necessarily appropriate
# for building PostgreSQL.)
- ldopts = run_command(perl, '-MExtUtils::Embed', '-e', 'ldopts', check: true).stdout().strip()
- undesired = run_command(perl_conf_cmd, 'ccdlflags', check: true).stdout().split()
- undesired += run_command(perl_conf_cmd, 'ldflags', check: true).stdout().split()
-
- perl_ldopts = []
- foreach ldopt : ldopts.split(' ')
- if ldopt == '' or ldopt in undesired
- continue
- endif
-
- perl_ldopts += ldopt.strip('"')
- endforeach
-
+ ldopts = run_command(perl, '-MExtUtils::Embed', '-e', '*ExtUtils::Embed::_ldflags = *Extutils::Embed::_ccdlflags = sub { return q[]; }; ldopts',
+ check: true).stdout().strip()
+ # avoid use of " char in this perl mini-program to avoid possibly
+ # confusing the Windows command processor
+ perl_ldopts = run_command(perl, '-MEnglish', '-e',
+ 'my $arg = shift; while ($arg =~ /\S/) { if ($arg =~ /^\s*([^\042 ]+)(?![\042])/) { print qq[$1\n]; $arg = $POSTMATCH;} elsif ($arg =~ /^\s*\042([^\042]+)\042/) { print qq[$1\n]; $arg = $POSTMATCH;} }',
+ '--', ldopts, check: true).stdout().strip().split('\n')
message('LDFLAGS recommended by perl: "@0@"'.format(ldopts))
- message('LDFLAGS for embedding perl: "@0@"'.format(' '.join(perl_ldopts)))
+ message('LDFLAGS for embedding perl:\n"@0@"'.format('\n'.join(perl_ldopts)))
perl_dep_int = declare_dependency(
compile_args: perl_ccflags,
Hi,
On 2024-04-05 16:12:12 -0400, Andrew Dunstan wrote:
OK, this has been fixed and checked. The attached is what I propose.
The perl command is pretty hard to read. What about using python's shlex
module instead? Rough draft attached. Still not very pretty, but seems easier
to read?
It'd be even better if we could just get perl to print out the flags in an
easier to parse way, but I couldn't immediately see a way.
Greetings,
Andres Freund
Attachments:
perl-detection-windows-wip.difftext/x-diff; charset=us-asciiDownload
diff --git i/meson.build w/meson.build
index d6401fb8e30..191a051defb 100644
--- i/meson.build
+++ w/meson.build
@@ -997,13 +997,20 @@ if not perlopt.disabled()
undesired = run_command(perl_conf_cmd, 'ccdlflags', check: true).stdout().split()
undesired += run_command(perl_conf_cmd, 'ldflags', check: true).stdout().split()
+ ldopts_split = run_command(python, '-c', '''
+import shlex
+import sys
+
+print('\n'.join(shlex.split(sys.argv[1])), end='')
+''',
+ ldopts, check: true).stdout().split('\n')
+
perl_ldopts = []
- foreach ldopt : ldopts.split(' ')
- if ldopt == '' or ldopt in undesired
+ foreach ldopt : ldopts_split
+ if ldopt in undesired
continue
endif
-
- perl_ldopts += ldopt.strip('"')
+ perl_ldopts += ldopt
endforeach
message('LDFLAGS recommended by perl: "@0@"'.format(ldopts))
On 2024-05-28 Tu 6:13 PM, Andres Freund wrote:
Hi,
On 2024-04-05 16:12:12 -0400, Andrew Dunstan wrote:
OK, this has been fixed and checked. The attached is what I propose.
The perl command is pretty hard to read. What about using python's shlex
module instead? Rough draft attached. Still not very pretty, but seems easier
to read?It'd be even better if we could just get perl to print out the flags in an
easier to parse way, but I couldn't immediately see a way.
Thanks for looking.
The attached should be easier to read. The perl package similar to shlex
is Text::ParseWords, which is already a requirement.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
ldopts4.patchtext/x-patch; charset=UTF-8; name=ldopts4.patchDownload
diff --git a/meson.build b/meson.build
index 5387bb6d5f..e244cbac92 100644
--- a/meson.build
+++ b/meson.build
@@ -993,20 +993,17 @@ if not perlopt.disabled()
# Config's ccdlflags and ldflags. (Those are the choices of those who
# built the Perl installation, which are not necessarily appropriate
# for building PostgreSQL.)
- ldopts = run_command(perl, '-MExtUtils::Embed', '-e', 'ldopts', check: true).stdout().strip()
- undesired = run_command(perl_conf_cmd, 'ccdlflags', check: true).stdout().split()
- undesired += run_command(perl_conf_cmd, 'ldflags', check: true).stdout().split()
+ perl_ldopts = run_command(perl, '-e', '''
+use ExtUtils::Embed;
+use Text::ParseWords;
+# tell perl to suppress including these in ldopts
+*ExtUtils::Embed::_ldflags =*ExtUtils::Embed::_ccdlflags = sub { return ""; };
+# adding an argument to ldopts makes it return a value instead of printing
+# print one of these per line so splitting will preserve spaces in file names.
+print "$_\n" foreach shellwords(ldopts(undef));
+''',
+ check: true).stdout().strip().split('\n')
- perl_ldopts = []
- foreach ldopt : ldopts.split(' ')
- if ldopt == '' or ldopt in undesired
- continue
- endif
-
- perl_ldopts += ldopt.strip('"')
- endforeach
-
- message('LDFLAGS recommended by perl: "@0@"'.format(ldopts))
message('LDFLAGS for embedding perl: "@0@"'.format(' '.join(perl_ldopts)))
perl_dep_int = declare_dependency(
On 2024-07-20 Sa 9:41 AM, Andrew Dunstan wrote:
On 2024-05-28 Tu 6:13 PM, Andres Freund wrote:
Hi,
On 2024-04-05 16:12:12 -0400, Andrew Dunstan wrote:
OK, this has been fixed and checked. The attached is what I propose.
The perl command is pretty hard to read. What about using python's shlex
module instead? Rough draft attached. Still not very pretty, but
seems easier
to read?It'd be even better if we could just get perl to print out the flags
in an
easier to parse way, but I couldn't immediately see a way.Thanks for looking.
The attached should be easier to read. The perl package similar to
shlex is Text::ParseWords, which is already a requirement.
It turns out that shellwords eats backslashes, so we would need
something like this version, which I have tested on Windows. I will
probably commit this in the next few days unless there's an objection.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
ldopts5.patchtext/x-patch; charset=UTF-8; name=ldopts5.patchDownload
diff --git a/meson.build b/meson.build
index 7de0371226..dfcfcfc15e 100644
--- a/meson.build
+++ b/meson.build
@@ -1065,20 +1065,19 @@ if not perlopt.disabled()
# Config's ccdlflags and ldflags. (Those are the choices of those who
# built the Perl installation, which are not necessarily appropriate
# for building PostgreSQL.)
- ldopts = run_command(perl, '-MExtUtils::Embed', '-e', 'ldopts', check: true).stdout().strip()
- undesired = run_command(perl_conf_cmd, 'ccdlflags', check: true).stdout().split()
- undesired += run_command(perl_conf_cmd, 'ldflags', check: true).stdout().split()
+ perl_ldopts = run_command(perl, '-e', '''
+use ExtUtils::Embed;
+use Text::ParseWords;
+# tell perl to suppress including these in ldopts
+*ExtUtils::Embed::_ldflags =*ExtUtils::Embed::_ccdlflags = sub { return ""; };
+# adding an argument to ldopts makes it return a value instead of printing
+# print one of these per line so splitting will preserve spaces in file names.
+# shellwords eats backslashes, so we need to escape them.
+(my $opts = ldopts(undef)) =~ s!\\!\\\\!g;
+print "$_\n" foreach shellwords($opts);
+''',
+ check: true).stdout().strip().split('\n')
- perl_ldopts = []
- foreach ldopt : ldopts.split(' ')
- if ldopt == '' or ldopt in undesired
- continue
- endif
-
- perl_ldopts += ldopt.strip('"')
- endforeach
-
- message('LDFLAGS recommended by perl: "@0@"'.format(ldopts))
message('LDFLAGS for embedding perl: "@0@"'.format(' '.join(perl_ldopts)))
perl_dep_int = declare_dependency(
On 2024-07-30 Tu 3:47 PM, Andrew Dunstan wrote:
On 2024-07-20 Sa 9:41 AM, Andrew Dunstan wrote:
On 2024-05-28 Tu 6:13 PM, Andres Freund wrote:
Hi,
On 2024-04-05 16:12:12 -0400, Andrew Dunstan wrote:
OK, this has been fixed and checked. The attached is what I propose.
The perl command is pretty hard to read. What about using python's
shlex
module instead? Rough draft attached. Still not very pretty, but
seems easier
to read?It'd be even better if we could just get perl to print out the flags
in an
easier to parse way, but I couldn't immediately see a way.Thanks for looking.
The attached should be easier to read. The perl package similar to
shlex is Text::ParseWords, which is already a requirement.It turns out that shellwords eats backslashes, so we would need
something like this version, which I have tested on Windows. I will
probably commit this in the next few days unless there's an objection.
pushed
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com