Postgres Windows build system doesn't work with python installed in Program Files

On Thu, Apr 30, 2020 at 03:06:08PM +0300, Victor Wagner wrote:

Fix is very simple, see attach.

Patch is made against REL_12_STABLE, but probably applicable to other
versions as well.

Indeed, thanks.

my $pythonprog = "import sys;print(sys.prefix);"
. "print(str(sys.version_info[0])+str(sys.version_info[1]))";
my $prefixcmd =
-		  $solution->{options}->{python} . "\\python -c \"$pythonprog\"";
+		'"' . $solution->{options}->{python} . "\\python\" -c \"$pythonprog\"";
my $pyout = `$prefixcmd`;
die "Could not query for python version!\n" if $?;
my ($pyprefix, $pyver) = split(/\r?\n/, $pyout);

This reminds me of ad7595b. Wouldn't it be better to use qq() here?
--
Michael

В Fri, 1 May 2020 17:52:15 +0900
Michael Paquier <michael@paquier.xyz> пишет:

On Thu, Apr 30, 2020 at 03:06:08PM +0300, Victor Wagner wrote:

Fix is very simple, see attach.

Patch is made against REL_12_STABLE, but probably applicable to
other versions as well.

Indeed, thanks.

my $pythonprog = "import sys;print(sys.prefix);"
.
"print(str(sys.version_info[0])+str(sys.version_info[1]))"; my
$prefixcmd =
-		  $solution->{options}->{python} . "\\python -c
\"$pythonprog\"";
+		'"' . $solution->{options}->{python} . "\\python\"
-c \"$pythonprog\""; my $pyout = `$prefixcmd`;
die "Could not query for python version!\n" if $?;
my ($pyprefix, $pyver) = split(/\r?\n/, $pyout);  

This reminds me of ad7595b. Wouldn't it be better to use qq() here?

Maybe. But probably original author of this code was afraid of using
too long chain of ->{} in the string substitution.

So, I left this style n place.

Nonetheless, using qq wouldn't save us from doubling backslashes.

--

Em qui., 30 de abr. de 2020 às 09:06, Victor Wagner <vitus@wagner.pp.ru>
escreveu:

Collegues,

Accidently I've come over minor bug in the Mkvcbuild.pm.
It happens, that it doesn't tolerate spaces in the $config->{python}
path, because it want to call python in order to find out version,
prefix and so on, and doesn't properly quote command.

Fix is very simple, see attach.

Patch is made against REL_12_STABLE, but probably applicable to other
versions as well.

I don't know if it applies to the same case, but from the moment I
installed python on the development machine, the Postgres build stopped
working correctly.
Although perl, flex and bison are available in the path, the build does not
generate files that depend on flex and bison.

Running bison on src/backend/bootstrap/bootparse.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/bootstrap/bootscanner.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running bison on src/backend/parser/gram.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
etc

Warning from build.pl
Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

regards,
Ranier Vilela

On Sun, May 03, 2020 at 04:23:24PM -0300, Ranier Vilela wrote:

I don't know if it applies to the same case, but from the moment I
installed python on the development machine, the Postgres build stopped
working correctly.
Although perl, flex and bison are available in the path, the build does not
generate files that depend on flex and bison.

Are you following the instructions of the documentation? Here is a
link to them:
https://www.postgresql.org/docs/devel/install-windows-full.html

My guess is that you would be just missing a PATH configuration or
such because python enforced a new setting?

Warning from build.pl
Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm. We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose. And note that @ARGV would be normally
defined, so your warning looks fishy to me.
--
Michael

On Mon, May 4, 2020 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:

Warning from build.pl
Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm. We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose. And note that @ARGV would be normally
defined, so your warning looks fishy to me.

I think these are two different issues, python PATH and build.pl warnings.
For the later, you can check woodloose logs and see the warning after
commit 8f00d84afc.

Regards,

Juan José Santamaría Flecha

Em seg., 4 de mai. de 2020 às 02:58, Michael Paquier <michael@paquier.xyz>
escreveu:

On Sun, May 03, 2020 at 04:23:24PM -0300, Ranier Vilela wrote:

I don't know if it applies to the same case, but from the moment I
installed python on the development machine, the Postgres build stopped
working correctly.
Although perl, flex and bison are available in the path, the build does

not

generate files that depend on flex and bison.

Are you following the instructions of the documentation? Here is a
link to them:
https://www.postgresql.org/docs/devel/install-windows-full.html

My environment was ok and worked 100%, compiling with msvc 2019 (64 bits).

My guess is that you would be just missing a PATH configuration or
such because python enforced a new setting?

Perl and flex and bison, are in the path, no doubt.

Warning from build.pl
Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm. We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose. And note that @ARGV would be normally
defined, so your warning looks fishy to me.

I'll redo from the beginning.
1. Make empty directory postgres
2. Clone postgres
3. Call msvc 2019 (64 bits) env batch
4. Setup path to perl, bison and flex
set path=%path%;c:\perl;\bin;c:\bin

5. C:\dll>perl -V
Summary of my perl5 (revision 5 version 30 subversion 1) configuration:

Platform:
osname=MSWin32
osvers=10.0.18363.476
archname=MSWin32-x64-multi-thread
uname='Win32 strawberry-perl 5.30.1.1 #1 Fri Nov 22 02:24:29 2019 x64'

6. C:\dll>bison -V
bison (GNU Bison) 2.7
Written by Robert Corbett and Richard Stallman.

7. C:\dll>flex -V
flex 2.6.4

8. cd\dll\postgres\src\tools\msvc
9. build

results:
...
PrepareForBuild:
Creating directory ".\Release\postgres\".
Creating directory ".\Release\postgres\postgres.tlog\".
InitializeBuildStatus:
Creating ".\Release\postgres\postgres.tlog\unsuccessfulbuild" because
"AlwaysCreate" was specified.
CustomBuild:
Running bison on src/backend/bootstrap/bootparse.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/bootstrap/bootscanner.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running bison on src/backend/parser/gram.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/parser/scan.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running bison on src/backend/replication/repl_gram.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/replication/repl_scanner.l
Running bison on src/backend/replication/syncrep_gram.y
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/replication/syncrep_scanner.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running bison on src/backend/utils/adt/jsonpath_gram.y
Running flex on src/backend/utils/adt/jsonpath_scan.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
Running flex on src/backend/utils/misc/guc-file.l
'perl' nao é reconhecido como um comando interno
ou externo, um programa operável ou um arquivo em lotes.
C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5):
error MSB6006: "cmd.exe" exited with code 9009.
[C:\dll\postgres\postgres.vcxproj]
Done Building Project "C:\dll\postgres\postgres.vcxproj" (default targets)
-- FAILED.

Done Building Project "C:\dll\postgres\pgsql.sln" (default targets) --
FAILED.

...

Build FAILED.

"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\cyrillic_and_mic.vcxproj" (default target) (2) ->
"C:\dll\postgres\postgres.vcxproj" (default target) (3) ->
(CustomBuild target) ->
C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5):
error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\postgres.vcxproj]

"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\initdb.vcxproj" (default target) (32) ->
"C:\dll\postgres\libpgfeutils.vcxproj" (default target) (33) ->
C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5):
error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\libpgfeutils.vcxproj]

"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\ecpg.vcxproj" (default target) (124) ->
C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5):
error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\ecpg.vcxproj]

"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\isolationtester.vcxproj" (default target) (129) ->
C:\Program Files (x86)\Microsoft Visual
Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5):
error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\isolationtester.vcxproj]

0 Warning(s)
4 Error(s)

No warnings, this time.

best regards,
Ranier Vilela

On Mon, May 04, 2020 at 09:45:54AM +0200, Juan José Santamaría Flecha wrote:

I think these are two different issues, python PATH and build.pl warnings.
For the later, you can check woodloose logs and see the warning after
commit 8f00d84afc.

Oh, indeed. I somewhat managed to miss these in the logs of the
buildfarm. What if we refactored the code of build.pl so as we'd
check first if $ARGV[0] is defined or not? If not defined, then we
need to have a release-quality build for all the components. How does
that sound? Something not documented is that using "release" as first
argument enforces also a release-quality build for all the components,
so we had better not break that part.
--
Michael

On Mon, May 4, 2020 at 2:18 PM Michael Paquier <michael@paquier.xyz> wrote:

On Mon, May 04, 2020 at 09:45:54AM +0200, Juan José Santamaría Flecha
wrote:

I think these are two different issues, python PATH and build.pl

warnings.

For the later, you can check woodloose logs and see the warning after
commit 8f00d84afc.

Oh, indeed. I somewhat managed to miss these in the logs of the
buildfarm. What if we refactored the code of build.pl so as we'd
check first if $ARGV[0] is defined or not? If not defined, then we
need to have a release-quality build for all the components. How does
that sound? Something not documented is that using "release" as first
argument enforces also a release-quality build for all the components,
so we had better not break that part.

+1, seems like the way to go to me.

Regards,

Juan José Santamaría Flecha

On Fri, May 01, 2020 at 12:48:17PM +0300, Victor Wagner wrote:

Maybe. But probably original author of this code was afraid of using
too long chain of ->{} in the string substitution.

So, I left this style n place.

Nonetheless, using qq wouldn't save us from doubling backslashes.

Looking at this part in more details, I find the attached much more
readable. I have been able to test it on my own Windows environment
and the problem gets fixed (I have reproduced the original problem as
well).
--
Michael

В Tue, 5 May 2020 15:45:48 +0900
Michael Paquier <michael@paquier.xyz> пишет:

On Fri, May 01, 2020 at 12:48:17PM +0300, Victor Wagner wrote:

Maybe. But probably original author of this code was afraid of using
too long chain of ->{} in the string substitution.

So, I left this style n place.

Nonetheless, using qq wouldn't save us from doubling backslashes.

Looking at this part in more details, I find the attached much more
readable. I have been able to test it on my own Windows environment

I agree, it is better.

and the problem gets fixed (I have reproduced the original problem as
well).

--

On Mon, May 04, 2020 at 03:42:20PM +0200, Juan José Santamaría Flecha wrote:

+1, seems like the way to go to me.

Attached is a patch for that and I have gone with a simple solution,
with some bonus comments about the way things happen. Here are the
patterns I tested for build.pl and the commands it generates, making
sure that we have the same commands with HEAD and the patch:
1) perl build.pl
msbuild pgsql.sln /verbosity:normal /p:Configuration=Release
2) perl build.pl debug
msbuild pgsql.sln /verbosity:normal /p:Configuration=Debug
3) perl build.pl release
msbuild pgsql.sln /verbosity:normal /p:Configuration=Release
4) perl build.pl foo
msbuild foo.vcxproj /verbosity:normal /p:Configuration=Release
5) perl build.pl debug foo
msbuild foo.vcxproj /verbosity:normal /p:Configuration=Debug
6) perl build.pl release foo
msbuild foo.vcxproj /verbosity:normal /p:Configuration=Release

The two warnings show up only in the first case, of course.
--
Michael

On Tue, May 5, 2020 at 4:06 PM Michael Paquier <michael@paquier.xyz> wrote:

On Mon, May 04, 2020 at 03:42:20PM +0200, Juan José Santamaría Flecha
wrote:

+1, seems like the way to go to me.

Attached is a patch for that and I have gone with a simple solution,
with some bonus comments about the way things happen.

This solves the issue.

Please forgive me if I am being too nitpicky, but I find the comments a
little too verbose, a usage format might be more visual and easier to
explain:

Usage: build [[CONFIGURATION] COMPONENT]

The options are case-insensitive.
CONFIGURATION sets the configuration to build, "debug" or "release" (by
default).
COMPONENT defines a component to build. An empty option means all
components.

Regards,

Juan José Santamaría Flecha

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..
--
Michael

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz>
escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It
works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the
path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of msvc
2019.
I'm still investigating.

regards,
Ranier Vilela

Em qua., 6 de mai. de 2020 às 10:21, Ranier Vilela <ranier.vf@gmail.com>
escreveu:

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz>
escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It
works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the
path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of
msvc 2019.
I'm still investigating.

In fact perl, it is found, otherwise, neither build.pl would be working.
But within the perl environment, when the system call is made, in this
case, neither perl, bison, nor flex is found.

regards,
Ranier Vilela

Em qua., 6 de mai. de 2020 às 10:25, Ranier Vilela <ranier.vf@gmail.com>
escreveu:

Em qua., 6 de mai. de 2020 às 10:21, Ranier Vilela <ranier.vf@gmail.com>
escreveu:

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz>
escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It
works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the
path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of
msvc 2019.
I'm still investigating.

In fact perl, it is found, otherwise, neither build.pl would be working.
But within the perl environment, when the system call is made, in this
case, neither perl, bison, nor flex is found.

I'm using it like this, for now.

File pgbison.pl:
system("c:\\bin\\bison $headerflag $input -o $output");
File pgflex.pl:
system("c:\\bin\\flex $flexflags -o$output $input");
system("c:\\perl\\bin\\perl src\\tools\\fix-old-flex-code.pl
$output");

File Solution.pm:
system(
system('perl generate-lwlocknames.pl lwlocknames.txt');
system(
system(
system(
system(
system(
system(
system("perl create_help.pl ../../../doc/src/sgml/ref
sql_help");
system(
system(
system(
system(
system(
system('perl parse.pl < ../../../backend/parser/gram.y >
preproc.y');
system(

C:\dll\postgres\src\tools\msvc>\bin\grep bison *pm
File MSBuildProject.pm:
<Message
Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">Running
bison on $grammarFile</Message>
<Command
Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">c:\\perl\\bin\\perl
"src\\tools\\msvc\\pgbison.pl" "$grammarFile"</Command>
<Message
Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">Running
bison on $grammarFile</Message>
<Command
Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">c:\\perl\\bin\\perl
"src\\tools\\msvc\\pgbison.pl" "$grammarFile"</Command>

C:\dll\postgres\src\tools\msvc>\bin\grep flex *pm
File MSBuildProject.pm:
<Message
Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">Running
flex on $grammarFile</Message>
<Command
Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">c:\\perl\\bin\\perl
"src\\tools\\msvc\\pgflex.pl" "$grammarFile"</Command>
<Message
Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">Running
flex on $grammarFile</Message>
<Command
Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">c:\\perl\\bin\\perl
"src\\tools\\msvc\\pgflex.pl" "$grammarFile"</Command>

regards,
Ranier Vilela

В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
<michael@paquier.xyz> escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with
path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't use it to invoke secondary
perl scripts.

--

On 5/6/20 1:14 PM, Victor Wagner wrote:

В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
<michael@paquier.xyz> escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with
path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't use it to invoke secondary
perl scripts.

We assume perl, flex and bison are in the PATH. That doesn't seem
unreasonable, it's worked well for quite a long time.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Em qua., 6 de mai. de 2020 às 14:14, Victor Wagner <vitus@wagner.pp.ru>
escreveu:

В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
<michael@paquier.xyz> escreveu:

On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:

I agree, it is better.

Thanks, applied and back-patched down to 9.5. Now for the second
problem of this thread..

Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with
path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't use it to invoke secondary
perl scripts.

I still don't think it's necessary, it was working well.
My main suspicions are:
1. Path with spaces;
2. Incompatibility with < symbol, some suggest use &quot;

<Exec Command="&quot;

3. Msbuid.exe It has been updated (version 16.5.0)
4. Perl scripts increased the level of security.
5. My user do not have administrator rights.

regards,
Ranier Vilela