Building Postgres with lz4 on Visual Studio

Hi,

Michael, Dilip, I think you worked most in this area? Based on
9ca40dcd4d0cad43d95a9a253fafaa9a9ba7de24

Robert, added you too, because zstd seems to have the same issue (based on the
tail of the quoted email below).

On 2022-04-13 17:21:41 +0300, Melih Mutlu wrote:

I tried to build Postgres from source using Visual Studio 19. It worked all
good.
Then I wanted to build it with some dependencies, started with the ones
listed here [1]. But I'm having some issues with lz4.

First I downloaded the latest release of lz4 from this link [2].
Modified the src\tools\msvc\config.pl file as follows:

use strict;
use warnings;

our $config;

$config->{"tap_tests"} = 1;
$config->{"asserts"} = 1;

$config->{"lz4"} = "c:/lz4/";

$config->{"openssl"} = "c:/openssl/1.1/";
$config->{"perl"} = "c:/strawberry/$ENV{DEFAULT_PERL_VERSION}/perl/";
$config->{"python"} = "c:/python/";

1;

based on /src/tools/ci/windows_build_config.pl

Then ran the following commands:

vcvarsall x64
perl src/tools/msvc/mkvcbuild.pl
msbuild -m -verbosity:minimal /p:IncludePath="C:\lz4" pgsql.sln

I don't think the /p:IncludePath should be needed, the build scripts should
add that.

msbuild command fails with the following error:
"LINK : fatal error LNK1181: cannot open input file
'c:\lz4\\lib\liblz4.lib' [C:\postgres\libpgtypes.vcxproj]"

What I realized is that c:\lz4\lib\liblz4.lib actually does not exist.
The latest versions of lz4, downloaded from [2], do not contain \liblz4.lib
anymore, as opposed to what's written here [3]. Also there isn't a lib
folder too.

After those changes on lz4 side, AFAIU there seems like this line adds
library from wrong path in Solution.pm file [4].

$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');

Even if I spent some time on this problem and tried to fix some places, I'm
not able to run a successful build yet.
This is also the case for zstd too. Enabling zstd gives the same error.

Has anyone had this issue before? Is this something that anyone is aware of
and somehow made it work?
I would appreciate any comment/suggestion etc.

Greetings,

Andres Freund

On Wed, Apr 13, 2022 at 05:21:41PM +0300, Melih Mutlu wrote:

I tried to build Postgres from source using Visual Studio 19. It worked all
good.
Then I wanted to build it with some dependencies, started with the ones
listed here [1]. But I'm having some issues with lz4.

First I downloaded the latest release of lz4 from this link [2].
Modified the src\tools\msvc\config.pl file as follows:

Yeah, that's actually quite an issue because there is no official
release for liblz4.lib, so one has to compile the code by himself to
be able to get his/her hands on liblz4.lib. zstd is similarly
consistent with its release contents.
--
Michael

On Wed, Apr 13, 2022 at 10:22 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:

What I realized is that c:\lz4\lib\liblz4.lib actually does not exist.
The latest versions of lz4, downloaded from [2], do not contain \liblz4.lib anymore, as opposed to what's written here [3]. Also there isn't a lib folder too.

After those changes on lz4 side, AFAIU there seems like this line adds library from wrong path in Solution.pm file [4].

$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');

Even if I spent some time on this problem and tried to fix some places, I'm not able to run a successful build yet.
This is also the case for zstd too. Enabling zstd gives the same error.

Has anyone had this issue before? Is this something that anyone is aware of and somehow made it work?
I would appreciate any comment/suggestion etc.

In Solution.pm we have this:

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
if ($self->{options}->{zstd})
{
$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
}

I think what you're saying is that the relative pathnames here may not
be correct, depending on which version of lz4/zstd you're using. The
solution is probably to use perl's -e to test which files actually
exists e.g.

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
if (-e $proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib')
{
$proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib');
}
else
{
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}

The trick, at least as it seems to me, is figuring out exactly what
the right set of conditions is, based on what kinds of different
builds exist out there and where they put stuff.

--
Robert Haas
EDB: http://www.enterprisedb.com

On 2022-04-26 Tu 16:26, Robert Haas wrote:

On Wed, Apr 13, 2022 at 10:22 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:

What I realized is that c:\lz4\lib\liblz4.lib actually does not exist.
The latest versions of lz4, downloaded from [2], do not contain \liblz4.lib anymore, as opposed to what's written here [3]. Also there isn't a lib folder too.

After those changes on lz4 side, AFAIU there seems like this line adds library from wrong path in Solution.pm file [4].

$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');

Even if I spent some time on this problem and tried to fix some places, I'm not able to run a successful build yet.
This is also the case for zstd too. Enabling zstd gives the same error.

Has anyone had this issue before? Is this something that anyone is aware of and somehow made it work?
I would appreciate any comment/suggestion etc.

In Solution.pm we have this:

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
if ($self->{options}->{zstd})
{
$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
}

I think what you're saying is that the relative pathnames here may not
be correct, depending on which version of lz4/zstd you're using. The
solution is probably to use perl's -e to test which files actually
exists e.g.

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
if (-e $proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib')
{
$proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib');
}
else
{
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}

The trick, at least as it seems to me, is figuring out exactly what
the right set of conditions is, based on what kinds of different
builds exist out there and where they put stuff.

I agree that we should use perl's -e to test that the files actually
exists. But I don't think we should try to adjust to everything the zstd
and lz4 people put in their release files. They are just horribly
inconsistent.

What I did was to install the packages using vcpkg[1]https://github.com/microsoft/vcpkg which is a
standard framework created by Microsoft for installing package
libraries. It does install the .lib files in a sane place
(installdir/lib), but it doesn't use the lib suffix. Also it names the
lib file for zlib differently.

I got around those things by renaming the lib files, but that's a bit
ugly. So I came up with this (untested) patch.

cheers

andrew

[1]: https://github.com/microsoft/vcpkg

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2022-04-29 Fr 08:50, Andrew Dunstan wrote:

On 2022-04-26 Tu 16:26, Robert Haas wrote:

On Wed, Apr 13, 2022 at 10:22 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:

What I realized is that c:\lz4\lib\liblz4.lib actually does not exist.
The latest versions of lz4, downloaded from [2], do not contain \liblz4.lib anymore, as opposed to what's written here [3]. Also there isn't a lib folder too.

After those changes on lz4 side, AFAIU there seems like this line adds library from wrong path in Solution.pm file [4].

$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');

Even if I spent some time on this problem and tried to fix some places, I'm not able to run a successful build yet.
This is also the case for zstd too. Enabling zstd gives the same error.

Has anyone had this issue before? Is this something that anyone is aware of and somehow made it work?
I would appreciate any comment/suggestion etc.

In Solution.pm we have this:

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
if ($self->{options}->{zstd})
{
$proj->AddIncludeDir($self->{options}->{zstd} . '\include');
$proj->AddLibrary($self->{options}->{zstd} . '\lib\libzstd.lib');
}

I think what you're saying is that the relative pathnames here may not
be correct, depending on which version of lz4/zstd you're using. The
solution is probably to use perl's -e to test which files actually
exists e.g.

if ($self->{options}->{lz4})
{
$proj->AddIncludeDir($self->{options}->{lz4} . '\include');
if (-e $proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib')
{
$proj->AddLibrary($self->{options}->{lz4} .
'\someplace\somelz4.lib');
}
else
{
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}
$proj->AddLibrary($self->{options}->{lz4} . '\lib\liblz4.lib');
}

The trick, at least as it seems to me, is figuring out exactly what
the right set of conditions is, based on what kinds of different
builds exist out there and where they put stuff.

I agree that we should use perl's -e to test that the files actually
exists. But I don't think we should try to adjust to everything the zstd
and lz4 people put in their release files. They are just horribly
inconsistent.

What I did was to install the packages using vcpkg[1] which is a
standard framework created by Microsoft for installing package
libraries. It does install the .lib files in a sane place
(installdir/lib), but it doesn't use the lib suffix. Also it names the
lib file for zlib differently.

I got around those things by renaming the lib files, but that's a bit
ugly. So I came up with this (untested) patch.

er this patch

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 2022-04-29 Fr 08:50, Andrew Dunstan wrote:

What I did was to install the packages using vcpkg[1] which is a
standard framework created by Microsoft for installing package
libraries. It does install the .lib files in a sane place
(installdir/lib), but it doesn't use the lib suffix. Also it names the
lib file for zlib differently.

Of course I meant "doesn't use the 'lib' prefix". So the files are named
just zstd.lib and lz4.lib.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On 2022-04-29 08:50:56 -0400, Andrew Dunstan wrote:

I agree that we should use perl's -e to test that the files actually
exists. But I don't think we should try to adjust to everything the zstd
and lz4 people put in their release files. They are just horribly
inconsistent.

Right now it's the source of packages we document for windows... It doesn't
seem that crazy to accept a few different paths with a glob or such?

What I did was to install the packages using vcpkg[1] which is a
standard framework created by Microsoft for installing package
libraries. It does install the .lib files in a sane place
(installdir/lib), but it doesn't use the lib suffix. Also it names the
lib file for zlib differently.

That doesn't seem much better :(

Greetings,

Andres Freund