compression libraries and CF bot

On Wed, Jan 20, 2021 at 9:56 AM Justin Pryzby <pryzby@telsasoft.com> wrote:

Do you know if the old travis build environment had liblz4 installed ?

It sounds like it.

I'm asking regarding Dilip's patch, which was getting to "check world" 2 weeks
ago but now failing to even compile, not apparently due to any change in the
patch. Also, are the historic logs available somewhere ?
http://cfbot.cputube.org/dilip-kumar.html

I can find some of them but not that one, because Travis's "branches"
page truncates well before our ~250 active branches, and that one
isn't in there.

https://travis-ci.org/github/postgresql-cfbot/postgresql/branches

Also, what's the process for having new libraries installed in the CI
environment ?

I have added lz4 to the FreeBSD and Ubuntu build tasks, so we'll see
if that helps at the next periodic build or when a new patch is
posted. It's failing on Windows because there is no HAVE_LIBLZ4 in
Solution.pm, and I don't know how to install that on a Mac. Is this
patch supposed to be adding a new required dependency, or a new
optional dependency?

In general, you could ask for changes here, or send me a pull request for eg:

https://github.com/macdice/cfbot/blob/master/cirrus/.cirrus.yml

If we eventually think the CI control file is good enough, and can get
past the various political discussions required to put CI
vendor-specific material in our tree, it'd be just a regular patch
proposal and could even be tweaked as part of a feature submission.

There's 3 compression patches going around, so I think eventually we'll ask to
get libzstd-devel (for libpq and pg_dump) and liblz4-devel (for toast and
libpq). Maybe all compression methods would be supported in each place - I
hope the patches will share common code.

+1, nice to see modern compression coming to PostgreSQL.

Thomas Munro <thomas.munro@gmail.com> writes:

On Wed, Jan 20, 2021 at 9:56 AM Justin Pryzby <pryzby@telsasoft.com> wrote:

Also, what's the process for having new libraries installed in the CI
environment ?

I have added lz4 to the FreeBSD and Ubuntu build tasks, so we'll see
if that helps at the next periodic build or when a new patch is
posted. It's failing on Windows because there is no HAVE_LIBLZ4 in
Solution.pm, and I don't know how to install that on a Mac. Is this
patch supposed to be adding a new required dependency, or a new
optional dependency?

It had better be optional.

regards, tom lane

On Wed, Jan 20, 2021 at 10:29:05AM +1300, Thomas Munro wrote:

I have added lz4 to the FreeBSD and Ubuntu build tasks, so we'll see
if that helps at the next periodic build or when a new patch is
posted. It's failing on Windows because there is no HAVE_LIBLZ4 in
Solution.pm, and I don't know how to install that on a Mac.

For mac, does it just need this ?
brew install lz4

Dilip's TOAST patch is passing on linux and bsd --with-lz4, so I think it's
desirable to install on mac now.

libzstd would be desirable for linux/bsd/mac for Konstantin's libpq patch, and
my pg_dump patch.
https://formulae.brew.sh/formula/zstd

--
Justin

On Sun, Feb 21, 2021 at 4:30 AM Justin Pryzby <pryzby@telsasoft.com> wrote:

Dilip's TOAST patch is passing on linux and bsd --with-lz4, so I think it's
desirable to install on mac now.

Justin figured this out, so now this patch is using lz4 and passing on
Linux, FreeBSD and macOS.

libzstd would be desirable for linux/bsd/mac for Konstantin's libpq patch, and
my pg_dump patch.
https://formulae.brew.sh/formula/zstd

I think I have added that too; let's see what happens :-)

Not done for Windows. It's probably easy with choco, though I assume
none of these patches have the right bits and pieces in our Perl build
scripting for Windows...