Updating IPC::Run in CI?

Hi,

On 2025-09-18 10:08:49 -0700, Jacob Champion wrote:

Andres pointed out that one of my new OAuth tests has flaked on NetBSD
[1]. The final line of debugging output is missing from the stderr
coming from psql.

I haven't ruled out a bug in my implementation, but I also remembered
that there was some discussion on the list about stderr being lost by
IPC::Run in rare circumstances [2], and that particular bug has since
been fixed [3]. Is it possible to (safely) switch the CI images to use
the most recent IPC::Run, so we can make use of Noah's fixes? NetBSD
is currently running 20231003.0.

A lot of things are safely possible :).

The stuff that's installed in the images is controlled by a bunch of scripts
in my pg-vm-images repository. For netbsd the relevant install is in

https://github.com/anarazel/pg-vm-images/blob/main/scripts/bsd/netbsd-prep-postgres.sh

As you can see we currently don't install packages from outside netbsd's
package repository. We could change that and install IPC::Run via cpan, but
imo it's obviously nicer - and more representative of real world setups - if
we just rely on OS/distro packages.

I don't know what would be required to get netbsd to upate their IPC::Run
package.
https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/p5-IPC-Run/index.html

If it were merged into netbsd, it'd be used for the CI images within a few
days, they are regenerated every few days [1]the cron-like expression is "0 30 0 ? * SUN,TUE,THU *", probably because I didn't want to have to look at failures on Friday, Saturday (or if a change the the image
sources is merged).

Greetings,

Andres Freund

[1]: the cron-like expression is "0 30 0 ? * SUN,TUE,THU *", probably because I didn't want to have to look at failures on Friday, Saturday
didn't want to have to look at failures on Friday, Saturday

On Thu, Sep 18, 2025 at 11:18 AM Andres Freund <andres@anarazel.de> wrote:

As you can see we currently don't install packages from outside netbsd's
package repository. We could change that and install IPC::Run via cpan, but
imo it's obviously nicer - and more representative of real world setups - if
we just rely on OS/distro packages.

I guess there's a bit of a philosophical question there: for a
test-only dependency in our CI, do we want it to be "representative of
real-world setups", or do we want it to work as well as it can?

I don't know what would be required to get netbsd to upate their IPC::Run
package.

It's listed as outdated here:

https://www.netbsd.org/~wiz/perl.html

but I'm not entirely clear on the process for requesting an update. I
guess I could ask pkgsrc-users@?

Thanks,
--Jacob

Hi,

On Thu, 18 Sept 2025 at 21:18, Andres Freund <andres@anarazel.de> wrote:

Hi,

On 2025-09-18 10:08:49 -0700, Jacob Champion wrote:

Andres pointed out that one of my new OAuth tests has flaked on NetBSD
[1]. The final line of debugging output is missing from the stderr
coming from psql.

I haven't ruled out a bug in my implementation, but I also remembered
that there was some discussion on the list about stderr being lost by
IPC::Run in rare circumstances [2], and that particular bug has since
been fixed [3]. Is it possible to (safely) switch the CI images to use
the most recent IPC::Run, so we can make use of Noah's fixes? NetBSD
is currently running 20231003.0.

A lot of things are safely possible :).

The stuff that's installed in the images is controlled by a bunch of scripts
in my pg-vm-images repository. For netbsd the relevant install is in

https://github.com/anarazel/pg-vm-images/blob/main/scripts/bsd/netbsd-prep-postgres.sh

As you can see we currently don't install packages from outside netbsd's
package repository. We could change that and install IPC::Run via cpan, but
imo it's obviously nicer - and more representative of real world setups - if
we just rely on OS/distro packages.

I have not checked it yet but we are on NetBSD 10.0 [1]https://github.com/anarazel/pg-vm-images/blob/main/packer/netbsd.pkrvars.hcl#L58C1-L58C75. I thought
upgrading to 10.1 might help... Apparently it does not, 10.1 too uses
p5-IPC-Run-20231003 [2]https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/index-all.html.

[1]: https://github.com/anarazel/pg-vm-images/blob/main/packer/netbsd.pkrvars.hcl#L58C1-L58C75
[2]: https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/index-all.html

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On 2025-09-18 11:48:15 -0700, Jacob Champion wrote:

On Thu, Sep 18, 2025 at 11:18 AM Andres Freund <andres@anarazel.de> wrote:

As you can see we currently don't install packages from outside netbsd's
package repository. We could change that and install IPC::Run via cpan, but
imo it's obviously nicer - and more representative of real world setups - if
we just rely on OS/distro packages.

I guess there's a bit of a philosophical question there: for a
test-only dependency in our CI, do we want it to be "representative of
real-world setups", or do we want it to work as well as it can?

I don't think there's a hard and fast rule, but I do think that it's a
reasonably important goal to make sure that the OS/distro actually can build
packages - and as part of that test those packages - reasonably well. So
requiring wildly different things than what is available on $platform to run
the tests reliably is bad.

Greetings,

Andres Freund

On Thu, Sep 18, 2025 at 12:16 PM Andres Freund <andres@anarazel.de> wrote:

I don't think there's a hard and fast rule, but I do think that it's a
reasonably important goal to make sure that the OS/distro actually can build
packages - and as part of that test those packages - reasonably well. So
requiring wildly different things than what is available on $platform to run
the tests reliably is bad.

I agree with that in general, but if a buildfarm operator reports
running into a already-fixed IPC::Run bug on an older distribution,
presumably we're going to tell them to install a newer IPC::Run,
right?

--Jacob

On Thu, Sep 18, 2025 at 11:49 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

I have not checked it yet but we are on NetBSD 10.0 [1]. I thought
upgrading to 10.1 might help... Apparently it does not, 10.1 too uses
p5-IPC-Run-20231003 [2].

Right. On the Debian side, looks like bookworm is stuck on 20220807.0
-- and updating to trixie won't help us either; that just gets us up
to 20231003.0.

--Jacob

On Thu, Sep 18, 2025 at 12:35 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

Right. On the Debian side, looks like bookworm is stuck on 20220807.0
-- and updating to trixie won't help us either; that just gets us up
to 20231003.0.

To more explicitly defend my position here: I can request updates from
package maintainers for <insert OS here>, but if the end result is
that our LTS distributions will still be lagging, there's not much
point. I'd rather bake the latest IPC::Run into our CI, and I'm happy
to contribute code and cycles towards that.

Any other opinions?

Thanks,
--Jacob

Jacob Champion <jacob.champion@enterprisedb.com> writes:

To more explicitly defend my position here: I can request updates from
package maintainers for <insert OS here>, but if the end result is
that our LTS distributions will still be lagging, there's not much
point. I'd rather bake the latest IPC::Run into our CI, and I'm happy
to contribute code and cycles towards that.

+1. Maybe we can stop doing this at some future time when all those
OSes have absorbed an up-to-date IPC::Run, but for the near term we're
merely exposing ourselves to known failures.

regards, tom lane

Hi,

On 2025-09-22 13:17:09 -0700, Jacob Champion wrote:

On Thu, Sep 18, 2025 at 12:35 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

Right. On the Debian side, looks like bookworm is stuck on 20220807.0
-- and updating to trixie won't help us either; that just gets us up
to 20231003.0.

To more explicitly defend my position here: I can request updates from
package maintainers for <insert OS here>, but if the end result is
that our LTS distributions will still be lagging, there's not much
point. I'd rather bake the latest IPC::Run into our CI, and I'm happy
to contribute code and cycles towards that.

Any other opinions?

I think your position has merit. However, I'd like to have at least one of the
tasks continue to use the non-built-in IPC::Run.

I'll review patches installing a newer IPC::Run.

Seperately, I wonder if we ought to add a warning to configure/meson output
about IPC::Run < 2025...

Greetings,

Andres Freund

On Mon, Sep 22, 2025 at 1:24 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

+1. Maybe we can stop doing this at some future time when all those
OSes have absorbed an up-to-date IPC::Run, but for the near term we're
merely exposing ourselves to known failures.

On Mon, Sep 22, 2025 at 1:27 PM Andres Freund <andres@anarazel.de> wrote:

I think your position has merit. However, I'd like to have at least one of the
tasks continue to use the non-built-in IPC::Run.

I'll review patches installing a newer IPC::Run.

I will work on that, then. Thanks both!

Seperately, I wonder if we ought to add a warning to configure/meson output
about IPC::Run < 2025...

Probably. I imagine it's going to be very noisy for a bit, though.

--Jacob

On Mon, Sep 22, 2025 at 3:35 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

On Mon, Sep 22, 2025 at 1:27 PM Andres Freund <andres@anarazel.de> wrote:

I think your position has merit. However, I'd like to have at least one of the
tasks continue to use the non-built-in IPC::Run.

I'll review patches installing a newer IPC::Run.

I will work on that, then. Thanks both!

I got stuck on this a few weeks back and had to put it down. Of all
things, I couldn't figure out how to download a Perl module in a
secure manner across _all_ of our supported platforms. I tried to get
--verify mode working with cpanm and found that it would simply
continue downloading and running code without verification if the
prerequisites weren't installed. (I am now reevaluating my own
personal use of cpanm.) The state of the world there looks horrifying
[1, 2].

I even considered pulling the tarball straight from Github and
hardcoding the shasum for it, but I haven't been able to figure out
how to Just Install That Thing I Downloaded via a blessed Perl
utility. (CPAN itself appears to _require_ interactive use at first
run, making it very difficult to use in a CI...)

Anyone have a recipe for doing this?

--Jacob

[1]: https://blogs.perl.org/users/neilb/2021/11/addressing-cpan-vulnerabilities-related-to-checksums.html
[2]: https://github.com/miyagawa/cpanminus/pull/674

Hi,

On 2025-10-31 14:40:41 -0700, Jacob Champion wrote:

On Mon, Sep 22, 2025 at 3:35 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

On Mon, Sep 22, 2025 at 1:27 PM Andres Freund <andres@anarazel.de> wrote:

I think your position has merit. However, I'd like to have at least one of the
tasks continue to use the non-built-in IPC::Run.

I'll review patches installing a newer IPC::Run.

I will work on that, then. Thanks both!

I got stuck on this a few weeks back and had to put it down. Of all
things, I couldn't figure out how to download a Perl module in a
secure manner across _all_ of our supported platforms. I tried to get
--verify mode working with cpanm and found that it would simply
continue downloading and running code without verification if the
prerequisites weren't installed. (I am now reevaluating my own
personal use of cpanm.) The state of the world there looks horrifying
[1, 2].

Yuck.

(CPAN itself appears to _require_ interactive use at first
run, making it very difficult to use in a CI...)

I think I had to figure this out in the past <dig>, ah, yea:
https://github.com/anarazel/pg-vm-images/blob/8ac88f59e368f35a63d3a0635ee7baf3436d5ee3/scripts/windows_install_mingw64.ps1#L25-L27

msys '(echo; echo o conf recommends_policy 0; echo notest install IPC::Run) | cpan'

Greetings,

Andres Freund

On Fri, Oct 31, 2025 at 2:55 PM Andres Freund <andres@anarazel.de> wrote:

I think I had to figure this out in the past <dig>, ah, yea:
https://github.com/anarazel/pg-vm-images/blob/8ac88f59e368f35a63d3a0635ee7baf3436d5ee3/scripts/windows_install_mingw64.ps1#L25-L27

msys '(echo; echo o conf recommends_policy 0; echo notest install IPC::Run) | cpan'

Heh. Okay, I'll give a version of that a shot.

Thanks!
--Jacob

On Fri, Oct 31, 2025 at 2:40 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

I even considered pulling the tarball straight from Github and
hardcoding the shasum for it

(For anyone who tries this in the future: Github's source tarballs
don't appear to be stable over time.)

On Fri, Oct 31, 2025 at 3:54 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

On Fri, Oct 31, 2025 at 2:55 PM Andres Freund <andres@anarazel.de> wrote:

msys '(echo; echo o conf recommends_policy 0; echo notest install IPC::Run) | cpan'

Heh. Okay, I'll give a version of that a shot.

The echo chain (plus additions) works, but I've gotten stuck at
signature verification:

https://cpan.org/authors/id/N/NJ/NJM/CHECKSUMS
gpg: Signature made Sun Aug 10 02:52:29 2025 UTC
gpg: using RSA key 071B468507812067912B951258FDA3CBB759E5C4
gpg: requesting key 58FDA3CBB759E5C4 from hkp://keyserver.ubuntu.com:11371
gpg: Can't check signature: No public key

Noah, do you know if that's a quick fix on the publishing side?

Thanks,
--Jacob

On Mon, Nov 17, 2025 at 10:13:30AM -0800, Jacob Champion wrote:

On Fri, Oct 31, 2025 at 3:54 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:

On Fri, Oct 31, 2025 at 2:55 PM Andres Freund <andres@anarazel.de> wrote:

msys '(echo; echo o conf recommends_policy 0; echo notest install IPC::Run) | cpan'

Heh. Okay, I'll give a version of that a shot.

The echo chain (plus additions) works, but I've gotten stuck at
signature verification:

https://cpan.org/authors/id/N/NJ/NJM/CHECKSUMS
gpg: Signature made Sun Aug 10 02:52:29 2025 UTC
gpg: using RSA key 071B468507812067912B951258FDA3CBB759E5C4
gpg: requesting key 58FDA3CBB759E5C4 from hkp://keyserver.ubuntu.com:11371
gpg: Can't check signature: No public key

Noah, do you know if that's a quick fix on the publishing side?

It's a CPAN-managed key. See if the key here gives what you need:
http://pgpkeys.eu/pks/lookup?search=071B468507812067912B951258FDA3CBB759E5C4&amp;fingerprint=on&amp;op=index

If it helps, you don't actually need to run an installation process on
IPC::Run. You can just unpack the tarball and add $PWD/IPC-Run/lib to the
PERL5LIB envar.

On Mon, Nov 17, 2025 at 12:22 PM Noah Misch <noah@leadboat.com> wrote:

It's a CPAN-managed key. See if the key here gives what you need:
http://pgpkeys.eu/pks/lookup?search=071B468507812067912B951258FDA3CBB759E5C4&amp;fingerprint=on&amp;op=index

It does, thanks, though I was surprised to see that I had to override
Module::Signature's defaults to get it to be used. gpg.conf was
ignored. So shouldn't Module::Signature default to a keyserver that
CPAN actually uses...?

If it helps, you don't actually need to run an installation process on
IPC::Run. You can just unpack the tarball and add $PWD/IPC-Run/lib to the
PERL5LIB envar.

I can't quite decide whether that would make things easier or harder.
The CPAN approach has been a lot of yak shaving, but once it works, it
should hopefully work the same across all the platforms...

Thanks,
--Jacob

On Tue, Nov 18, 2025 at 08:29:14AM -0800, Jacob Champion wrote:

On Mon, Nov 17, 2025 at 12:22 PM Noah Misch <noah@leadboat.com> wrote:

It's a CPAN-managed key. See if the key here gives what you need:
http://pgpkeys.eu/pks/lookup?search=071B468507812067912B951258FDA3CBB759E5C4&amp;fingerprint=on&amp;op=index

It does, thanks, though I was surprised to see that I had to override
Module::Signature's defaults to get it to be used. gpg.conf was
ignored. So shouldn't Module::Signature default to a keyserver that
CPAN actually uses...?

Yes, Module::Signature should do that, or the key should be on more
keyservers, or both. More keyservers would always be a good thing anyway for
such a prominent key.

If it helps, you don't actually need to run an installation process on
IPC::Run. You can just unpack the tarball and add $PWD/IPC-Run/lib to the
PERL5LIB envar.

I can't quite decide whether that would make things easier or harder.
The CPAN approach has been a lot of yak shaving, but once it works, it
should hopefully work the same across all the platforms...

Yep. Whichever works better for you.

Hi,

On 2025-11-18 09:14:38 -0800, Noah Misch wrote:

On Tue, Nov 18, 2025 at 08:29:14AM -0800, Jacob Champion wrote:

On Mon, Nov 17, 2025 at 12:22 PM Noah Misch <noah@leadboat.com> wrote:

It's a CPAN-managed key. See if the key here gives what you need:
http://pgpkeys.eu/pks/lookup?search=071B468507812067912B951258FDA3CBB759E5C4&amp;fingerprint=on&amp;op=index

It does, thanks, though I was surprised to see that I had to override
Module::Signature's defaults to get it to be used. gpg.conf was
ignored. So shouldn't Module::Signature default to a keyserver that
CPAN actually uses...?

Yes, Module::Signature should do that, or the key should be on more
keyservers, or both. More keyservers would always be a good thing anyway for
such a prominent key.

If it helps, you don't actually need to run an installation process on
IPC::Run. You can just unpack the tarball and add $PWD/IPC-Run/lib to the
PERL5LIB envar.

I can't quite decide whether that would make things easier or harder.
The CPAN approach has been a lot of yak shaving, but once it works, it
should hopefully work the same across all the platforms...

Yep. Whichever works better for you.

A quick ping on this thread, triggered a by a few related spurious cfbot
failures:

https://cirrus-ci.com/task/4697674038902784
https://cirrus-ci.com/task/6272264783527936

Are you stuck on anything right now, or just busy otherwise?

Greetings,

Andres Freund

On Wed, Dec 17, 2025 at 9:06 AM Andres Freund <andres@anarazel.de> wrote:

Are you stuck on anything right now, or just busy otherwise?

Busy otherwise; the work-in-progress is at

https://github.com/anarazel/pg-vm-images/pull/125

I need to fix the NetBSD installation recipe and then shave the next yak.

(The cycle time for building the image on my machine is high, between
configuring packer not to fail at various race conditions and having
to emulate x86-64, so in retrospect this was not an "easy win" of a
project.)

--Jacob

Hi,

On 2025-12-17 09:14:11 -0800, Jacob Champion wrote:

On Wed, Dec 17, 2025 at 9:06 AM Andres Freund <andres@anarazel.de> wrote:

Are you stuck on anything right now, or just busy otherwise?

Busy otherwise; the work-in-progress is at

https://github.com/anarazel/pg-vm-images/pull/125

Huh. I completely missed the notification for that. Ah, probably because it's
marked as draft...

I need to fix the NetBSD installation recipe and then shave the next yak.

(The cycle time for building the image on my machine is high, between
configuring packer not to fail at various race conditions and having
to emulate x86-64, so in retrospect this was not an "easy win" of a
project.)

If you promise to not push a thousand times a day, I'll add you as a
contributor to the repo so CI automatically runs and therefore builds the
images [1]They are built in a separate image repo. To use them, GCP_PROJECT in .cirrus.yml needs to be changed to pg-ci-images-dev....

Greetings,

Andres Freund

[1]: They are built in a separate image repo. To use them, GCP_PROJECT in .cirrus.yml needs to be changed to pg-ci-images-dev.
.cirrus.yml needs to be changed to pg-ci-images-dev.

On Wed, Dec 17, 2025 at 9:24 AM Andres Freund <andres@anarazel.de> wrote:

If you promise to not push a thousand times a day, I'll add you as a
contributor to the repo so CI automatically runs and therefore builds the
images [1]...

That'd be helpful. What scary power does this entail, though --

[1] They are built in a separate image repo. To use them, GCP_PROJECT in
.cirrus.yml needs to be changed to pg-ci-images-dev.

-- like, if I forget to do that, does it obliterate our prod images?

--Jacob

Hi,

On 2025-12-17 09:46:11 -0800, Jacob Champion wrote:

On Wed, Dec 17, 2025 at 9:24 AM Andres Freund <andres@anarazel.de> wrote:

If you promise to not push a thousand times a day, I'll add you as a
contributor to the repo so CI automatically runs and therefore builds the
images [1]...

That'd be helpful. What scary power does this entail, though --

Unless you merge the PR, not much, other than having the generated images to
pg-ci-images-dev (which nothing uses in the normal process of things). Only
builds in the "main" branch are used for pg-ci-images, the repo that postgres'
.cirrus.yml references.

[1] They are built in a separate image repo. To use them, GCP_PROJECT in
.cirrus.yml needs to be changed to pg-ci-images-dev.

-- like, if I forget to do that, does it obliterate our prod images?

No, the change you would need to make is to postgresql's .cirrus.yml to *use*
the generated images, since PR builds are not published to pg-ci-images.

Greetings,

Andres Freund