[PATCH] CI: Add a CPAN cache on Windows

Started by Jacob Champion18 days ago5 messageshackers
Jump to latest
#1Jacob Champion
jacob.champion@enterprisedb.com

Based on the conversation at [1]/messages/by-id/CAOYmi+kyuJHC7NTJWbzLBX25QnCsMVDbYWmv5tveSveNUyBAUw@mail.gmail.com.

This caches the Perl site directory after we've installed IPC::Run.
After consulting with Christoph Berg and David Wheeler on the Discord,
I've set up the cache to key on the output of `perl -V`, so that we'll
rebuild it after any change to Perl underneath us.

I went back and forth on doing the bare minimum (just save off the
CPAN source+build caches, then run CPAN each time) vs the more
involved strategy presented here (don't use CPAN at all on a cache
hit). The simpler approach only saved us about half of the up-to-60s
installation delay, because I can't seem to get CPAN to skip the
installation if IPC::Run is already installed.

So this patch is more code, but it's a lot faster once the cache is
warm. (The cache is also much smaller, as a minor bonus, but it was
only ~3MB to start with.)

--Jacob

[1]: /messages/by-id/CAOYmi+kyuJHC7NTJWbzLBX25QnCsMVDbYWmv5tveSveNUyBAUw@mail.gmail.com

Attachments:

0001-ci-Cache-the-CPAN-installation-on-Windows-runners.patchapplication/octet-stream; name=0001-ci-Cache-the-CPAN-installation-on-Windows-runners.patchDownload+78-21
#2Andres Freund
andres@anarazel.de
In reply to: Jacob Champion (#1)
Re: [PATCH] CI: Add a CPAN cache on Windows

Hi,

On 2026-06-05 11:05:08 -0700, Jacob Champion wrote:

Based on the conversation at [1].

This caches the Perl site directory after we've installed IPC::Run.

Cool!

Worth noting that while the baseline install time isn't that slow, it does
occasionally take a good bit longer, sometimes it has to retry for quite a few
minutes to download.

After consulting with Christoph Berg and David Wheeler on the Discord,
I've set up the cache to key on the output of `perl -V`, so that we'll
rebuild it after any change to Perl underneath us.

Makes sense.

@@ -808,6 +816,8 @@ jobs:
PG_TEST_USE_UNIX_SOCKETS: 1
PG_REGRESS_SOCK_DIR: 'd:\pgsock'
TAR: "c:/windows/system32/tar.exe"
+      CPAN_CACHE_DIRS: |
+        C:\Strawberry\perl\site

In 93d97349461347d952e8cebdf62f5aa84b4bd20a, I kinda had implemented such
caching for macos. There I used
PERL5LIB: ${HOME}/perl5/lib/perl5

to put the cache somewhere under our control. Any reason to not do that?

-      - name: Install dependencies
+      # Rebuild the CPAN cache whenever `perl -V` changes.
+      - name: Compute Perl version cache key
+        id: perlkey
+        shell: 'C:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"'
+        run: |
+          perl_hash=$(/c/Strawberry/perl/bin/perl -V | md5sum | cut -f1 -d ' ')
+          echo "key=perl${perl_hash}" >> "$GITHUB_OUTPUT"

Hm. It'd be nicer if we didn't have to repeat this part between the two
tasks... Perhaps we can make it work using the small overlap between the
different shells?

Or maybe shell: bash would actually work well enough between the tasks (without
-- login it might not overwrite PATH).

Greetings,

Andres Freund

#3Jacob Champion
jacob.champion@enterprisedb.com
In reply to: Andres Freund (#2)
Re: [PATCH] CI: Add a CPAN cache on Windows

On Fri, Jun 5, 2026 at 11:38 AM Andres Freund <andres@anarazel.de> wrote:

In 93d97349461347d952e8cebdf62f5aa84b4bd20a, I kinda had implemented such
caching for macos. There I used
PERL5LIB: ${HOME}/perl5/lib/perl5

to put the cache somewhere under our control. Any reason to not do that?

As a Perl novice, I was worried about setting that. The Windows
runners have multiple installations of Perl and I don't want them to
ever collide (similar to my PYTHONHOME concern in the other thread).
Is that a reasonable fear?

-      - name: Install dependencies
+      # Rebuild the CPAN cache whenever `perl -V` changes.
+      - name: Compute Perl version cache key
+        id: perlkey
+        shell: 'C:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"'
+        run: |
+          perl_hash=$(/c/Strawberry/perl/bin/perl -V | md5sum | cut -f1 -d ' ')
+          echo "key=perl${perl_hash}" >> "$GITHUB_OUTPUT"

Hm. It'd be nicer if we didn't have to repeat this part between the two
tasks... Perhaps we can make it work using the small overlap between the
different shells?

Or maybe shell: bash would actually work well enough between the tasks (without
-- login it might not overwrite PATH).

I'll see if I can make something work.

Thanks!
--Jacob

#4Andres Freund
andres@anarazel.de
In reply to: Jacob Champion (#3)
Re: [PATCH] CI: Add a CPAN cache on Windows

On 2026-06-05 12:27:50 -0700, Jacob Champion wrote:

On Fri, Jun 5, 2026 at 11:38 AM Andres Freund <andres@anarazel.de> wrote:

In 93d97349461347d952e8cebdf62f5aa84b4bd20a, I kinda had implemented such
caching for macos. There I used
PERL5LIB: ${HOME}/perl5/lib/perl5

to put the cache somewhere under our control. Any reason to not do that?

As a Perl novice, I was worried about setting that. The Windows
runners have multiple installations of Perl and I don't want them to
ever collide (similar to my PYTHONHOME concern in the other thread).
Is that a reasonable fear?

I'm not particularly concerned about that, tbh. If you suddenly switch which
perl you use, you'll might get some weird errors. Which seems fine with me.

#5Jacob Champion
jacob.champion@enterprisedb.com
In reply to: Andres Freund (#4)
Re: [PATCH] CI: Add a CPAN cache on Windows

On Fri, Jun 5, 2026 at 1:12 PM Andres Freund <andres@anarazel.de> wrote:

I'm not particularly concerned about that, tbh. If you suddenly switch which
perl you use, you'll might get some weird errors.

I'm finding that's easy to do by accident, though.

The interactions between
- Git-for-Windows MSYS,
- our full version(s) of MSYS on either C: or D:,
- the environment variables we set (esp. MSYSTEM), and
- the GITHUB_PATH
are fairly unintuitive to me. The windows-vs job pushes Strawberry
Perl into the PATH -- but I think that only works because there's no
MSYSTEM setting? I haven't found a way to get the windows-mingw job to
do the same thing, because it forces its own "native" Perl to take
precedence.

So if we want the two bash shells to behave similarly, I think some
more fundamental things need to be changed first, probably by someone
who understands MSYS2 better than I do. Maybe that's just a really
small tweak, but I can't see it.

--Jacob