ci: Follow-up CI patches
Hi,
Andres shared some CI patches in the 'Heads Up: cirrus-ci is shutting
down June 1st' thread [1]/messages/by-id/iggjozfshwbqpv33x5jqwtju5k5zrkyu3257dlifxkhtpg7eoq@k2ccyrdi5dtu. Since this thread is already quite long, I
wanted to create a new thread for them.
There were 9 patches in the thread and 3 of them already committed:
- 0001-ci-Generate-crashlogs-on-Windows.patch: Committed at d374280e183.
- 0002-ci-Use-meson-for-most-of-CompilerWarnings-it-s-.patch:
Committed at 3ca57291ff.
- 0006-ci-Use-optimized-build-for-mingw.patch: Committed at 4075409de0.
----------
These are the uncommitted patches, I rebased them with small changes:
- 0003-ci-Make-msys2-install-smaller.patch
No functional changes; the commit message is updated (where -> were).
- 0004-ci-Make-our-own-msys2-install-from-scratch-inst.patch
'msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.31.1'
this SHA points to v2.32.0, it is fixed. Also Jacob's
```
Can we just get rid of the MSYSTEM envvar now? (And the
MSYS2_PATH_TYPE, if we replace it with `path-type: inherit` in the
action parameters?) See attached -0011.
```
comment [2]/messages/by-id/CAOYmi+kOGycb6Vw-Z19=M8js6d36=6=kUw4Cu5+ueQej77ig+g@mail.gmail.com is applied.
- 0005-squash-or-drop-Use-pacboy-for-shorter-package-n.patch
No change. I lean toward dropping this because explicitly naming
packages makes it easier to search if something goes wrong. Jacob said
it looks nicer with this change [3]/messages/by-id/CAOYmi+nEqAYKjh1r7HiKQ=LhFU5LMHw=XHuEkUr9We6b0judsg@mail.gmail.com.
- 0007-ci-get-build-test-concurrency-from-environment.patch
I used %NUMBER_OF_PROCESSORS% instead of 'bash -c nproc' to not depend
on bash.
- 0008-ci-Make-runs-on-overridable.patch
Separatd MacPorts caches by runner architecture to avoid restoring
incompatible binaries when changing runners.
- 0009-ci-Move-to-per-job-naming-for-containers.patch
Removed 'options: *linux_container_options' from CompilerWarnings
since it is not used.
----------
Rebased and updated patches are attached. Order of the patches has
changed because I got them from Andres' working repository. I CCed
people from the [1]/messages/by-id/iggjozfshwbqpv33x5jqwtju5k5zrkyu3257dlifxkhtpg7eoq@k2ccyrdi5dtu thread.
[1]: /messages/by-id/iggjozfshwbqpv33x5jqwtju5k5zrkyu3257dlifxkhtpg7eoq@k2ccyrdi5dtu
[2]: /messages/by-id/CAOYmi+kOGycb6Vw-Z19=M8js6d36=6=kUw4Cu5+ueQej77ig+g@mail.gmail.com
[3]: /messages/by-id/CAOYmi+nEqAYKjh1r7HiKQ=LhFU5LMHw=XHuEkUr9We6b0judsg@mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
v14-0001-ci-get-build-test-concurrency-from-environment.patchtext/x-patch; charset=US-ASCII; name=v14-0001-ci-get-build-test-concurrency-from-environment.patchDownload+27-18
v14-0002-ci-Make-msys2-install-smaller.patchtext/x-patch; charset=US-ASCII; name=v14-0002-ci-Make-msys2-install-smaller.patchDownload+6-3
v14-0003-ci-Make-our-own-msys2-install-from-scratch-inste.patchtext/x-patch; charset=US-ASCII; name=v14-0003-ci-Make-our-own-msys2-install-from-scratch-inste.patchDownload+33-42
v14-0004-squash-or-drop-Use-pacboy-for-shorter-package-na.patchtext/x-patch; charset=US-ASCII; name=v14-0004-squash-or-drop-Use-pacboy-for-shorter-package-na.patchDownload+15-14
v14-0005-ci-Make-runs-on-overridable.patchtext/x-patch; charset=US-ASCII; name=v14-0005-ci-Make-runs-on-overridable.patchDownload+63-25
v14-0006-ci-Move-to-per-job-naming-for-containers.patchtext/x-patch; charset=US-ASCII; name=v14-0006-ci-Move-to-per-job-naming-for-containers.patchDownload+20-12
Hi,
On 2026-11-04 11:28:02 +0300, Nazir Bilal Yavuz wrote:
Andres shared some CI patches in the 'Heads Up: cirrus-ci is shutting
down June 1st' thread [1]. Since this thread is already quite long, I
wanted to create a new thread for them.
Thanks!
Peter, after all of this, I'd like to backpatch CI back to where we used to
have CI support. One of the things that makes that slightly more painful is
that there already are divergences between 19 and master. I'd like to
backport
2026-10-25 4075409de08 ci: Enable PCH in mingw cross job
2026-10-25 3ca57291ff3 ci: Convert CompilerWarnings job to meson
to 19, to keep the divergences smaller.
I think
2026-10-19 0f3b42ce806 Add missing dependencies to meson headerscheck target
ought to be in 19 regardless of anything, given that meson headerscheck was
added to 19?
All: I'm planning to put all of these into both 19 and master, I don't want to
end up with different container names across branches, nor with a partial
ability to change which runner CI runs on. Speak up now, or forever hold your
piece :)
- 0004-ci-Make-our-own-msys2-install-from-scratch-inst.patch
'msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.31.1'
this SHA points to v2.32.0, it is fixed.
We should probably update it to v2.33.0 now. I'll do that, no need to send a
new version just for that.
- 0007-ci-get-build-test-concurrency-from-environment.patch
I used %NUMBER_OF_PROCESSORS% instead of 'bash -c nproc' to not depend
on bash.
Ah, nice.
- 0008-ci-Make-runs-on-overridable.patch
Separatd MacPorts caches by runner architecture to avoid restoring
incompatible binaries when changing runners.
Hm. I'd perhaps split that aspect out into is own commit.
Subject: [PATCH v14 1/6] ci: get build/test concurrency from environment
LGTM, I think I'll test and then apply this.
Subject: [PATCH v14 2/6] ci: Make msys2 install smaller
Subject: [PATCH v14 3/6] ci: Make our own msys2 install from scratch, instead
of moving
Dito.
Subject: [PATCH v14 4/6] squash-or-drop: Use pacboy for shorter package names
Since there were no clear votes, I'm going to use a roll of dice to decide :)
More seriously, I am inclined to go with it, it seems possible that we'll want
to run CI on arm windows in the future, and due to the lack of environment
variables support in the install: block (at least i don't think there is),
that'll be more painful with the prefixes.
Subject: [PATCH v14 5/6] ci: Make runs-on overridable
Previously what runners CI ran on was hardcoded. Unfortunately that makes it
harder to use faster or self hosted runners (e.g. in a private repository
where the runners are very slow, or self hosted runners for cfbot).Fix that by allowing to override the runner types via repository/organization
variables. How to do that is documented in src/tools/ci/README.Testing faster github hosted runners showed that we have some dependencies on
specific locations, fix most of those. The MSYS2 installation changes are
handled in a separate commit.
Note to self: I think this msys2 note should be dropped before pushing, given
that those were addressed in a preceding commit.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: /messages/by-id/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33
Note to self^2: Add additional links.
@@ -154,7 +148,8 @@ jobs: warn-if-not-opted-in: name: Report if not opted into CI if: ${{vars.PG_CI_ENABLED != '1'}} - runs-on: ubuntu-slim + # See the setup task for an explanation + runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} steps: - name: Warn env: ... + runs_on_linux: ${{ case(vars.PG_CI_RUNS_ON_LINUX != '', vars.PG_CI_RUNS_ON_LINUX, 'ubuntu-24.04') }}
I'd love if there were a neater way to do this, but I don't know of any :(.
I'll play around with it to see if I can come up with something neater,
particularly the repetition of the variable name is annoying.
Greetings,
Andres Freund
Hi,
On Wed, 4 Nov 2026 at 17:36, Andres Freund <andres@anarazel.de> wrote:
On 2026-11-04 11:28:02 +0300, Nazir Bilal Yavuz wrote:
Andres shared some CI patches in the 'Heads Up: cirrus-ci is shutting
down June 1st' thread [1]. Since this thread is already quite long, I
wanted to create a new thread for them.Thanks!
Here are two more follow-up patches; the other patches are unchanged.
First one (0007) upgrades the OS versions:
Ubuntu 24.04 -> Ubuntu 26.04
Windows 2022 -> Windows 2025
Macos 15 -> Macos 26 and Macports 2.10.1 -> 2.12.7
All tested and work correctly.
Second one (0008) uses parallel steps feature [1]https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#example-running-steps-in-parallel in the
CompilerWarnings task. This feature helps run some steps in parallel
in the background. I tested a bit to find the best method for
parallelizing, from the commit message:
```
Different methods are tested with cold and warm compiler caches on a
four-core GitHub-hosted runner. Total CompilerWarnings job times are:
Cold Warm
Default 20m 56s 3m 57s
Parallel configure, sequential builds 19m 47s 3m 04s
Pairs of checks, ninja -j 2 18m 10s 3m 49s
All checks in parallel, ninja -j 1 15m 26s 2m 30s
All checks in parallel, default ninja jobs 17m 03s 3m 48s
- Parallel configure, sequential builds: Configure all seven checks
concurrently, then build them one at a time using Ninja’s default
concurrency.
- Pairs of checks, ninja -j 2: Run configuration and build in three
successive pairs with two Ninja jobs per check, then run the
remaining header check alone with default concurrency.
- All checks in parallel, ninja -j 1: Run all seven configure/build
sequences concurrently, limiting each Ninja invocation to one job.
- All checks in parallel, default Ninja jobs: Same as above but using
Ninja's default concurrency.
Based on this result, 'all checks in parallel, ninja -j 1' is selected.
```
I used $j = max($available_cpus / $parallel_jobs, 1) and that
calculates 1 since we have 4 CPU. I am not sure this is the right
method because if some jobs finish early and free up CPU resources,
other tasks will still use $j. However, this method is the fastest,
which I didn't expect it. So this one is used.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
v15-0001-ci-get-build-test-concurrency-from-environment.patchtext/x-patch; charset=US-ASCII; name=v15-0001-ci-get-build-test-concurrency-from-environment.patchDownload+27-18
v15-0002-ci-Make-msys2-install-smaller.patchtext/x-patch; charset=US-ASCII; name=v15-0002-ci-Make-msys2-install-smaller.patchDownload+6-3
v15-0003-ci-Make-our-own-msys2-install-from-scratch-inste.patchtext/x-patch; charset=US-ASCII; name=v15-0003-ci-Make-our-own-msys2-install-from-scratch-inste.patchDownload+33-42
v15-0004-squash-or-drop-Use-pacboy-for-shorter-package-na.patchtext/x-patch; charset=US-ASCII; name=v15-0004-squash-or-drop-Use-pacboy-for-shorter-package-na.patchDownload+15-14
v15-0005-ci-Make-runs-on-overridable.patchtext/x-patch; charset=US-ASCII; name=v15-0005-ci-Make-runs-on-overridable.patchDownload+63-25
v15-0006-ci-Move-to-per-job-naming-for-containers.patchtext/x-patch; charset=US-ASCII; name=v15-0006-ci-Move-to-per-job-naming-for-containers.patchDownload+20-12
v15-0007-ci-Upgrade-GitHub-Actions-runner-OS-versions.patchtext/x-patch; charset=US-ASCII; name=v15-0007-ci-Upgrade-GitHub-Actions-runner-OS-versions.patchDownload+16-8
v15-0008-ci-Run-compiler-warning-checks-in-parallel.patchtext/x-patch; charset=UTF-8; name=v15-0008-ci-Run-compiler-warning-checks-in-parallel.patchDownload+117-102
Hi,
On 2026-11-04 09:36:03 -0500, Andres Freund wrote:
Peter, after all of this, I'd like to backpatch CI back to where we used to
have CI support. One of the things that makes that slightly more painful is
that there already are divergences between 19 and master. I'd like to
backport2026-10-25 4075409de08 ci: Enable PCH in mingw cross job
2026-10-25 3ca57291ff3 ci: Convert CompilerWarnings job to mesonto 19, to keep the divergences smaller.
I think
2026-10-19 0f3b42ce806 Add missing dependencies to meson headerscheck targetought to be in 19 regardless of anything, given that meson headerscheck was
added to 19?
I'll do those backpatches in the next hour, unless somebody protests soon.
@@ -154,7 +148,8 @@ jobs: warn-if-not-opted-in: name: Report if not opted into CI if: ${{vars.PG_CI_ENABLED != '1'}} - runs-on: ubuntu-slim + # See the setup task for an explanation + runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} steps: - name: Warn env: ... + runs_on_linux: ${{ case(vars.PG_CI_RUNS_ON_LINUX != '', vars.PG_CI_RUNS_ON_LINUX, 'ubuntu-24.04') }}I'd love if there were a neater way to do this, but I don't know of any :(.
I'll play around with it to see if I can come up with something neater,
particularly the repetition of the variable name is annoying.
It turns out one can just do ${{ vars.PG_CI_RUNS_ON_LINUX || 'ubuntu-24.04' }}.
Not super well documented, but looks to be fairly widely used. Think that's a
good bit neater.
Also worked on making the image: simpler. There's now one format() template
(still a long line), but the individual tasks now are
- image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-autoconf:${{ needs.setup.outputs.container_tag }}
+ image: ${{ format(needs.setup.outputs.container_image, 'linux-autoconf') }}
which seems a lot easier to read to me.
Here's a mildly polished version:
- Adoped the simpler runs-on: and image: formulations
- An AI review raised the following, which I addressed:
- location: ${{github.workspace}} is suboptimal, because it means that the
directory will be searched for logs in case of a failure. Testing shows
that setup-msys2 will pick a sane directory, so I think just not
specifying a directory makes sense
- CHERE_INVOKING is now obsolete
- The socket dir could end up with a too long path. I don't entirely remember
how I ended up with the previous formulation, I think it had to do with
issues around the paths in the container being different than outside.
The new formulation of just creating it in the container seemed better
- added the discussion links, dropped references, as mentioned in my earlier
email
What do you think?
Greetings,
Andres Freund
Attachments:
v15a-0001-ci-Get-build-test-concurrency-from-environment.patchtext/x-diff; charset=us-asciiDownload+26-18
v15a-0002-ci-Make-msys2-install-smaller.patchtext/x-diff; charset=us-asciiDownload+6-3
v15a-0003-ci-Use-setup-msys2-instead-of-relocating-preins.patchtext/x-diff; charset=us-asciiDownload+36-43
v15a-0004-ci-Make-runs-on-overridable.patchtext/x-diff; charset=us-asciiDownload+66-28
v15a-0005-ci-Move-to-per-job-naming-for-containers.patchtext/x-diff; charset=us-asciiDownload+22-12
On 2026-11-05 14:29:41 -0500, Andres Freund wrote:
On 2026-11-04 09:36:03 -0500, Andres Freund wrote:
I think
2026-10-19 0f3b42ce806 Add missing dependencies to meson headerscheck targetought to be in 19 regardless of anything, given that meson headerscheck was
added to 19?I'll do those backpatches in the next hour, unless somebody protests soon.
Done now. That's done now.
Hi,
On Thu, 5 Nov 2026 at 22:29, Andres Freund <andres@anarazel.de> wrote:
@@ -154,7 +148,8 @@ jobs: warn-if-not-opted-in: name: Report if not opted into CI if: ${{vars.PG_CI_ENABLED != '1'}} - runs-on: ubuntu-slim + # See the setup task for an explanation + runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} steps: - name: Warn env: ... + runs_on_linux: ${{ case(vars.PG_CI_RUNS_ON_LINUX != '', vars.PG_CI_RUNS_ON_LINUX, 'ubuntu-24.04') }}I'd love if there were a neater way to do this, but I don't know of any :(.
I'll play around with it to see if I can come up with something neater,
particularly the repetition of the variable name is annoying.It turns out one can just do ${{ vars.PG_CI_RUNS_ON_LINUX || 'ubuntu-24.04' }}.
Not super well documented, but looks to be fairly widely used. Think that's a
good bit neater.Also worked on making the image: simpler. There's now one format() template (still a long line), but the individual tasks now are - image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-autoconf:${{ needs.setup.outputs.container_tag }} + image: ${{ format(needs.setup.outputs.container_image, 'linux-autoconf') }}which seems a lot easier to read to me.
I agree, that is definitely better.
Here's a mildly polished version:
- Adoped the simpler runs-on: and image: formulations
- An AI review raised the following, which I addressed:
- location: ${{github.workspace}} is suboptimal, because it means that the
directory will be searched for logs in case of a failure. Testing shows
that setup-msys2 will pick a sane directory, so I think just not
specifying a directory makes sense
I ran another AI review and it found similar problem for the
'PG_REGRESS_SOCK_DIR'. It said 'RUNNER_TEMP' could be too long and
cause a problem. From AI review:
```
PostgreSQL’s Windows sockaddr_un.sun_path is 108 bytes. The server
rejects socket paths longer than 107 bytes, including the appended
/.s.PGSQL.<port>. With a five-digit port, this leaves 85 bytes for
RUNNER_TEMP.
```
Currently it is set to `RUNNER_TEMP=D:\a\_temp` [1]https://docs.github.com/en/actions/reference/workflows-and-actions/variables#default-environment-variables so I don't expect
it to be a problem but just wanted to mention it.
- CHERE_INVOKING is now obsolete
- The socket dir could end up with a too long path. I don't entirely remember
how I ended up with the previous formulation, I think it had to do with
issues around the paths in the container being different than outside.
The new formulation of just creating it in the container seemed better- added the discussion links, dropped references, as mentioned in my earlier
What do you think?
Rest LGTM.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Hi,
I've pushed the previous patches I had posted to 19 and master. I also have
an LLM translated set of patches for CI for 15-18 that needs some more
polish...
On 2026-11-05 17:22:36 +0300, Nazir Bilal Yavuz wrote:
Here are two more follow-up patches; the other patches are unchanged.
Can you rebase them over what's now been merged?
Different methods are tested with cold and warm compiler caches on a
four-core GitHub-hosted runner. Total CompilerWarnings job times are:Cold Warm
Default 20m 56s 3m 57s
Parallel configure, sequential builds 19m 47s 3m 04s
Pairs of checks, ninja -j 2 18m 10s 3m 49s
All checks in parallel, ninja -j 1 15m 26s 2m 30s
All checks in parallel, default ninja jobs 17m 03s 3m 48s
I think the gains of "All checks in parallel, ninja -j 1" are large enough to
make this worthwhile pursuing?
Subject: [PATCH v15 7/8] ci: Upgrade GitHub Actions' runner OS versions
Use Ubuntu 26.04, macOS 26, and Windows Server 2025 as runner operating
sytems.Update the MacPorts version to one providing a macOS 26 installer.
Locate Visual Studio with vswhere instead of assuming the installation
path, as new versions might provide different Visual Studio versions
(2026 in this case).
Might be worth breaking out the vswhere stuff might best be broken out into
its own commit?
Subject: [PATCH v15 8/8] ci: Run compiler warning checks in parallel
Use a GitHub Actions parallel step group for the CompilerWarnings job. + PARALLEL_CHECKS: 7 + steps: - *nix_sysinfo_step - *checkout_step - *ccache_restore_default_step - *ccache_restore_branch_step+ # Share the runner's CPUs between the seven parallel builds below, + # rather than letting each ninja invocation use all of them. + - name: Set build concurrency run: | + jobs=$(( $(nproc) / $PARALLEL_CHECKS )) + if [ "$jobs" -lt 1 ]; then jobs=1; fi + echo "Ninja jobs per build: $jobs" + echo "NINJA_JOBS=$jobs" >> "$GITHUB_ENV" + + # Each check needs its own build directory. The parallel group waits + # for all checks before the cache is saved and failure logs uploaded. + # Update PARALLEL_CHECKS when adding or removing a check.
Kinda annoying to have to maintain PARALLEL_CHECKS that way, but I can't
immediately come up with a better way.
Thanks,
Andres Freund
Hi,
On Sat, 7 Nov 2026 at 23:10, Andres Freund <andres@anarazel.de> wrote:
I've pushed the previous patches I had posted to 19 and master. I also have
an LLM translated set of patches for CI for 15-18 that needs some more
polish...On 2026-11-05 17:22:36 +0300, Nazir Bilal Yavuz wrote:
Here are two more follow-up patches; the other patches are unchanged.
Can you rebase them over what's now been merged?
Done in v16.
Different methods are tested with cold and warm compiler caches on a
four-core GitHub-hosted runner. Total CompilerWarnings job times are:Cold Warm
Default 20m 56s 3m 57s
Parallel configure, sequential builds 19m 47s 3m 04s
Pairs of checks, ninja -j 2 18m 10s 3m 49s
All checks in parallel, ninja -j 1 15m 26s 2m 30s
All checks in parallel, default ninja jobs 17m 03s 3m 48sI think the gains of "All checks in parallel, ninja -j 1" are large enough to
make this worthwhile pursuing?
I agree it is %~25 better compared to upstream.
Subject: [PATCH v15 7/8] ci: Upgrade GitHub Actions' runner OS versions
Use Ubuntu 26.04, macOS 26, and Windows Server 2025 as runner operating
sytems.Update the MacPorts version to one providing a macOS 26 installer.
Locate Visual Studio with vswhere instead of assuming the installation
path, as new versions might provide different Visual Studio versions
(2026 in this case).Might be worth breaking out the vswhere stuff might best be broken out into
its own commit?
Yes, the vswhere stuff is in 0001 now.
Subject: [PATCH v15 8/8] ci: Run compiler warning checks in parallel
Use a GitHub Actions parallel step group for the CompilerWarnings job. + PARALLEL_CHECKS: 7 + steps: - *nix_sysinfo_step - *checkout_step - *ccache_restore_default_step - *ccache_restore_branch_step+ # Share the runner's CPUs between the seven parallel builds below, + # rather than letting each ninja invocation use all of them. + - name: Set build concurrency run: | + jobs=$(( $(nproc) / $PARALLEL_CHECKS )) + if [ "$jobs" -lt 1 ]; then jobs=1; fi + echo "Ninja jobs per build: $jobs" + echo "NINJA_JOBS=$jobs" >> "$GITHUB_ENV" + + # Each check needs its own build directory. The parallel group waits + # for all checks before the cache is saved and failure logs uploaded. + # Update PARALLEL_CHECKS when adding or removing a check.Kinda annoying to have to maintain PARALLEL_CHECKS that way, but I can't
immediately come up with a better way.
Yes, I couldn't come up with better solution. Having a comment makes
it easier to maintain at least.
--
Regards,
Nazir Bilal Yavuz
Microsoft