ci: namespace ccache by PostgreSQL major version

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

When the Postgres major version is updated, CI fails with [1]:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI
information exists in the code, and so changes should be visible to
ccache. Maybe we are using ccache in the wrong mode or something (see
"depend mode", "direct mode", etc.).

Hi,

On Mon, 6 Jul 2026 at 10:52, Peter Eisentraut <peter@eisentraut.org> wrote:

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI
information exists in the code, and so changes should be visible to
ccache. Maybe we are using ccache in the wrong mode or something (see
"depend mode", "direct mode", etc.).

Thanks for the feedback!

Yes, you are right. I realized that after doing the further research
(I should have done that sooner, before sending the patch).

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On 2026-07-06 09:52:19 +0200, Peter Eisentraut wrote:

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

When the Postgres major version is updated, CI fails with [1]:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI information
exists in the code, and so changes should be visible to ccache. Maybe we
are using ccache in the wrong mode or something (see "depend mode", "direct
mode", etc.).

Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:

/messages/by-id/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb

Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.

We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.

Greetings,

Andres Freund

Hi,

On 2026-07-07 13:35:34 -0400, Andres Freund wrote:

On 2026-07-06 09:52:19 +0200, Peter Eisentraut wrote:

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

When the Postgres major version is updated, CI fails with [1]:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI information
exists in the code, and so changes should be visible to ccache. Maybe we
are using ccache in the wrong mode or something (see "depend mode", "direct
mode", etc.).

Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:

/messages/by-id/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb

Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.

We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.

Attached is the change I propose.

Greetings,

Andres

On Thu, 16 Jul 2026 at 23:27, Andres Freund <andres@anarazel.de> wrote:

Attached is the change I propose.

Seems reasonable (assuming it indeed fixes the issue)

Regardless, I think we should still scope the caches on PG version
though for a completely different reason: Any backpatches (currently
only to 19) will become less and less cached as PG20 diverges from
PG19 if we have a shared cache. We should in addition make sure that
we create shared cached from REL_XX_STABLE branches and generate their
cache keys in a way that other branches restore them.

To be clear, this would look quite different from Bilal's patch.

Hi,

On Fri, 17 Jul 2026 at 00:27, Andres Freund <andres@anarazel.de> wrote:

On 2026-07-07 13:35:34 -0400, Andres Freund wrote:

Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:

/messages/by-id/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb

Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.

We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.

Attached is the change I propose.

Thanks for the patch! LGTM and I confirm it fixes the problem.

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On Fri, 17 Jul 2026 at 01:19, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

Regardless, I think we should still scope the caches on PG version
though for a completely different reason: Any backpatches (currently
only to 19) will become less and less cached as PG20 diverges from
PG19 if we have a shared cache. We should in addition make sure that
we create shared cached from REL_XX_STABLE branches and generate their
cache keys in a way that other branches restore them.

AFAIK, if the cache hit ratio is below some point
(gha_ccache_decide.py -> 80 for branches other than master), the
branch regenerates its own cache. So, this shouldn't be a problem.

--
Regards,
Nazir Bilal Yavuz
Microsoft

On Fri, 17 Jul 2026 at 11:41, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

AFAIK, if the cache hit ratio is below some point
(gha_ccache_decide.py -> 80 for branches other than master), the
branch regenerates its own cache. So, this shouldn't be a problem.

Sure, but it would be nice to be able to create a shared cache by
pushing REL_19_STABLE if you're backporting multiple things.

Hi,

On 2026-07-17 13:50:30 +0200, Jelte Fennema-Nio wrote:

On Fri, 17 Jul 2026 at 11:41, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

AFAIK, if the cache hit ratio is below some point
(gha_ccache_decide.py -> 80 for branches other than master), the
branch regenerates its own cache. So, this shouldn't be a problem.

Sure, but it would be nice to be able to create a shared cache by
pushing REL_19_STABLE if you're backporting multiple things.

That's how it works. We restore the cache for master and for the current
branch. We save the cache, after pruning it, for the current branch iff the
hit rate was < 80%.

It'd be nice if we could restore the cache for the "base branch" of a branch,
instead of master, but that's not currently possible in GHA, from what I can
tell. Only caches on the default branch are accessible to other branches.

Greetings,

Andres Freund

On Fri, Jul 17, 2026 at 6:06 AM Andres Freund <andres@anarazel.de> wrote:

That's how it works. We restore the cache for master and for the current
branch. We save the cache, after pruning it, for the current branch iff the
hit rate was < 80%.

It'd be nice if we could restore the cache for the "base branch" of a branch,
instead of master, but that's not currently possible in GHA, from what I can
tell. Only caches on the default branch are accessible to other branches.

Quick tangent: is there a reason we put the refname into the cache
key? It seems like some of the logic there would happen by default if
we used the same cache prefix for all branches, since I thought GitHub
would perform the fallback-to-default-branch for us.

--Jacob

Hi,

On 2026-07-17 11:57:18 +0300, Nazir Bilal Yavuz wrote:

On Fri, 17 Jul 2026 at 00:27, Andres Freund <andres@anarazel.de> wrote:

On 2026-07-07 13:35:34 -0400, Andres Freund wrote:

Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:

/messages/by-id/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb

Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.

We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.

Attached is the change I propose.

Thanks for the patch! LGTM and I confirm it fixes the problem.

Thanks for reviewing. Pushed.

Greetings,

Andres

Hi,

On 2026-07-17 08:27:39 -0700, Jacob Champion wrote:

On Fri, Jul 17, 2026 at 6:06 AM Andres Freund <andres@anarazel.de> wrote:

That's how it works. We restore the cache for master and for the current
branch. We save the cache, after pruning it, for the current branch iff the
hit rate was < 80%.

It'd be nice if we could restore the cache for the "base branch" of a branch,
instead of master, but that's not currently possible in GHA, from what I can
tell. Only caches on the default branch are accessible to other branches.

Quick tangent: is there a reason we put the refname into the cache
key? It seems like some of the logic there would happen by default if
we used the same cache prefix for all branches, since I thought GitHub
would perform the fallback-to-default-branch for us.

For cfbot it's kinda important to have *both* the master and branch specific
caches for a run. Often, after a rebase of a branch to a newer master, the
branch specific key will have a lower hit rate than during the last run, but
master's cache applies to a lot of the part of the build not modified by the
branch.

I also have some hope that eventually github will allow using caches from more
than just the default branch. For testing to-be-backpatched bug fixes - on fix
specific, per-major branches, it's pretty painful that they can only start
with master's cache.

Greetings,

Andres Freund

On Fri Jul 17, 2026 at 3:06 PM CEST, Andres Freund wrote:

It'd be nice if we could restore the cache for the "base branch" of a branch,
instead of master, but that's not currently possible in GHA, from what I can
tell. Only caches on the default branch are accessible to other branches.

Hmm... The GHA permission model seems to be more complex every time I
look into it.... It's indeed not possible to restore caches from
non-default branches, *except* if the workflow was triggered by a PR and
that PR tries to restore a cache from the base branch of the PR.

So attached is something that makes this work as long as you open a PR
against the respective release branch. The more complex logic is
actually the part that cancels the branch build to make sure that you
don't run push and PR workflows at the same time.

After implementing that I think there's another approach possible too:
Allow users to trigger a cache build workflow *from the master branch*
that builds the cache for specific release branches (by checking them
out and building them).

On Fri, Jul 17, 2026 at 8:38 AM Andres Freund <andres@anarazel.de> wrote:

For cfbot it's kinda important to have *both* the master and branch specific
caches for a run. Often, after a rebase of a branch to a newer master, the
branch specific key will have a lower hit rate than during the last run, but
master's cache applies to a lot of the part of the build not modified by the
branch.

Oh, the rebase use case is interesting to think about. Thanks for the
explanation!

--Jacob

On Fri Jul 17, 2026 at 5:55 PM CEST, Jelte Fennema-Nio wrote:

After implementing that I think there's another approach possible too:
Allow users to trigger a cache build workflow *from the master branch*
that builds the cache for specific release branches (by checking them
out and building them).

Something like the attached, all done Claude. I'm gonna enjoy my weekend
now though. Feel free to take this over.

On Fri Jul 17, 2026 at 6:14 PM CEST, Jelte Fennema-Nio wrote:

Something like the attached, all done Claude. I'm gonna enjoy my weekend
now though. Feel free to take this over.

Changed it a bit and tried it out. This works pretty well and seems like
a nice improvement for people that actually backpatch things. I think I
like this simplicity a lot better than the patch I submitted earlier to
run CI on PRs.

Hi,

On Mon, 20 Jul 2026 at 23:46, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

On Fri Jul 17, 2026 at 6:14 PM CEST, Jelte Fennema-Nio wrote:

Something like the attached, all done Claude. I'm gonna enjoy my weekend
now though. Feel free to take this over.

Changed it a bit and tried it out. This works pretty well and seems like
a nice improvement for people that actually backpatch things. I think I
like this simplicity a lot better than the patch I submitted earlier to
run CI on PRs.

I think this is a nice improvement for the people who backpatch things
and have feature branches.

Since this CI run will only run for updating the cache, I think
running tests etc. are not important and time consuming. What do you
think about disabling them for these types of CI runs; so they will
only build, then update the cache and done?

--
Regards,
Nazir Bilal Yavuz
Microsoft

On Tue, 21 Jul 2026 at 13:26, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

Since this CI run will only run for updating the cache, I think
running tests etc. are not important and time consuming. What do you
think about disabling them for these types of CI runs; so they will
only build, then update the cache and done?

Sounds reasonable to me