ci: namespace ccache by PostgreSQL major version

Started by Nazir Bilal Yavuz7 days ago4 messageshackers
Jump to latest
#1Nazir Bilal Yavuz
byavuz81@gmail.com

Hi,

When the Postgres major version is updated, CI fails with [1]https://github.com/postgresql-cfbot/postgresql/actions/runs/28606913807/job/84829900634:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

[1]: https://github.com/postgresql-cfbot/postgresql/actions/runs/28606913807/job/84829900634

--
Regards,
Nazir Bilal Yavuz
Microsoft

Attachments:

v1-0001-ci-namespace-ccache-by-PostgreSQL-major-version.patchtext/x-patch; charset=US-ASCII; name=v1-0001-ci-namespace-ccache-by-PostgreSQL-major-version.patchDownload+14-5
#2Peter Eisentraut
peter_e@gmx.net
In reply to: Nazir Bilal Yavuz (#1)
Re: ci: namespace ccache by PostgreSQL major version

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

When the Postgres major version is updated, CI fails with [1]:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI
information exists in the code, and so changes should be visible to
ccache. Maybe we are using ccache in the wrong mode or something (see
"depend mode", "direct mode", etc.).

#3Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Peter Eisentraut (#2)
Re: ci: namespace ccache by PostgreSQL major version

Hi,

On Mon, 6 Jul 2026 at 10:52, Peter Eisentraut <peter@eisentraut.org> wrote:

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI
information exists in the code, and so changes should be visible to
ccache. Maybe we are using ccache in the wrong mode or something (see
"depend mode", "direct mode", etc.).

Thanks for the feedback!

Yes, you are right. I realized that after doing the further research
(I should have done that sooner, before sending the patch).

--
Regards,
Nazir Bilal Yavuz
Microsoft

#4Andres Freund
andres@anarazel.de
In reply to: Peter Eisentraut (#2)
Re: ci: namespace ccache by PostgreSQL major version

Hi,

On 2026-07-06 09:52:19 +0200, Peter Eisentraut wrote:

On 03.07.26 16:26, Nazir Bilal Yavuz wrote:

Hi,

When the Postgres major version is updated, CI fails with [1]:

2026-07-02 17:01:01.938 UTC [828] FATAL: incompatible library
"D:/a/postgresql/postgresql/build/tmp_install/usr/local/pgsql/lib/utf8_and_win.dll":
version mismatch
2026-07-02 17:01:01.938 UTC [828] DETAIL: Server is version 20,
library is version 19.

This happens because the GitHub Actions ccache is restored across a
version bump, so objects built against the previous version can be
reused, producing libraries that no longer match the server version.

Here is an attempt to solve this problem by namespacing ccache by
Postgres major version. I added a 'PG_MAJOR_VERSION' variable to the
CI file and used that as a prefix to ccache key. I made this
'PG_MAJOR_VERSION' variable automatically updated by
'version_stamp.pl' script.

Another solution could be reading the version from build files (e.g
meson.build), but then this read needs to be done at each CI run.

I'm suspicious about this direction. The major version is not the only
piece of data that determines ABI compatibility between the server and
extensions. This would only be a partial information. The ABI information
exists in the code, and so changes should be visible to ccache. Maybe we
are using ccache in the wrong mode or something (see "depend mode", "direct
mode", etc.).

Yea, I don't think that's the right answer. I'm reasonably sure I debugged
this issue a while ago:

/messages/by-id/phsrssp75npoyalqsolcd7fmnmlbzbmquc2p7w7mqjlw7432jk@bzskz3luyjvb

Not a lot has happened on the ccache front, so I suspect we should start
adding the -fpch-deps flag I mentioned in that thread.

We could make adding -fpch-deps conditional on pch support being enabled, but
given that it doesn't do anything when pch is not used, I'd just add it when
supported by the compiler.

Greetings,

Andres Freund