Upgrade Debian CI images to Bookworm
Hi,
Bookworm versions of the Debian CI images are available now [0]https://github.com/anarazel/pg-vm-images/pull/91. The
patches to use these images are attached.
'v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_16+.patch' patch can
be applied to both upstream and REL_16 and all of the tasks finish
successfully.
'v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_15.patch' patch can
be applied to REL_15 but it gives a compiler warning. The fix for this
warning is proposed here [1]postgr.es/m/CAN55FZ0o9wqVoMTh_gJCmj_%2B4XbX9VXzQF8OySPZ0R1saxV3bA%40mail.gmail.com. After the fix is applied, all of the
tasks finish successfully.
Any kind of feedback would be appreciated.
[0]: https://github.com/anarazel/pg-vm-images/pull/91
[1]: postgr.es/m/CAN55FZ0o9wqVoMTh_gJCmj_%2B4XbX9VXzQF8OySPZ0R1saxV3bA%40mail.gmail.com
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_16+.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_16+.patchDownload
From 26f4b2425cb04dc7218142f24f151d3698f33191 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Mon, 13 May 2024 10:56:28 +0300
Subject: [PATCH v1] Upgrade Debian CI images to Bookworm
New Debian version, namely Bookworm, is released. Use these new images
in CI tasks.
Perl version is upgraded in the Bookworm images, so update Perl version
at 'Linux - Debian Bookworm - Meson' task as well.
Upgrading Debian CI images to Bookworm PR: https://github.com/anarazel/pg-vm-images/pull/91
---
.cirrus.tasks.yml | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 82261eccb85..d2ff833fcad 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -65,7 +65,7 @@ task:
CPUS: 4
BUILD_JOBS: 8
TEST_JOBS: 8
- IMAGE_FAMILY: pg-ci-bullseye
+ IMAGE_FAMILY: pg-ci-bookworm
CCACHE_DIR: ${CIRRUS_WORKING_DIR}/ccache_dir
# no options enabled, should be small
CCACHE_MAXSIZE: "150M"
@@ -241,7 +241,7 @@ task:
CPUS: 4
BUILD_JOBS: 4
TEST_JOBS: 8 # experimentally derived to be a decent choice
- IMAGE_FAMILY: pg-ci-bullseye
+ IMAGE_FAMILY: pg-ci-bookworm
CCACHE_DIR: /tmp/ccache_dir
DEBUGINFOD_URLS: "https://debuginfod.debian.net"
@@ -312,7 +312,7 @@ task:
#DEBIAN_FRONTEND=noninteractive apt-get -y install ...
matrix:
- - name: Linux - Debian Bullseye - Autoconf
+ - name: Linux - Debian Bookworm - Autoconf
env:
SANITIZER_FLAGS: -fsanitize=address
@@ -346,7 +346,7 @@ task:
on_failure:
<<: *on_failure_ac
- - name: Linux - Debian Bullseye - Meson
+ - name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
@@ -373,7 +373,7 @@ task:
${LINUX_MESON_FEATURES} \
-Dllvm=disabled \
--pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \
- -DPERL=perl5.32-i386-linux-gnu \
+ -DPERL=perl5.36-i386-linux-gnu \
-DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
build-32
EOF
@@ -648,7 +648,7 @@ task:
env:
CPUS: 4
BUILD_JOBS: 4
- IMAGE_FAMILY: pg-ci-bullseye
+ IMAGE_FAMILY: pg-ci-bookworm
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
--
2.43.0
v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_15.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_15.patchDownload
From d8f4b660bdd408fa22c857e24d1b4d05ff41df03 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
Date: Mon, 13 May 2024 11:55:48 +0300
Subject: [PATCH v1] Upgrade Debian CI images to Bookworm
New Debian version, namely Bookworm, is released. Use these new images
in CI tasks.
Upgrading Debian CI images to Bookworm PR: https://github.com/anarazel/pg-vm-images/pull/91
---
.cirrus.tasks.yml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index b03d128184d..a2735f8bb60 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -138,13 +138,13 @@ LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >-
task:
- name: Linux - Debian Bullseye
+ name: Linux - Debian Bookworm
env:
CPUS: 4
BUILD_JOBS: 4
TEST_JOBS: 8 # experimentally derived to be a decent choice
- IMAGE_FAMILY: pg-ci-bullseye
+ IMAGE_FAMILY: pg-ci-bookworm
CCACHE_DIR: /tmp/ccache_dir
DEBUGINFOD_URLS: "https://debuginfod.debian.net"
@@ -451,12 +451,12 @@ task:
# To limit unnecessary work only run this once the normal linux test succeeds
depends_on:
- - Linux - Debian Bullseye
+ - Linux - Debian Bookworm
env:
CPUS: 4
BUILD_JOBS: 4
- IMAGE_FAMILY: pg-ci-bullseye
+ IMAGE_FAMILY: pg-ci-bookworm
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
--
2.43.0
On 13.05.24 12:57, Nazir Bilal Yavuz wrote:
Bookworm versions of the Debian CI images are available now [0]. The
patches to use these images are attached.'v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_16+.patch' patch can
be applied to both upstream and REL_16 and all of the tasks finish
successfully.'v1-0001-Upgrade-Debian-CI-images-to-Bookworm_REL_15.patch' patch can
be applied to REL_15 but it gives a compiler warning. The fix for this
warning is proposed here [1]. After the fix is applied, all of the
tasks finish successfully.Any kind of feedback would be appreciated.
These updates are very welcome and look straightforward enough.
I'm not sure what the backpatching expectations of this kind of thing
is. The history of this CI setup is relatively short, so this hasn't
been stressed too much. I see that we once backpatched the macOS
update, but that might have been all.
If we start backpatching this kind of thing, then this will grow as a
job over time. We'll have 5 or 6 branches to keep up to date, with
several operating systems. And once in a while we'll have to make
additional changes like this warning fix you mention here. I'm not sure
how much we want to take this on. Is there ongoing value in the CI
setup in backbranches?
With these patches, we could do either of the following:
1) We update only master and only after it branches for PG18. (The
update is a "new feature".)
2) We update only master but do it now. (This gives us the most amount
of buffer time before the next release.)
3) We update master and PG16 now. We ignore PG15.
4) We update master and PG16 now. We update PG15 whenever that warning
is fixed.
5) We update master, PG16, and PG15, but we hold all of them until the
warning in PG15 is fixed.
6) We update all of them now and let the warning in PG15 be fixed
independently.
Hi,
On 2024-05-24 16:17:37 +0200, Peter Eisentraut wrote:
I'm not sure what the backpatching expectations of this kind of thing is.
The history of this CI setup is relatively short, so this hasn't been
stressed too much. I see that we once backpatched the macOS update, but
that might have been all.
I've backpatched a few other changes too.
If we start backpatching this kind of thing, then this will grow as a job
over time. We'll have 5 or 6 branches to keep up to date, with several
operating systems. And once in a while we'll have to make additional
changes like this warning fix you mention here. I'm not sure how much we
want to take this on. Is there ongoing value in the CI setup in
backbranches?
I find it extremely useful to run CI on backbranches before
batckpatching. Enough so that I've thought about proposing backpatching CI all
the way.
I don't think it's that much work to fix this kind of thing in the
backbranches. We don't need to backpatch new tasks or such. Just enough stuff
to keep e.g. the base image the same - otherwise we end up running CI on
unsupported distros, which doesn't help anybody.
With these patches, we could do either of the following:
5) We update master, PG16, and PG15, but we hold all of them until the
warning in PG15 is fixed.
I think we should apply the fix in <= 15 - IMO it's a correct compiler
warning, what we do right now is wrong.
Greetings,
Andres Freund
Hi,
On 2024-05-24 10:30:00 -0700, Andres Freund wrote:
On 2024-05-24 16:17:37 +0200, Peter Eisentraut wrote:
I'm not sure what the backpatching expectations of this kind of thing is.
The history of this CI setup is relatively short, so this hasn't been
stressed too much. I see that we once backpatched the macOS update, but
that might have been all.I've backpatched a few other changes too.
If we start backpatching this kind of thing, then this will grow as a job
over time. We'll have 5 or 6 branches to keep up to date, with several
operating systems. And once in a while we'll have to make additional
changes like this warning fix you mention here. I'm not sure how much we
want to take this on. Is there ongoing value in the CI setup in
backbranches?I find it extremely useful to run CI on backbranches before
batckpatching. Enough so that I've thought about proposing backpatching CI all
the way.I don't think it's that much work to fix this kind of thing in the
backbranches. We don't need to backpatch new tasks or such. Just enough stuff
to keep e.g. the base image the same - otherwise we end up running CI on
unsupported distros, which doesn't help anybody.With these patches, we could do either of the following:
5) We update master, PG16, and PG15, but we hold all of them until the
warning in PG15 is fixed.I think we should apply the fix in <= 15 - IMO it's a correct compiler
warning, what we do right now is wrong.
I've now applied the guc fix to all branches and the CI changes to 15+.
Thanks Bilal!
Greetings,
Andres
Hi,
On 2024-07-15 09:43:26 -0700, Andres Freund wrote:
I've now applied the guc fix to all branches and the CI changes to 15+.
Ugh. I see that this fails on master, because of
commit 0c3930d0768
Author: Peter Eisentraut <peter@eisentraut.org>
Date: 2024-07-01 07:30:38 +0200
Apply COPT to CXXFLAGS as well
I hadn't seen that because of an independent failure (the macos stuff, I'll
send an email about it in a bit).
Not sure what the best real fix here is, this is outside of our code. I'm
inclined to just disable llvm for the compiler warning task for now.
Greetings,
Andres Freund
Hi,
On 2024-07-15 11:30:59 -0700, Andres Freund wrote:
On 2024-07-15 09:43:26 -0700, Andres Freund wrote:
I've now applied the guc fix to all branches and the CI changes to 15+.
Ugh. I see that this fails on master, because of
commit 0c3930d0768
Author: Peter Eisentraut <peter@eisentraut.org>
Date: 2024-07-01 07:30:38 +0200Apply COPT to CXXFLAGS as well
I hadn't seen that because of an independent failure (the macos stuff, I'll
send an email about it in a bit).Not sure what the best real fix here is, this is outside of our code. I'm
inclined to just disable llvm for the compiler warning task for now.
Oh - there's a better fix: Turns out bookworm does have llvm 16, where the
warning has been fixed. Upgrading the CI image to install llvm 16 should fix
this. Any arguments against that approach?
Greetings,
Andres
Hi,
On 2024-07-15 12:37:54 -0700, Andres Freund wrote:
On 2024-07-15 11:30:59 -0700, Andres Freund wrote:
On 2024-07-15 09:43:26 -0700, Andres Freund wrote:
I've now applied the guc fix to all branches and the CI changes to 15+.
Ugh. I see that this fails on master, because of
commit 0c3930d0768
Author: Peter Eisentraut <peter@eisentraut.org>
Date: 2024-07-01 07:30:38 +0200Apply COPT to CXXFLAGS as well
I hadn't seen that because of an independent failure (the macos stuff, I'll
send an email about it in a bit).Not sure what the best real fix here is, this is outside of our code. I'm
inclined to just disable llvm for the compiler warning task for now.Oh - there's a better fix: Turns out bookworm does have llvm 16, where the
warning has been fixed. Upgrading the CI image to install llvm 16 should fix
this. Any arguments against that approach?
Specifically, something like the attached.
Due to the CI failure this is causing, I'm planning to apply this soon...
Arguably we could backpatch this, the warning are present on older branches
too. Except that they don't cause errors, as 0c3930d0768 is only on master.
Greetings,
Andres
Attachments:
v1-0001-ci-Use-newer-llvm-version-with-gcc.patchtext/x-diff; charset=us-asciiDownload
From e696e8ded144095c4c4435d2e729fa37ed9e6903 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 15 Jul 2024 14:08:38 -0700
Subject: [PATCH v1 1/4] ci: Use newer llvm version with gcc
gcc emits a warning for LLVM 14 code outside of our control. To avoid that,
update to a newer LLVM version. Do so both in the CompilerWarnings and normal
tasks - the latter don't fail, but the warnings make it more likely that we'd
miss other warnings.
The warning has been fixed in newer LLVM versions.
Discussion: https://postgr.es/m/20240715193754.awdxgrzurxnwwu2t@awork3.anarazel.de
---
.cirrus.tasks.yml | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index b14fe91cdb7..df476a1d65a 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -272,6 +272,8 @@ task:
LDFLAGS: $SANITIZER_FLAGS
CC: ccache gcc
CXX: ccache g++
+ # GCC emits a warning for llvm-14, so switch to a newer one.
+ LLVM_CONFIG: llvm-config-16
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
@@ -334,7 +336,7 @@ task:
\
${LINUX_CONFIGURE_FEATURES} \
\
- CLANG="ccache clang"
+ CLANG="ccache clang-16"
EOF
build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
upload_caches: ccache
@@ -689,6 +691,8 @@ task:
# different compilers to build with different combinations of dtrace on/off
# and cassert on/off.
+ # GCC emits a warning for llvm-14, so switch to a newer one.
+
# gcc, cassert off, dtrace on
always:
gcc_warning_script: |
@@ -696,7 +700,8 @@ task:
--cache gcc.cache \
--enable-dtrace \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang-16" \
+ LLVM_CONFIG=llvm-config-16
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -707,7 +712,8 @@ task:
--cache gcc.cache \
--enable-cassert \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang-16" \
+ LLVM_CONFIG=llvm-config-16
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -755,7 +761,7 @@ task:
--cache gcc.cache \
CC="ccache gcc" \
CXX="ccache g++" \
- CLANG="ccache clang"
+ CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} -C doc
--
2.45.2.746.g06e570c0df.dirty
Hi,
On 2024-07-15 14:35:14 -0700, Andres Freund wrote:
Specifically, something like the attached.
Due to the CI failure this is causing, I'm planning to apply this soon...
Arguably we could backpatch this, the warning are present on older branches
too. Except that they don't cause errors, as 0c3930d0768 is only on master.
Here's a v2, to address two things:
- there was an error in the docs build, because LLVM_CONFIG changed
- there were also deprecation warnings in headerscheck/cpluspluscheck
So I just made the change apply a bit more widely.
Greetings,
Andres Freund
Attachments:
v2-0001-ci-Use-newer-llvm-version-with-gcc.patchtext/x-diff; charset=us-asciiDownload
From a331747fd237c3b859b1da24ffc7768067c18d98 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 15 Jul 2024 14:08:38 -0700
Subject: [PATCH v2 1/3] ci: Use newer llvm version with gcc
gcc emits a warning for LLVM 14 code outside of our control. To avoid that,
update to a newer LLVM version. Do so both in the CompilerWarnings and normal
tasks - the latter don't fail, but the warnings make it more likely that we'd
miss other warnings.
The warning has been fixed in newer LLVM versions.
Discussion: https://postgr.es/m/20240715193754.awdxgrzurxnwwu2t@awork3.anarazel.de
---
.cirrus.tasks.yml | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index b14fe91cdb7..99ca74d5133 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -272,6 +272,8 @@ task:
LDFLAGS: $SANITIZER_FLAGS
CC: ccache gcc
CXX: ccache g++
+ # GCC emits a warning for llvm-14, so switch to a newer one.
+ LLVM_CONFIG: llvm-config-16
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
@@ -334,7 +336,7 @@ task:
\
${LINUX_CONFIGURE_FEATURES} \
\
- CLANG="ccache clang"
+ CLANG="ccache clang-16"
EOF
build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
upload_caches: ccache
@@ -661,6 +663,9 @@ task:
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
+ # GCC emits a warning for llvm-14, so switch to a newer one.
+ LLVM_CONFIG: llvm-config-16
+
<<: *linux_task_template
sysinfo_script: |
@@ -696,7 +701,7 @@ task:
--cache gcc.cache \
--enable-dtrace \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -707,7 +712,7 @@ task:
--cache gcc.cache \
--enable-cassert \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -717,7 +722,7 @@ task:
time ./configure \
--cache clang.cache \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
+ CC="ccache clang" CXX="ccache clang++-16" CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -729,7 +734,7 @@ task:
--enable-cassert \
--enable-dtrace \
${LINUX_CONFIGURE_FEATURES} \
- CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
+ CC="ccache clang" CXX="ccache clang++-16" CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
@@ -753,9 +758,7 @@ task:
docs_build_script: |
time ./configure \
--cache gcc.cache \
- CC="ccache gcc" \
- CXX="ccache g++" \
- CLANG="ccache clang"
+ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang-16"
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} -C doc
@@ -774,7 +777,7 @@ task:
${LINUX_CONFIGURE_FEATURES} \
--without-icu \
--quiet \
- CC="gcc" CXX"=g++" CLANG="clang"
+ CC="gcc" CXX"=g++" CLANG="clang-16"
make -s -j${BUILD_JOBS} clean
time make -s headerscheck EXTRAFLAGS='-fmax-errors=10'
headers_cpluspluscheck_script: |
--
2.45.2.746.g06e570c0df.dirty