ci: Allow running mingw tests by default via environment variable
Hi,
We have CI support for mingw, but don't run the task by default, as it eats up
precious CI credits. However, for cfbot we are using custom compute resources
(currently donated by google), so we can afford to run the mingw tests. Right
now that'd require cfbot to patch .cirrus.tasks.yml.
While one can manually trigger manual task in one one's own repo, most won't
have the permissions to do so for cfbot.
I propose that we instead run the task automatically if
$REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
configuration.
Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
configuration, the set of conditional expressions supported is very
simplistic.
To deal with that, I extended .cirrus.star to compute the required environment
variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
set to 'automatic', if not it's 'manual'.
We've also talked in other threads about adding CI support for
1) windows, building with visual studio
2) linux, with musl libc
3) free/netbsd
That becomes more enticing, if we can enable them by default on cfbot but not
elsewhere. With this change, it'd be easy to add further variables to control
such future tasks.
I also attached a second commit, that makes the "code" dealing with ci-os-only
in .cirrus.tasks.yml simpler. While I think it does nicely simplify
.cirrus.tasks.yml, overall it adds lines, possibly making this not worth it.
I'm somewhat on the fence.
Thoughts?
On the code level, I thought if it'd be good to have a common prefix for all
the automatically set variables. Right now that's CI_, but I'm not at all
wedded to that.
Greetings,
Andres Freund
Attachments:
v2-0001-ci-Allow-running-mingw-tests-by-default-via-envir.patchtext/x-diff; charset=us-asciiDownload
From 5d26ecfedbcbc83b4cb6e41a34c1af9a3ab24cdb Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v2 1/2] ci: Allow running mingw tests by default via
environment variable
---
.cirrus.yml | 12 ++++++++++--
.cirrus.star | 39 +++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 12 ++++++------
src/tools/ci/README | 12 ++++++++++++
4 files changed, 63 insertions(+), 12 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..7a164bb00de 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,26 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # See definition of mingw task in .cirrus.tasks.yml
+ if env.get("REPO_MINGW_TRIGGER_BY_DEFAULT") in ['true', '1', 'yes']:
+ cenv['CI_MINGW_TRIGGER_TYPE'] = 'automatic'
+ else:
+ cenv['CI_MINGW_TRIGGER_TYPE'] = 'manual'
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index a2388cd5036..3446d9d1ba4 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -583,13 +583,13 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # Due to resource constraints we don't run this task by default, unless the
+ # the environment variable REPO_MINGW_TRIGGER_BY_DEFAULT is true. This
+ # computation is done in .cirrus.star:compute_environment_vars()
+ trigger_type: $CI_MINGW_TRIGGER_TYPE
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 30ddd200c96..d39017b4eb7 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,15 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not run automatically by default, to avoid using up CI credits
+too quickly. This can be changed on the repository level, e.g. when custom
+compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_MINGW_TRIGGER_BY_DEFAULT - if set to true, the mingw test is run
+ automatically
--
2.44.0.279.g3bd955d269
v2-0002-ci-simplify-ci-os-only-handling.patchtext/x-diff; charset=us-asciiDownload
From b4278f5b5f013dbe4eb1cec0e996f95bd2bfc6e4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v2 2/2] ci: simplify ci-os-only handling
---
.cirrus.star | 28 +++++++++++++++++++++++++++-
.cirrus.tasks.yml | 17 ++++++++---------
2 files changed, 35 insertions(+), 10 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 7a164bb00de..685be056e3c 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -66,6 +66,32 @@ def main():
def compute_environment_vars():
cenv = {}
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ # We want to disable SanityCheck if testing just a specific OS. This
+ # shortens push-wait-for-ci cycle time a bit when debugging operating
+ # system specific failures. Just treating it as an OS in that case
+ # suffices.
+
+ operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'mingw',
+ 'sanitycheck', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+
+
# See definition of mingw task in .cirrus.tasks.yml
if env.get("REPO_MINGW_TRIGGER_BY_DEFAULT") in ['true', '1', 'yes']:
cenv['CI_MINGW_TRIGGER_TYPE'] = 'automatic'
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 3446d9d1ba4..aa3d90f5e0d 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -59,7 +59,7 @@ task:
# push-wait-for-ci cycle time a bit when debugging operating system specific
# failures. Uses skip instead of only_if, as cirrus otherwise warns about
# only_if conditions not matching.
- skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+ skip: $CI_SANITYCHECK_ENABLED == false
env:
CPUS: 4
@@ -141,7 +141,7 @@ task:
<<: *freebsd_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -279,7 +279,7 @@ task:
<<: *linux_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -436,7 +436,7 @@ task:
<<: *macos_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -548,7 +548,7 @@ task:
<<: *windows_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
setup_additional_packages_script: |
REM choco install -y --no-progress ...
@@ -589,7 +589,7 @@ task:
trigger_type: $CI_MINGW_TRIGGER_TYPE
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CI_MINGW_ENABLED
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -644,10 +644,9 @@ task:
# To limit unnecessary work only run this once the SanityCheck
# succeeds. This is particularly important for this task as we intentionally
- # use always: to continue after failures. Task that did not run count as a
- # success, so we need to recheck SanityChecks's condition here ...
+ # use always: to continue after failures.
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
env:
CPUS: 4
--
2.44.0.279.g3bd955d269
Hi,
Thank you for working on this!
On Sat, 13 Apr 2024 at 05:12, Andres Freund <andres@anarazel.de> wrote:
Hi,
We have CI support for mingw, but don't run the task by default, as it eats up
precious CI credits. However, for cfbot we are using custom compute resources
(currently donated by google), so we can afford to run the mingw tests. Right
now that'd require cfbot to patch .cirrus.tasks.yml.
And I think mingw ends up not running most of the time. +1 to running
it as default at least on cfbot. Also, this gives people a chance to
run mingw as default on their personal repositories (which I would
like to run).
While one can manually trigger manual task in one one's own repo, most won't
have the permissions to do so for cfbot.I propose that we instead run the task automatically if
$REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
configuration.Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
configuration, the set of conditional expressions supported is very
simplistic.To deal with that, I extended .cirrus.star to compute the required environment
variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
set to 'automatic', if not it's 'manual'.
Changes look good to me. My only complaint could be using only 'true'
for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
but this is not important.
We've also talked in other threads about adding CI support for
1) windows, building with visual studio
2) linux, with musl libc
3) free/netbsdThat becomes more enticing, if we can enable them by default on cfbot but not
elsewhere. With this change, it'd be easy to add further variables to control
such future tasks.
I agree.
I also attached a second commit, that makes the "code" dealing with ci-os-only
in .cirrus.tasks.yml simpler. While I think it does nicely simplify
.cirrus.tasks.yml, overall it adds lines, possibly making this not worth it.
I'm somewhat on the fence.Thoughts?
Although it adds more lines, this makes the .cirrus.tasks.yml file
more readable and understandable which is more important in my
opinion.
On the code level, I thought if it'd be good to have a common prefix for all
the automatically set variables. Right now that's CI_, but I'm not at all
wedded to that.
I agree with your thoughts and CI_ prefix.
I tested both patches and they work as expected.
--
Regards,
Nazir Bilal Yavuz
Microsoft
On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
On Sat, 13 Apr 2024 at 05:12, Andres Freund <andres@anarazel.de> wrote:
I propose that we instead run the task automatically if
$REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
configuration.Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
configuration, the set of conditional expressions supported is very
simplistic.To deal with that, I extended .cirrus.star to compute the required environment
variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
set to 'automatic', if not it's 'manual'.Changes look good to me. My only complaint could be using only 'true'
for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
but this is not important.
Here's a generalised version of 0001. If you put this in your
repository settings on Cirrus's website:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
... then it defines:
CI_TRIGGER_TYPE_MINGW=automatic
CI_TRIGGER_TYPE_NETBSD=automatic
CI_TRIGGER_TYPE_OPENBSD=manual
The set of tasks that get this treatment is defined by this line in
.cirrus.star:
default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
Then the individual tasks in .cirrus.tasks.yml use those variables:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
Unfortunately the funky Starlark language doesn't seem to have regular
expressions, so it's just searching for those substrings without
contemplating delimiters. That doesn't seem to be a problem at this
scale...
(I didn't look at the second patch today.)
Attachments:
v3-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patchtext/x-patch; charset=US-ASCII; name=v3-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patchDownload
From 74f4fb7a5af0b53eb4caf7de0ee186ffaaf6ee20 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v3] ci: Per-repo triggers for MinGW, NetBSD, OpenBSD.
We're not ready to trigger these tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 15 +++++++-------
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 75 insertions(+), 13 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The following tasks are manually triggered by default because they
+ # might use too many resources for users of free Cirrus credits, but you can
+ # trigger them automatically by naming them in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+ # Settings" on Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..f5ff0ccf222 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -218,7 +218,6 @@ task:
task:
depends_on: SanityCheck
- trigger_type: manual
env:
# Below are experimentally derived to be a decent choice.
@@ -236,6 +235,8 @@ task:
matrix:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
env:
OS_NAME: netbsd
@@ -253,6 +254,8 @@ task:
<<: *netbsd_task_template
- name: OpenBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_OPENBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
env:
OS_NAME: openbsd
@@ -706,13 +709,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
--
2.48.1
Hi,
On Tue, 4 Mar 2025 at 03:28, Thomas Munro <thomas.munro@gmail.com> wrote:
On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
On Sat, 13 Apr 2024 at 05:12, Andres Freund <andres@anarazel.de> wrote:
I propose that we instead run the task automatically if
$REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
configuration.Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
configuration, the set of conditional expressions supported is very
simplistic.To deal with that, I extended .cirrus.star to compute the required environment
variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
set to 'automatic', if not it's 'manual'.Changes look good to me. My only complaint could be using only 'true'
for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
but this is not important.Here's a generalised version of 0001. If you put this in your
repository settings on Cirrus's website:
Thank you for working on this!
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
... then it defines:
CI_TRIGGER_TYPE_MINGW=automatic
CI_TRIGGER_TYPE_NETBSD=automatic
CI_TRIGGER_TYPE_OPENBSD=manualThe set of tasks that get this treatment is defined by this line in
.cirrus.star:default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
Then the individual tasks in .cirrus.tasks.yml use those variables:
- name: NetBSD - Meson + # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star + trigger_type: $CI_TRIGGER_TYPE_NETBSD
I confirm that this works as expected and LGTM.
Unfortunately the funky Starlark language doesn't seem to have regular
expressions, so it's just searching for those substrings without
contemplating delimiters. That doesn't seem to be a problem at this
scale...
You can load 're' and use it as in the v2-0002 but I think what you
did is good enough.
I rebased Andres' v2-0002 on top of recent changes and wrote a small
commit message. v4 is attached.
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
v4-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patchtext/x-patch; charset=US-ASCII; name=v4-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patchDownload
From 566a729bf13a5e5c782ea367780cce68ae839b92 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v4 1/2] ci: Per-repo triggers for MinGW, NetBSD, OpenBSD.
We're not ready to trigger these tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 15 +++++++-------
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 75 insertions(+), 13 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The following tasks are manually triggered by default because they
+ # might use too many resources for users of free Cirrus credits, but you can
+ # trigger them automatically by naming them in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+ # Settings" on Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..f5ff0ccf222 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -218,7 +218,6 @@ task:
task:
depends_on: SanityCheck
- trigger_type: manual
env:
# Below are experimentally derived to be a decent choice.
@@ -236,6 +235,8 @@ task:
matrix:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
env:
OS_NAME: netbsd
@@ -253,6 +254,8 @@ task:
<<: *netbsd_task_template
- name: OpenBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_OPENBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
env:
OS_NAME: openbsd
@@ -706,13 +709,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
--
2.47.2
v4-0002-ci-Simplify-ci-os-only-handling.patchtext/x-patch; charset=US-ASCII; name=v4-0002-ci-Simplify-ci-os-only-handling.patchDownload
From f83dc32773290bfd5ccdade259d810adf2cfc456 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v4 2/2] ci: Simplify ci-os-only handling
Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
.cirrus.star | 31 ++++++++++++++++++++++++++++++-
.cirrus.tasks.yml | 21 ++++++++++-----------
2 files changed, 40 insertions(+), 12 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 218fce887b5..83e93e48d5f 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -66,6 +66,7 @@ def main():
def compute_environment_vars():
cenv = {}
+ ###
# The following tasks are manually triggered by default because they
# might use too many resources for users of free Cirrus credits, but you can
# trigger them automatically by naming them in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
else:
value = 'manual'
cenv[name] = value
+ ###
+
+ ###
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ # We want to disable SanityCheck if testing just a specific OS. This
+ # shortens push-wait-for-ci cycle time a bit when debugging operating
+ # system specific failures. Just treating it as an OS in that case
+ # suffices.
+
+ operating_systems = ['freebsd', 'netbsd', 'openbsd', 'linux', 'macos',
+ 'windows', 'mingw', 'sanitycheck', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+ ###
return cenv
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index f5ff0ccf222..fbaf074b13c 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
# push-wait-for-ci cycle time a bit when debugging operating system specific
# failures. Uses skip instead of only_if, as cirrus otherwise warns about
# only_if conditions not matching.
- skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+ skip: $CI_SANITYCHECK_ENABLED == false
env:
CPUS: 4
@@ -145,7 +145,7 @@ task:
<<: *freebsd_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -237,7 +237,7 @@ task:
- name: NetBSD - Meson
# See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
trigger_type: $CI_TRIGGER_TYPE_NETBSD
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
+ only_if: $CI_NETBSD_ENABLED
env:
OS_NAME: netbsd
IMAGE_FAMILY: pg-ci-netbsd-postgres
@@ -256,7 +256,7 @@ task:
- name: OpenBSD - Meson
# See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
trigger_type: $CI_TRIGGER_TYPE_OPENBSD
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
+ only_if: $CI_OPENBSD_ENABLED
env:
OS_NAME: openbsd
IMAGE_FAMILY: pg-ci-openbsd-postgres
@@ -395,7 +395,7 @@ task:
<<: *linux_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -568,7 +568,7 @@ task:
<<: *macos_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -674,7 +674,7 @@ task:
<<: *windows_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
setup_additional_packages_script: |
REM choco install -y --no-progress ...
@@ -713,7 +713,7 @@ task:
trigger_type: $CI_TRIGGER_TYPE_MINGW
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CI_MINGW_ENABLED
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -767,10 +767,9 @@ task:
# To limit unnecessary work only run this once the SanityCheck
# succeeds. This is particularly important for this task as we intentionally
- # use always: to continue after failures. Task that did not run count as a
- # success, so we need to recheck SanityChecks's condition here ...
+ # use always: to continue after failures.
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
env:
CPUS: 4
--
2.47.2
Hi,
On 2025-03-04 13:27:29 +1300, Thomas Munro wrote:
On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
On Sat, 13 Apr 2024 at 05:12, Andres Freund <andres@anarazel.de> wrote:
I propose that we instead run the task automatically if
$REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
configuration.Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
configuration, the set of conditional expressions supported is very
simplistic.To deal with that, I extended .cirrus.star to compute the required environment
variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
set to 'automatic', if not it's 'manual'.Changes look good to me. My only complaint could be using only 'true'
for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
but this is not important.Here's a generalised version of 0001. If you put this in your
repository settings on Cirrus's website:REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
... then it defines:
CI_TRIGGER_TYPE_MINGW=automatic
CI_TRIGGER_TYPE_NETBSD=automatic
CI_TRIGGER_TYPE_OPENBSD=manual
Nice!
The set of tasks that get this treatment is defined by this line in
.cirrus.star:default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
Then the individual tasks in .cirrus.tasks.yml use those variables:
- name: NetBSD - Meson + # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star + trigger_type: $CI_TRIGGER_TYPE_NETBSDUnfortunately the funky Starlark language doesn't seem to have regular
expressions, so it's just searching for those substrings without
contemplating delimiters. That doesn't seem to be a problem at this
scale...
FWIW, It does:
https://cirrus-ci.org/guide/programming-tasks/#re
https://github.com/qri-io/starlib/tree/master/re
Your edited version is looking good to me!
Greetings,
Andres Freund
Hi,
On 2025-03-04 11:53:30 +0300, Nazir Bilal Yavuz wrote:
I rebased Andres' v2-0002 on top of recent changes and wrote a small
commit message. v4 is attached.
Thanks for doing that, it does look a lot better after, I think!
Greetings,
Andres Freund
Hi,
I'm inclined to think we should apply to this to all branches with CI support,
not just master. It's kind of annoying to have CI infrastructure changes like
this that differ between branches. Thoughts?
Greetings,
Andres Freund
Hi,
On Wed, 5 Mar 2025 at 18:51, Andres Freund <andres@anarazel.de> wrote:
Hi,
I'm inclined to think we should apply to this to all branches with CI support,
not just master. It's kind of annoying to have CI infrastructure changes like
this that differ between branches. Thoughts?
I am sharing patches per branch until we come up with a better
solution. Some remarks:
REL_15: It doesn't have any manual tasks, so per repository trigger
patch (Thomas' patch) is omitted.
REL_16: Only Mingw task is manual.
REL_17: Only Mingw task is manual.
While rebasing them, I realized that we use
'REPO_CI_AUTOMATIC_TRIGGER_TASKS' for all branches. So, we don't have
per branch 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' configuration option. Is
that a problem?
--
Regards,
Nazir Bilal Yavuz
Microsoft
Attachments:
REL_15_v5-0001-ci-Simplify-ci-os-only-handling.patchtext/x-patch; charset=US-ASCII; name=REL_15_v5-0001-ci-Simplify-ci-os-only-handling.patchDownload
From c872f537375416289d3fdc302716403266899c1f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5] ci: Simplify ci-os-only handling
Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
.cirrus.star | 52 +++++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 10 ++++-----
.cirrus.yml | 12 +++++++++--
3 files changed, 63 insertions(+), 11 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index d2d6ceca207..3264ce014c5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,39 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 995ae7cefb9..ecf683b7b54 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -47,7 +47,7 @@ task:
<<: *freebsd_task_template
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -153,7 +153,7 @@ task:
<<: *linux_task_template
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -239,7 +239,7 @@ task:
<<: *macos_task_template
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -389,7 +389,7 @@ task:
<<: *windows_task_template
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
sysinfo_script: |
chcp
@@ -476,7 +476,7 @@ task:
# task that did not run, count as a success, so we need to recheck Linux'
# condition here ...
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
<<: *linux_task_template
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
--
2.49.0
REL_16_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchtext/x-patch; charset=US-ASCII; name=REL_16_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchDownload
From b9276d7cc5713008a986c68d2f00b5bf23cb50fd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks
We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 10 ++++-----
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 71 insertions(+), 12 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index d2d6ceca207..ff31d191565 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The mingw task is manually triggered by default because it might use too
+ # many resources for users of free Cirrus credits, but you can trigger it
+ # automatically by naming it in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw" under "Repository Settings" on
+ # Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index d12360aa356..54c6f3a65ee 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -591,13 +591,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 30ddd200c96..771243c4da0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
--
2.49.0
REL_16_v5-0002-ci-Simplify-ci-os-only-handling.patchtext/x-patch; charset=US-ASCII; name=REL_16_v5-0002-ci-Simplify-ci-os-only-handling.patchDownload
From 5975233e10c8bb4e87049cccb13c98cbdc342532 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling
Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
.cirrus.star | 31 ++++++++++++++++++++++++++++++-
.cirrus.tasks.yml | 17 ++++++++---------
2 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index ff31d191565..06666e55a4b 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -66,6 +66,7 @@ def main():
def compute_environment_vars():
cenv = {}
+ ###
# The mingw task is manually triggered by default because it might use too
# many resources for users of free Cirrus credits, but you can trigger it
# automatically by naming it in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
else:
value = 'manual'
cenv[name] = value
+ ###
+
+ ###
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ # We want to disable SanityCheck if testing just a specific OS. This
+ # shortens push-wait-for-ci cycle time a bit when debugging operating
+ # system specific failures. Just treating it as an OS in that case
+ # suffices.
+
+ operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'mingw',
+ 'sanitycheck', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+ ###
return cenv
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 54c6f3a65ee..59d69acf2a7 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
# push-wait-for-ci cycle time a bit when debugging operating system specific
# failures. Uses skip instead of only_if, as cirrus otherwise warns about
# only_if conditions not matching.
- skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+ skip: $CI_SANITYCHECK_ENABLED == false
env:
CPUS: 4
@@ -143,7 +143,7 @@ task:
<<: *freebsd_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -280,7 +280,7 @@ task:
<<: *linux_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -449,7 +449,7 @@ task:
<<: *macos_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -556,7 +556,7 @@ task:
<<: *windows_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
setup_additional_packages_script: |
REM choco install -y --no-progress ...
@@ -595,7 +595,7 @@ task:
trigger_type: $CI_TRIGGER_TYPE_MINGW
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CI_MINGW_ENABLED
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -649,10 +649,9 @@ task:
# To limit unnecessary work only run this once the SanityCheck
# succeeds. This is particularly important for this task as we intentionally
- # use always: to continue after failures. Task that did not run count as a
- # success, so we need to recheck SanityChecks's condition here ...
+ # use always: to continue after failures.
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
env:
CPUS: 4
--
2.49.0
REL_17_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchtext/x-patch; charset=US-ASCII; name=REL_17_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchDownload
From 340782a5e770b85d05f4e583d6dc1f12b1888a96 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks
We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 10 ++++-----
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 71 insertions(+), 12 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..cccdbd1bfbb 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The mingw task is manually triggered by default because it might use too
+ # many resources for users of free Cirrus credits, but you can trigger it
+ # automatically by naming it in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw" under "Repository Settings" on
+ # Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index aa7c7dcd34d..af33716a004 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -595,13 +595,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 30ddd200c96..771243c4da0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
--
2.49.0
REL_17_v5-0002-ci-Simplify-ci-os-only-handling.patchtext/x-patch; charset=US-ASCII; name=REL_17_v5-0002-ci-Simplify-ci-os-only-handling.patchDownload
From 903489cc02b40f44e3672dda2fdbcd9dd7147056 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling
Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
.cirrus.star | 31 ++++++++++++++++++++++++++++++-
.cirrus.tasks.yml | 17 ++++++++---------
2 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index cccdbd1bfbb..c26cf52d10c 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -66,6 +66,7 @@ def main():
def compute_environment_vars():
cenv = {}
+ ###
# The mingw task is manually triggered by default because it might use too
# many resources for users of free Cirrus credits, but you can trigger it
# automatically by naming it in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
else:
value = 'manual'
cenv[name] = value
+ ###
+
+ ###
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ # We want to disable SanityCheck if testing just a specific OS. This
+ # shortens push-wait-for-ci cycle time a bit when debugging operating
+ # system specific failures. Just treating it as an OS in that case
+ # suffices.
+
+ operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'mingw',
+ 'sanitycheck', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+ ###
return cenv
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index af33716a004..93459e808f0 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
# push-wait-for-ci cycle time a bit when debugging operating system specific
# failures. Uses skip instead of only_if, as cirrus otherwise warns about
# only_if conditions not matching.
- skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+ skip: $CI_SANITYCHECK_ENABLED == false
env:
CPUS: 4
@@ -144,7 +144,7 @@ task:
<<: *freebsd_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -282,7 +282,7 @@ task:
<<: *linux_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -453,7 +453,7 @@ task:
<<: *macos_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -560,7 +560,7 @@ task:
<<: *windows_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
setup_additional_packages_script: |
REM choco install -y --no-progress ...
@@ -599,7 +599,7 @@ task:
trigger_type: $CI_TRIGGER_TYPE_MINGW
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CI_MINGW_ENABLED
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -653,10 +653,9 @@ task:
# To limit unnecessary work only run this once the SanityCheck
# succeeds. This is particularly important for this task as we intentionally
- # use always: to continue after failures. Task that did not run count as a
- # success, so we need to recheck SanityChecks's condition here ...
+ # use always: to continue after failures.
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
env:
CPUS: 4
--
2.49.0
Upstream_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchtext/x-patch; charset=US-ASCII; name=Upstream_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patchDownload
From c50fdccc3b3e1aee9f51c576da5942bcc1732e14 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks
We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them. This avoids using too many credits by default.
With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:
REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"
This will enable cfbot to turn them on by default when running tests for
the Commitfest app. It is not subject to free credit limits as it runs
on credits donated by Google.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
.cirrus.star | 50 +++++++++++++++++++++++++++++++++++++++++----
.cirrus.tasks.yml | 15 +++++++-------
.cirrus.yml | 12 +++++++++--
src/tools/ci/README | 11 ++++++++++
4 files changed, 75 insertions(+), 13 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
def main():
@@ -18,19 +18,36 @@ def main():
1) the contents of .cirrus.yml
- 2) if defined, the contents of the file referenced by the, repository
+ 2) computed environment variables
+
+ 3) if defined, the contents of the file referenced by the, repository
level, REPO_CI_CONFIG_GIT_URL variable (see
https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
format)
- 3) .cirrus.tasks.yml
+ 4) .cirrus.tasks.yml
"""
output = ""
# 1) is evaluated implicitly
+
# Add 2)
+ additional_env = compute_environment_vars()
+ env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+ output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+ # Add 3)
repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
if repo_config_url != None:
print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
else:
output += "\n# REPO_CI_CONFIG_URL was not set\n"
- # Add 3)
+
+ # Add 4)
output += config_from(".cirrus.tasks.yml")
+
return output
+def compute_environment_vars():
+ cenv = {}
+
+ # The following tasks are manually triggered by default because they
+ # might use too many resources for users of free Cirrus credits, but you can
+ # trigger them automatically by naming them in an environment variable eg
+ # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+ # Settings" on Cirrus CI's website.
+
+ default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+ repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+ for task in default_manual_trigger_tasks:
+ name = 'CI_TRIGGER_TYPE_' + task.upper()
+ if repo_ci_automatic_trigger_tasks.find(task) != -1:
+ value = 'automatic'
+ else:
+ value = 'manual'
+ cenv[name] = value
+
+ return cenv
+
+
def config_from(config_src):
"""return contents of config file `config_src`, surrounded by markers
indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 86a1fa9bbdb..0cf6ea67ba5 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -239,7 +239,6 @@ task:
task:
depends_on: SanityCheck
- trigger_type: manual
env:
# Below are experimentally derived to be a decent choice.
@@ -257,6 +256,8 @@ task:
matrix:
- name: NetBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_NETBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
env:
OS_NAME: netbsd
@@ -274,6 +275,8 @@ task:
<<: *netbsd_task_template
- name: OpenBSD - Meson
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+ trigger_type: $CI_TRIGGER_TYPE_OPENBSD
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
env:
OS_NAME: openbsd
@@ -743,13 +746,11 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64 - Meson
- # due to resource constraints we don't run this task by default for now
- trigger_type: manual
- # worth using only_if despite being manual, otherwise this task will show up
- # when e.g. ci-os-only: linux is used.
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
- # otherwise it'll be sorted before other tasks
+ # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+ trigger_type: $CI_TRIGGER_TYPE_MINGW
+
depends_on: SanityCheck
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
#
# 1) the contents of this file
#
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+# Used to enable/disable tasks based on the execution environment. See
+# .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
# level, REPO_CI_CONFIG_GIT_URL variable (see
# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
# format)
#
-# 3) .cirrus.tasks.yml
+# This allows running tasks in a different execution environment than the
+# default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
#
# This composition is done by .cirrus.star
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
Custom compute resources can be provided using
- https://cirrus-ci.org/guide/supported-computing-services/
- https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
--
2.49.0
Upstream_v5-0002-ci-Simplify-ci-os-only-handling.patchtext/x-patch; charset=US-ASCII; name=Upstream_v5-0002-ci-Simplify-ci-os-only-handling.patchDownload
From bcd64a5fb6d6721fceba1684a3bfc0b46c7fd9d5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling
Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
.cirrus.star | 31 ++++++++++++++++++++++++++++++-
.cirrus.tasks.yml | 21 ++++++++++-----------
2 files changed, 40 insertions(+), 12 deletions(-)
diff --git a/.cirrus.star b/.cirrus.star
index 218fce887b5..83e93e48d5f 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
See also .cirrus.yml and src/tools/ci/README
"""
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
def main():
@@ -66,6 +66,7 @@ def main():
def compute_environment_vars():
cenv = {}
+ ###
# The following tasks are manually triggered by default because they
# might use too many resources for users of free Cirrus credits, but you can
# trigger them automatically by naming them in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
else:
value = 'manual'
cenv[name] = value
+ ###
+
+ ###
+ # Parse "ci-os-only:" tag in commit message and set
+ # CI_{$OS}_ENABLED variable for each OS
+
+ # We want to disable SanityCheck if testing just a specific OS. This
+ # shortens push-wait-for-ci cycle time a bit when debugging operating
+ # system specific failures. Just treating it as an OS in that case
+ # suffices.
+
+ operating_systems = ['freebsd', 'netbsd', 'openbsd', 'linux', 'macos',
+ 'windows', 'mingw', 'sanitycheck', 'compilerwarnings']
+ commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+ match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+ # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+ m = re.match(match_re, commit_message)
+ if m and len(m) > 0:
+ os_only = m[0][2]
+ os_only_list = re.split(r'[, ]+', os_only)
+ else:
+ os_only_list = operating_systems
+
+ for os in operating_systems:
+ os_enabled = os in os_only_list
+ cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+ ###
return cenv
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 0cf6ea67ba5..744b1255ed9 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -72,7 +72,7 @@ task:
# push-wait-for-ci cycle time a bit when debugging operating system specific
# failures. Uses skip instead of only_if, as cirrus otherwise warns about
# only_if conditions not matching.
- skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+ skip: $CI_SANITYCHECK_ENABLED == false
env:
CPUS: 4
@@ -167,7 +167,7 @@ task:
<<: *freebsd_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ only_if: $CI_FREEBSD_ENABLED
sysinfo_script: |
id
@@ -258,7 +258,7 @@ task:
- name: NetBSD - Meson
# See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
trigger_type: $CI_TRIGGER_TYPE_NETBSD
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
+ only_if: $CI_NETBSD_ENABLED
env:
OS_NAME: netbsd
IMAGE_FAMILY: pg-ci-netbsd-postgres
@@ -277,7 +277,7 @@ task:
- name: OpenBSD - Meson
# See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
trigger_type: $CI_TRIGGER_TYPE_OPENBSD
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
+ only_if: $CI_OPENBSD_ENABLED
env:
OS_NAME: openbsd
IMAGE_FAMILY: pg-ci-openbsd-postgres
@@ -417,7 +417,7 @@ task:
<<: *linux_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ only_if: $CI_LINUX_ENABLED
ccache_cache:
folder: ${CCACHE_DIR}
@@ -605,7 +605,7 @@ task:
<<: *macos_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ only_if: $CI_MACOS_ENABLED
sysinfo_script: |
id
@@ -711,7 +711,7 @@ task:
<<: *windows_task_template
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ only_if: $CI_WINDOWS_ENABLED
setup_additional_packages_script: |
REM choco install -y --no-progress ...
@@ -750,7 +750,7 @@ task:
trigger_type: $CI_TRIGGER_TYPE_MINGW
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CI_MINGW_ENABLED
env:
TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -804,10 +804,9 @@ task:
# To limit unnecessary work only run this once the SanityCheck
# succeeds. This is particularly important for this task as we intentionally
- # use always: to continue after failures. Task that did not run count as a
- # success, so we need to recheck SanityChecks's condition here ...
+ # use always: to continue after failures.
depends_on: SanityCheck
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+ only_if: $CI_COMPILERWARNINGS_ENABLED
env:
CPUS: 4
--
2.49.0
Hi,
On 2025-04-07 16:03:48 +0300, Nazir Bilal Yavuz wrote:
On Wed, 5 Mar 2025 at 18:51, Andres Freund <andres@anarazel.de> wrote:
I'm inclined to think we should apply to this to all branches with CI support,
not just master. It's kind of annoying to have CI infrastructure changes like
this that differ between branches. Thoughts?I am sharing patches per branch until we come up with a better
solution. Some remarks:REL_15: It doesn't have any manual tasks, so per repository trigger
patch (Thomas' patch) is omitted.
REL_16: Only Mingw task is manual.
REL_17: Only Mingw task is manual.While rebasing them, I realized that we use
'REPO_CI_AUTOMATIC_TRIGGER_TASKS' for all branches. So, we don't have
per branch 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' configuration option. Is
that a problem?
I think that's not a problem. I can't think of a case where one would want to
do this only for some of the branches. Does anybody disagree?
Greetings,
Andres Freund
On Tue, Apr 8, 2025 at 1:43 AM Andres Freund <andres@anarazel.de> wrote:
On 2025-04-07 16:03:48 +0300, Nazir Bilal Yavuz wrote:
On Wed, 5 Mar 2025 at 18:51, Andres Freund <andres@anarazel.de> wrote:
I'm inclined to think we should apply to this to all branches with CI support,
not just master. It's kind of annoying to have CI infrastructure changes like
this that differ between branches. Thoughts?I am sharing patches per branch until we come up with a better
solution. Some remarks:REL_15: It doesn't have any manual tasks, so per repository trigger
patch (Thomas' patch) is omitted.
REL_16: Only Mingw task is manual.
REL_17: Only Mingw task is manual.While rebasing them, I realized that we use
'REPO_CI_AUTOMATIC_TRIGGER_TASKS' for all branches. So, we don't have
per branch 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' configuration option. Is
that a problem?I think that's not a problem. I can't think of a case where one would want to
do this only for some of the branches. Does anybody disagree?
Nope.
I think we should push all this stuff and turn it on for cfbot, not
only for the benefit of new development but also for the v18
stabilisation period.
Is it safe to assume that CI changes come under the same code-freeze
exception as tests and docs? I would assume so, it's just scripting
glue in .cirrus.XXX files not affecting one bit of the software, but I
don't recall that we specifically clarified that before, so can
someone from the RMT please +1 this? Then I'll go ahead and push
these.
On Sat, Apr 12, 2025 at 08:18:35PM +1200, Thomas Munro wrote:
Is it safe to assume that CI changes come under the same code-freeze
exception as tests and docs? I would assume so, it's just scripting
glue in .cirrus.XXX files not affecting one bit of the software, but I
don't recall that we specifically clarified that before, so can
someone from the RMT please +1 this? Then I'll go ahead and push
these.
That seems reasonable to me. But to play devil's advocate, is there a
strong reason for doing this before July?
--
nathan
Hi,
On April 14, 2025 9:22:40 PM GMT+02:00, Nathan Bossart <nathandbossart@gmail.com> wrote:
On Sat, Apr 12, 2025 at 08:18:35PM +1200, Thomas Munro wrote:
Is it safe to assume that CI changes come under the same code-freeze
exception as tests and docs? I would assume so, it's just scripting
glue in .cirrus.XXX files not affecting one bit of the software, but I
don't recall that we specifically clarified that before, so can
someone from the RMT please +1 this? Then I'll go ahead and push
these.That seems reasonable to me. But to play devil's advocate, is there a
strong reason for doing this before July?
We get more coverage of the changes we merge until then.
Greetings,
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Mon, Apr 14, 2025 at 09:26:59PM +0200, Andres Freund wrote:
On April 14, 2025 9:22:40 PM GMT+02:00, Nathan Bossart <nathandbossart@gmail.com> wrote:
On Sat, Apr 12, 2025 at 08:18:35PM +1200, Thomas Munro wrote:
Is it safe to assume that CI changes come under the same code-freeze
exception as tests and docs? I would assume so, it's just scripting
glue in .cirrus.XXX files not affecting one bit of the software, but I
don't recall that we specifically clarified that before, so can
someone from the RMT please +1 this? Then I'll go ahead and push
these.That seems reasonable to me. But to play devil's advocate, is there a
strong reason for doing this before July?We get more coverage of the changes we merge until then.
Sure. To be clear, I think this change is fine to proceed, especially
given we're less than a week into feature freeze and have plenty of time to
stabilize any cfbot breakage. If anything, this change should _help_
stabilize v18. I do wonder if we should be more cautious about these kinds
of changes closer to release dates, but that's not a concern here.
--
nathan