Add '--ignore-errors' into pg_regress
Hi,
I want to add the '--ignore-errors' option into the pg_regress module.
I understand it can't be used in the regression or TAP tests. But such
option is useful to test a custom extension. A custom extension couldn't
pass all check-world tests and will be stopped at the end of first stage.
My use case of this feature is to run check-world, look for core files
and analyze diffs of failed (but ignored) tests. Maybe it is not
difficult to apply this patch before the test in CI script, but annoying
procedure.
I think this patch could be applied because it is trivial and can be
easy reverted if needed.
An example:
TEMP_CONFIG=/tmp/extra.config \
EXTRA_REGRESS_OPTS="--load-extension=aqo --ignore-errors
--schedule=src/test/regress/parallel_schedule" \
make check-world
Maybe I just don't know the right way?
--
regards,
Andrey Lepikhov
Postgres Professional
Attachments:
0001-Add-a-ignore-all-errors-option-into-pg_regress.patchtext/plain; charset=UTF-8; name=0001-Add-a-ignore-all-errors-option-into-pg_regress.patch; x-mac-creator=0; x-mac-type=0Download
From 3bd6861c7c0557effc37212dcf8e24cf15930627 Mon Sep 17 00:00:00 2001
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Date: Tue, 29 Jun 2021 07:20:18 +0300
Subject: [PATCH] Add a ignore all errors option into pg_regress
---
src/test/regress/pg_regress.c | 31 +++++++++++++++++++++++++------
1 file changed, 25 insertions(+), 6 deletions(-)
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 05296f7ee1..28a70984e6 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -88,6 +88,7 @@ static char *temp_instance = NULL;
static _stringlist *temp_configs = NULL;
static bool nolocale = false;
static bool use_existing = false;
+static bool ignore_errors = false;
static char *hostname = NULL;
static int port = -1;
static bool port_specified_by_user = false;
@@ -1849,12 +1850,17 @@ run_schedule(const char *schedule, test_start_function startfunc,
bool ignore = false;
_stringlist *sl;
- for (sl = ignorelist; sl != NULL; sl = sl->next)
+ if (ignore_errors)
+ ignore = true;
+ else
{
- if (strcmp(tests[i], sl->str) == 0)
+ for (sl = ignorelist; sl != NULL; sl = sl->next)
{
- ignore = true;
- break;
+ if (strcmp(tests[i], sl->str) == 0)
+ {
+ ignore = true;
+ break;
+ }
}
}
if (ignore)
@@ -1948,8 +1954,16 @@ run_single_test(const char *test, test_start_function startfunc,
if (differ)
{
- status(_("FAILED"));
- fail_count++;
+ if (ignore_errors)
+ {
+ status(_("failed (ignored)"));
+ fail_ignore_count++;
+ }
+ else
+ {
+ status(_("FAILED"));
+ fail_count++;
+ }
}
else
{
@@ -2102,6 +2116,7 @@ help(void)
printf(_(" (can be used multiple times to concatenate)\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
+ printf(_(" --ignore-errors ignore all errors during the tests\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_("\n"));
printf(_("Options for \"temp-instance\" mode:\n"));
@@ -2152,6 +2167,7 @@ regression_main(int argc, char *argv[],
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
{"make-testtablespace-dir", no_argument, NULL, 26},
+ {"ignore-errors", required_argument, NULL, 27},
{NULL, 0, NULL, 0}
};
@@ -2285,6 +2301,9 @@ regression_main(int argc, char *argv[],
case 26:
make_testtablespace_dir = true;
break;
+ case 27: /* Set ignore all errors parameter to true */
+ ignore_errors = true;
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
--
2.31.1
Andrey Lepikhov <a.lepikhov@postgrespro.ru> writes:
I want to add the '--ignore-errors' option into the pg_regress module.
I understand it can't be used in the regression or TAP tests. But such
option is useful to test a custom extension.
I'm really skeptical that this has any positive use. It seems more
likely to be a foot-gun.
Also, pg_regress will already complete all the tests in a particular
suite, and I'm not clear on why you wouldn't try to get (say) the core
suite passing before trying something else. If the core suite has got
problems it seems unlikely that you can learn much from other suites.
BTW, I wonder if you can't get much or all of the same effect
from "make -k check-world".
regards, tom lane
On 29/6/21 20:59, Tom Lane wrote:
Andrey Lepikhov <a.lepikhov@postgrespro.ru> writes:
BTW, I wonder if you can't get much or all of the same effect
from "make -k check-world".
Thank you, 'make -k' is suitable solution in such situation.
--
regards,
Andrey Lepikhov
Postgres Professional