Issue with markers in isolation tester? Or not?
Hello, everyone!
While stabilizing tests for [0]/messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz I have noticed unclear (and confusing in my
opinion) behavior of markers in the isolation tester.
I have attached a test with reproducer.
There are two permutations in the test:
permutation
after(before)
before
detach1
wakeup1
detach2
wakeup2
In that case, I see expected results:
step before: <... completed>
step after: <... completed>
But changing the order of steps slightly:
permutation
after(before)
wakeup1 <------- wakeup moved here
before
detach1
detach2
wakeup2
makes "after" to be completed before "before". Does that make sense?
In my cases, there are 2-steps of injection points for each of 3 sessions,
so, it is difficult to make it bullet-proof in that case without a lot of
additional injection points.
Best regards,
Mikhail.
[0]: /messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz
/messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz
Attachments:
v1-0001-test-Add-isolation-test-for-injection-points-mark.patchapplication/octet-stream; name=v1-0001-test-Add-isolation-test-for-injection-points-mark.patchDownload
From 94d899683907698430c1c095e74a6c55c2406168 Mon Sep 17 00:00:00 2001
From: nkey <michail.nikolaev@gmail.com>
Date: Mon, 13 Jan 2025 00:51:37 +0100
Subject: [PATCH v1] test: Add isolation test for injection points markers
Add a new isolation test case that verifies the behavior of injection points
with wait/wakeup operations across multiple sessions. The test ensures proper
synchronization when detaching and waking up injection points in different
orders.
---
src/test/modules/injection_points/Makefile | 2 +-
.../injection_points/expected/markers.out | 101 ++++++++++++++++++
src/test/modules/injection_points/meson.build | 1 +
.../injection_points/specs/markers.spec | 45 ++++++++
4 files changed, 148 insertions(+), 1 deletion(-)
create mode 100644 src/test/modules/injection_points/expected/markers.out
create mode 100644 src/test/modules/injection_points/specs/markers.spec
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 0753a9df58c..4f47f4a0e7e 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -13,7 +13,7 @@ PGFILEDESC = "injection_points - facility for injection points"
REGRESS = injection_points reindex_conc
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
-ISOLATION = basic inplace
+ISOLATION = basic inplace markers
TAP_TESTS = 1
diff --git a/src/test/modules/injection_points/expected/markers.out b/src/test/modules/injection_points/expected/markers.out
new file mode 100644
index 00000000000..136bbae8338
--- /dev/null
+++ b/src/test/modules/injection_points/expected/markers.out
@@ -0,0 +1,101 @@
+Parsed test spec with 3 sessions
+
+starting permutation: after before detach1 wakeup1 detach2 wakeup2
+injection_points_attach
+-----------------------
+
+(1 row)
+
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step after: SELECT injection_points_run('injection-points-wait-1'); <waiting ...>
+step before: SELECT injection_points_run('injection-points-wait-2'); <waiting ...>
+step detach1: SELECT injection_points_detach('injection-points-wait-1');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup1: SELECT injection_points_wakeup('injection-points-wait-1');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step detach2: SELECT injection_points_detach('injection-points-wait-2');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup2: SELECT injection_points_wakeup('injection-points-wait-2');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+step after: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+
+starting permutation: after wakeup1 before detach1 detach2 wakeup2
+injection_points_attach
+-----------------------
+
+(1 row)
+
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step after: SELECT injection_points_run('injection-points-wait-1'); <waiting ...>
+step wakeup1: SELECT injection_points_wakeup('injection-points-wait-1');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: SELECT injection_points_run('injection-points-wait-2'); <waiting ...>
+step detach1: SELECT injection_points_detach('injection-points-wait-1');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step detach2: SELECT injection_points_detach('injection-points-wait-2');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup2: SELECT injection_points_wakeup('injection-points-wait-2');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+step after: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 989b4db226b..3998bac852f 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -44,6 +44,7 @@ tests += {
'specs': [
'basic',
'inplace',
+ 'markers',
],
},
'tap': {
diff --git a/src/test/modules/injection_points/specs/markers.spec b/src/test/modules/injection_points/specs/markers.spec
new file mode 100644
index 00000000000..e9f56d7567b
--- /dev/null
+++ b/src/test/modules/injection_points/specs/markers.spec
@@ -0,0 +1,45 @@
+setup
+{
+ CREATE EXTENSION injection_points;
+}
+teardown
+{
+ DROP EXTENSION injection_points;
+}
+
+# Wait happens in the first session, wakeup in the second session.
+session s1
+setup {
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('injection-points-wait-1', 'wait');
+}
+step after { SELECT injection_points_run('injection-points-wait-1'); }
+
+session s2
+setup {
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('injection-points-wait-2', 'wait');
+}
+step before { SELECT injection_points_run('injection-points-wait-2'); }
+
+session s3
+step wakeup1 { SELECT injection_points_wakeup('injection-points-wait-1'); }
+step detach1 { SELECT injection_points_detach('injection-points-wait-1'); }
+step wakeup2 { SELECT injection_points_wakeup('injection-points-wait-2'); }
+step detach2 { SELECT injection_points_detach('injection-points-wait-2'); }
+
+permutation
+ after(before)
+ before
+ detach1
+ wakeup1
+ detach2
+ wakeup2
+
+permutation
+ after(before)
+ wakeup1
+ before
+ detach1
+ detach2
+ wakeup2
\ No newline at end of file
--
2.43.0
Hello!
In case if someone will look for a workaround - it is possible to use
"notices N" mark.\
Instead of
step before {
SELECT injection_points_run('injection-points-wait-2');
}
use something like:
step before {
DO $$
BEGIN
PERFORM injection_points_run('injection-points-wait-2');
RAISE NOTICE 'before is complete';
END $$
}
and then:
permutation
after(before notices 1)
before
detach1
wakeup1
detach2
wakeup2
permutation
after(before notices 1)
wakeup1
before
detach1
detach2
wakeup2
in such a case, both permutations report "before" to be completed before
"after", not after.
Best regards,
Michail.
Show quoted text
On Mon, Jan 13, 2025 at 02:02:00AM +0100, Michail Nikolaev wrote:
While stabilizing tests for [0] I have noticed unclear (and confusing in my
opinion) behavior of markers in the isolation tester.I have attached a test with reproducer.
There are two permutations in the test:
permutation
after(before)
before
detach1
wakeup1
detach2
wakeup2In that case, I see expected results:
step before: <... completed>
step after: <... completed>But changing the order of steps slightly:
permutation
after(before)
wakeup1 <------- wakeup moved here
before
detach1
detach2
wakeup2makes "after" to be completed before "before". Does that make sense?
Yes. I don't see a good reason for isolationtester to disregard that
"(before)" marker, so I think you've found a bug in isolationtester. How
might we fix it?
Hello, Noah!
Thanks for your attention!
It looks like the simplest solution is just to count the number of
not-yet-completed steps and check that value.
A patch with such changes (+ the same test + README update + commit
message) is attached.
Best regards,
Mikhail.
Attachments:
v2-0001-isolation-tester-Fix-synchronization-of-steps-wit.patchapplication/octet-stream; name=v2-0001-isolation-tester-Fix-synchronization-of-steps-wit.patchDownload
From 78caa0b89b2611e1a6a8e27d88761a6f5ae5ce8e Mon Sep 17 00:00:00 2001
From: nkey <michail.nikolaev@gmail.com>
Date: Tue, 14 Jan 2025 12:09:18 +0100
Subject: [PATCH v2] isolation tester: Fix synchronization of steps with
blockers
Fix a race condition in the isolation tester where steps with blockers could
complete in an unexpected order. Previously, step completion was determined
by checking if the blocking step was active in its session. This led to
inconsistent behavior when steps were ordered differently in the permutation.
The fix changes the completion logic to track the number of incomplete
instances of each step, allowing more precise control over step
synchronization. This ensures that a step with a blocker waits for all
instances of its blocking step to complete before proceeding, regardless
of the execution order in the permutation.
As a side effect, the 'used' boolean field in the Step struct was replaced
with 'non_complete' counter because there is not sense to keep both.
Test case added to verify proper synchronization behavior with injection
points in different orderings.
---
src/test/isolation/README | 6 +-
src/test/isolation/isolationtester.c | 16 +--
src/test/isolation/isolationtester.h | 2 +-
src/test/isolation/specparse.y | 2 +-
src/test/modules/injection_points/Makefile | 2 +-
.../injection_points/expected/markers.out | 101 ++++++++++++++++++
src/test/modules/injection_points/meson.build | 1 +
.../injection_points/specs/markers.spec | 45 ++++++++
8 files changed, 163 insertions(+), 12 deletions(-)
create mode 100644 src/test/modules/injection_points/expected/markers.out
create mode 100644 src/test/modules/injection_points/specs/markers.spec
diff --git a/src/test/isolation/README b/src/test/isolation/README
index 5818ca50038..33f929b200f 100644
--- a/src/test/isolation/README
+++ b/src/test/isolation/README
@@ -183,9 +183,9 @@ A marker consisting solely of a step name indicates that this step may
not be reported as completing until that other step has completed.
This allows stabilizing cases where two queries might be seen to complete
in either order. Note that this step can be *launched* before the other
-step has completed. (If the other step is used more than once in the
-current permutation, this step cannot complete while any of those
-instances is active.)
+step has completed or launched. (If the other step is used more than once
+in the current permutation, this step cannot complete while any of those
+instances are not yet complete.)
A marker of the form "<other step name> notices <n>" (where <n> is a
positive integer) indicates that this step may not be reported as
diff --git a/src/test/isolation/isolationtester.c b/src/test/isolation/isolationtester.c
index e01c0c9de93..676563f959d 100644
--- a/src/test/isolation/isolationtester.c
+++ b/src/test/isolation/isolationtester.c
@@ -319,8 +319,8 @@ check_testspec(TestSpec *testspec)
}
pstep->step = *this;
- /* Mark the step used, for check below */
- pstep->step->used = true;
+ /* Mark the step used by incrementing non-complete counter*/
+ pstep->step->non_complete++;
}
/*
@@ -378,7 +378,7 @@ check_testspec(TestSpec *testspec)
{
for (i = 0; i < nallsteps; i++)
{
- if (!allsteps[i]->used)
+ if (allsteps[i]->non_complete == 0)
fprintf(stderr, "unused step name: %s\n", allsteps[i]->name);
}
}
@@ -568,6 +568,11 @@ run_permutation(TestSpec *testspec, int nsteps, PermutationStep **steps)
}
}
+ for (i = 0; i < nsteps; i++)
+ steps[i]->step->non_complete = 0;
+ for (i = 0; i < nsteps; i++)
+ steps[i]->step->non_complete++;
+
/* Perform steps */
for (i = 0; i < nsteps; i++)
{
@@ -1002,6 +1007,7 @@ try_complete_step(TestSpec *testspec, PermutationStep *pstep, int flags)
printf("step %s: <... completed>\n", step->name);
else
printf("step %s: %s\n", step->name, step->sql);
+ step->non_complete--;
while ((res = PQgetResult(conn)))
{
@@ -1093,9 +1099,7 @@ step_has_blocker(PermutationStep *pstep)
break;
case PSB_OTHER_STEP:
/* Block if referenced step is active */
- iconn = &conns[1 + blocker->step->session];
- if (iconn->active_step &&
- iconn->active_step->step == blocker->step)
+ if (blocker->step->non_complete > 0)
return true;
break;
case PSB_NUM_NOTICES:
diff --git a/src/test/isolation/isolationtester.h b/src/test/isolation/isolationtester.h
index 1ef14dc9909..44b4c028a21 100644
--- a/src/test/isolation/isolationtester.h
+++ b/src/test/isolation/isolationtester.h
@@ -36,7 +36,7 @@ struct Step
char *sql;
/* These fields are filled by check_testspec(): */
int session; /* identifies owning session */
- bool used; /* has step been used in a permutation? */
+ int non_complete; /* number of step execution in permutation which are not yet completed */
};
typedef enum
diff --git a/src/test/isolation/specparse.y b/src/test/isolation/specparse.y
index 98949a446e5..7705b34dc2e 100644
--- a/src/test/isolation/specparse.y
+++ b/src/test/isolation/specparse.y
@@ -156,7 +156,7 @@ step:
$$->name = $2;
$$->sql = $3;
$$->session = -1; /* until filled */
- $$->used = false;
+ $$->non_complete = 0;
}
;
diff --git a/src/test/modules/injection_points/Makefile b/src/test/modules/injection_points/Makefile
index 0753a9df58c..4f47f4a0e7e 100644
--- a/src/test/modules/injection_points/Makefile
+++ b/src/test/modules/injection_points/Makefile
@@ -13,7 +13,7 @@ PGFILEDESC = "injection_points - facility for injection points"
REGRESS = injection_points reindex_conc
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
-ISOLATION = basic inplace
+ISOLATION = basic inplace markers
TAP_TESTS = 1
diff --git a/src/test/modules/injection_points/expected/markers.out b/src/test/modules/injection_points/expected/markers.out
new file mode 100644
index 00000000000..136bbae8338
--- /dev/null
+++ b/src/test/modules/injection_points/expected/markers.out
@@ -0,0 +1,101 @@
+Parsed test spec with 3 sessions
+
+starting permutation: after before detach1 wakeup1 detach2 wakeup2
+injection_points_attach
+-----------------------
+
+(1 row)
+
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step after: SELECT injection_points_run('injection-points-wait-1'); <waiting ...>
+step before: SELECT injection_points_run('injection-points-wait-2'); <waiting ...>
+step detach1: SELECT injection_points_detach('injection-points-wait-1');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup1: SELECT injection_points_wakeup('injection-points-wait-1');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step detach2: SELECT injection_points_detach('injection-points-wait-2');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup2: SELECT injection_points_wakeup('injection-points-wait-2');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+step after: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+
+starting permutation: after wakeup1 before detach1 detach2 wakeup2
+injection_points_attach
+-----------------------
+
+(1 row)
+
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step after: SELECT injection_points_run('injection-points-wait-1'); <waiting ...>
+step wakeup1: SELECT injection_points_wakeup('injection-points-wait-1');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: SELECT injection_points_run('injection-points-wait-2'); <waiting ...>
+step detach1: SELECT injection_points_detach('injection-points-wait-1');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step detach2: SELECT injection_points_detach('injection-points-wait-2');
+injection_points_detach
+-----------------------
+
+(1 row)
+
+step wakeup2: SELECT injection_points_wakeup('injection-points-wait-2');
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step before: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
+step after: <... completed>
+injection_points_run
+--------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/meson.build b/src/test/modules/injection_points/meson.build
index 989b4db226b..3998bac852f 100644
--- a/src/test/modules/injection_points/meson.build
+++ b/src/test/modules/injection_points/meson.build
@@ -44,6 +44,7 @@ tests += {
'specs': [
'basic',
'inplace',
+ 'markers',
],
},
'tap': {
diff --git a/src/test/modules/injection_points/specs/markers.spec b/src/test/modules/injection_points/specs/markers.spec
new file mode 100644
index 00000000000..e9f56d7567b
--- /dev/null
+++ b/src/test/modules/injection_points/specs/markers.spec
@@ -0,0 +1,45 @@
+setup
+{
+ CREATE EXTENSION injection_points;
+}
+teardown
+{
+ DROP EXTENSION injection_points;
+}
+
+# Wait happens in the first session, wakeup in the second session.
+session s1
+setup {
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('injection-points-wait-1', 'wait');
+}
+step after { SELECT injection_points_run('injection-points-wait-1'); }
+
+session s2
+setup {
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('injection-points-wait-2', 'wait');
+}
+step before { SELECT injection_points_run('injection-points-wait-2'); }
+
+session s3
+step wakeup1 { SELECT injection_points_wakeup('injection-points-wait-1'); }
+step detach1 { SELECT injection_points_detach('injection-points-wait-1'); }
+step wakeup2 { SELECT injection_points_wakeup('injection-points-wait-2'); }
+step detach2 { SELECT injection_points_detach('injection-points-wait-2'); }
+
+permutation
+ after(before)
+ before
+ detach1
+ wakeup1
+ detach2
+ wakeup2
+
+permutation
+ after(before)
+ wakeup1
+ before
+ detach1
+ detach2
+ wakeup2
\ No newline at end of file
--
2.43.0
On Tue, Jan 14, 2025 at 12:16:22PM +0100, Michail Nikolaev wrote:
--- a/src/test/isolation/README +++ b/src/test/isolation/README @@ -183,9 +183,9 @@ A marker consisting solely of a step name indicates that this step may not be reported as completing until that other step has completed. This allows stabilizing cases where two queries might be seen to complete in either order. Note that this step can be *launched* before the other -step has completed. (If the other step is used more than once in the -current permutation, this step cannot complete while any of those -instances is active.) +step has completed or launched. (If the other step is used more than once +in the current permutation, this step cannot complete while any of those +instances are not yet complete.)
I misunderstood, and I was mistaken to see this as a bug fix. The
isolationtester is acting per its definition, and this would be a definition
change. Do others have opinions on the merits of today's definition vs. the
proposed definition?
I'd need to review the motivating test to form my own opinion on whether the
new definition makes it easier to write tests. I'm optimistic that it does
make things easier, because I think one can get the old behavior by defining
two steps with the same commands. Thanks for the patch. You could bundle
this in the thread that wants this to stabilize that thread's CI results.
As a side note on the broader topic of the difficulty of stabilizing isolation
test race conditions, I've thought it would be helpful to have a "strict
blocking condition mode" that complains at the end of the test run if the spec
had completion order ambiguities. I think these are the rules sufficient to
have a fully-specified completion order:
1. Every step that waits needs a blocking condition.
2. No two steps A & B name the same step C as a blocking condition. Instead,
make B block on A.
3. If a step appears in another step's blocking condition and is not the last
step listed in the permutation, the step listed after the unblocker needs a
blocking condition on a formerly-blocked step.
+session s1 +step after { SELECT injection_points_run('injection-points-wait-1'); }
FYI, I prefer the convention of ending each step name with the session number,
i.e. s/after/after1/ here. The first isolation specs used that style. I
mildly prefer it over the newer alternative styles, and I strongly prefer
styles that convey the session identity over styles that don't. No need to
send a new patch version, though.
Hello, Noah!
I misunderstood, and I was mistaken to see this as a bug fix. The
isolationtester is acting per its definition, and this would be a
definition
change.
Yes, you are right, but I think it is better to clarify it somehow
because from my point of view that definition feels like logic in patched
version.
But maybe my non-native English prevents me from understanding it correctly.
In my opinion, it is better to add some clarification like "this marker
affects only other steps which were launched before this step or during its
execution".
I'd need to review the motivating test to form my own opinion on whether
the
new definition makes it easier to write tests.
You could bundle this in the thread that wants this to stabilize that
thread's CI results.
I'll think about it, but currently it is stabilized using "notices"
and, you know, more changes - harder to merge :)
Also, I'll try one more time to stabilize it without "notices" - maybe I'll
get some insight.
Anyway - I'll ping you in that thread [0]/messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz (you're already in recipients) or
bring it here.
Best regards,
Mikhail.
[0]: /messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz
/messages/by-id/ZnoZ6GNwkJmq-gTh@paquier.xyz
On Tue, Jan 14, 2025 at 11:48:28AM -0800, Noah Misch wrote:
I misunderstood, and I was mistaken to see this as a bug fix. The
isolationtester is acting per its definition, and this would be a definition
change. Do others have opinions on the merits of today's definition vs. the
proposed definition?
I agree that Michail's case with the handling of the markers is kind
of strange in the case he has reported. Anyway, do you think that it
would be a good idea to change that knowing for how long we've relied
on isolationtester to do things the way they are? If we do it only on
HEAD, it would make the back-patching of newly-added tests for bug
fixes potentially more complex to deal with. If applying the new
definition across all the stable branches, this could impact something
outside code :/
I'd need to review the motivating test to form my own opinion on whether the
new definition makes it easier to write tests. I'm optimistic that it does
make things easier, because I think one can get the old behavior by defining
two steps with the same commands. Thanks for the patch. You could bundle
this in the thread that wants this to stabilize that thread's CI results.
If you can get a backward-compatible behavior, well, no objections
from here, I guess.
FYI, I prefer the convention of ending each step name with the session number,
i.e. s/after/after1/ here. The first isolation specs used that style. I
mildly prefer it over the newer alternative styles, and I strongly prefer
styles that convey the session identity over styles that don't. No need to
send a new patch version, though.
+1. Including the session number in the steps makes debugging easier.
--
Michael
Michael Paquier <michael@paquier.xyz> writes:
On Tue, Jan 14, 2025 at 11:48:28AM -0800, Noah Misch wrote:
I misunderstood, and I was mistaken to see this as a bug fix. The
isolationtester is acting per its definition, and this would be a definition
change. Do others have opinions on the merits of today's definition vs. the
proposed definition?
I agree that Michail's case with the handling of the markers is kind
of strange in the case he has reported. Anyway, do you think that it
would be a good idea to change that knowing for how long we've relied
on isolationtester to do things the way they are?
We've changed it before, so I'm not totally averse to considering
more change. But I'm not happy about the lack of any analysis in
this thread of the impact on existing test cases. In particular,
it seems to me that this proposal amounts to reducing the number
of states that can be reached during an isolation test. So it
seems to me that means strictly less coverage than we had before.
I'd prefer to start by asking why the proposed new test can't be
made stable without such a restriction.
(I don't mean to minimize the question of whether we need better
documentation about this behavior.)
regards, tom lane
On Wed, Jan 15, 2025 at 02:41:31PM +0900, Michael Paquier wrote:
On Tue, Jan 14, 2025 at 11:48:28AM -0800, Noah Misch wrote:
I misunderstood, and I was mistaken to see this as a bug fix. The
isolationtester is acting per its definition, and this would be a definition
change. Do others have opinions on the merits of today's definition vs. the
proposed definition?I agree that Michail's case with the handling of the markers is kind
of strange in the case he has reported. Anyway, do you think that it
would be a good idea to change that knowing for how long we've relied
on isolationtester to do things the way they are?
I think that would depend on what proposed tests become easier. Since the
original poster used the "notices" marker type to solve the motivating use
case, let's shelve this until there's new demand.
If we ever unshelve this and find much in today's specs that would malfunction
under the new definition, we might offer both definitions. A new marker
syntax would reach the new definition, and old specs would work as before.
If we do it only on
HEAD, it would make the back-patching of newly-added tests for bug
fixes potentially more complex to deal with. If applying the new
definition across all the stable branches, this could impact something
outside code :/
When I'm exposed to non-core tests, those tests usually target many major
versions simultaneously. Having the spec meaning vary across major versions
would hurt them more than it would help. Hence, back-patch if we do adopt the
change. As above, the impact on existing specs could inform whether we add
syntax vs. redefine existing syntax.
Hello, everyone!
I was able to stabilize (I hope so) tests on [0]/messages/by-id/CANtu0ojXmqjmEzp-=aJSxjsdE76iAsRgHBoK0QtYHimb_mEfsg@mail.gmail.com without any changes to
isolationtester and without "notices N".
So, I agree, it is better to put it on a shelf now. But a few words in
documentation may be a good idea.
Best regards,
Mikhail.
[0]: /messages/by-id/CANtu0ojXmqjmEzp-=aJSxjsdE76iAsRgHBoK0QtYHimb_mEfsg@mail.gmail.com
/messages/by-id/CANtu0ojXmqjmEzp-=aJSxjsdE76iAsRgHBoK0QtYHimb_mEfsg@mail.gmail.com
On Sat, Jan 18, 2025 at 02:03:33PM +0100, Michail Nikolaev wrote:
I was able to stabilize (I hope so) tests on [0] without any changes to
isolationtester and without "notices N".So, I agree, it is better to put it on a shelf now. But a few words in
documentation may be a good idea.
Could you summarize here what you have done to achieve test
stabilization in your new patch set posted at [1]/messages/by-id/CANtu0ohStYQkJL+_=40Odcqu2A4QPrM_VmofwkkudHGHFgymkA@mail.gmail.com -- Michael without using the
proposal of this thread?
Perhaps that's OK to leave things as they are, but your proposal here
is leading to a more natural definition in terms of how test ordering
can be enforced with injection points, leading to a less confusing
definition. The analysis of existing tests if this proposal moves
forward would be required, of course.
[1]: /messages/by-id/CANtu0ohStYQkJL+_=40Odcqu2A4QPrM_VmofwkkudHGHFgymkA@mail.gmail.com -- Michael
--
Michael
Hello, Michael!
Could you summarize here what you have done to achieve test
stabilization in your new patch set posted at [1] without using the
proposal of this thread?
Mostly idea is next:
Let's imagine we have two steps - step_before and step_after which may end
in either order.
Then instead of such step/markers structure:
step_before(step_after)
stepN
stepN+1
step_after
use the next:
step_before
stepN
stepN+1
step_after(step_before)
In the first case, there are two possible results:
1) step_before is finished before step_after - reported as step_before,
step_after
2) step_after is launched before step_before ends - reported as step_after,
step_before
But in the case second variant:
1) step_before is finished before step_after - reported as step_before,
step_after
2) step_after is launched before step_before ends - reported as
step_before, step_after
So, the second option provides the same result regardless of order of
finishing of step_before and step_after, which is the thing I want to
achieve here.
Best regards,
Mikhail.
On Sun, Jan 19, 2025 at 07:26:58PM +0100, Michail Nikolaev wrote:
Mostly idea is next:
Let's imagine we have two steps - step_before and step_after which may end
in either order.
Then instead of such step/markers structure:step_before(step_after)
stepN
stepN+1
step_afteruse the next:
step_before
stepN
stepN+1
step_after(step_before)In the first case, there are two possible results:
1) step_before is finished before step_after - reported as step_before,
step_after
2) step_after is launched before step_before ends - reported as step_after,
step_beforeBut in the case second variant:
1) step_before is finished before step_after - reported as step_before,
step_after
2) step_after is launched before step_before ends - reported as
step_before, step_afterSo, the second option provides the same result regardless of order of
finishing of step_before and step_after, which is the thing I want to
achieve here.
Stepping back, any variation in the total ordering of step-start and
step-complete events necessarily changes the expected output. Hence, we want
to freeze that ordering. We could do that directly, by having isolationtester
read a format that lists start and complete events. We could also have the
isolationtester write the same format, so you could draft a permutation
without blocking conditions and just save the order that isolationtester
discovers by experiment. I'd find that less mentally taxing than following
rules to reach a frozen ordering via blocking conditions.
As an example of one concrete way to achieve that, have isolationtester write
a results/SPECNAME.wait file that contains syntax like "permutation-event
waiting:S1 S2 complete:S1 S3". You'd copy that file to
expected/SPECNAME.wait, which isolationtester would read at startup. This
wouldn't replace "notices" blocking conditions for certain complex tests, and
tests with multiple expected outputs might not want it. It would cover the
rest. Any unprefixed entry (e.g. the plain "S2") would get the simplification
of skipping the check for whether the step has entered a wait.
How well would that meet your test's needs?
Hello, Noah!
Stepping back, any variation in the total ordering of step-start and
step-complete events necessarily changes the expected output. Hence, we
want
to freeze that ordering. We could do that directly, by having
isolationtester
read a format that lists start and complete events. We could also have
the
isolationtester write the same format, so you could draft a permutation
without blocking conditions and just save the order that isolationtester
discovers by experiment. I'd find that less mentally taxing than
following
rules to reach a frozen ordering via blocking conditions.
How well would that meet your test's needs?
It feels a bit strange to me to force isolationtest to output something in
a specific order, especially since we don't actually care about the order
in the test.
But why do we need that output at all? I think a simpler solution would be
to mark a step in a way that prevents isolationtester from outputting
detailed information about the completion of that step—just an overview of
the steps launched and completed at the end of the permutation.
In my tests, I evaluate how [CREATE|RE] INDEX CONCURRENTLY interferes with
UPSERTs. I’m not testing the exact ordering; I simply set the backends into
a specific state (with different index information) and let them proceed
one by one. I don’t care which one finishes first or last—I’m only
interested in identifying errors in the output.
Best regards,
Mikhail.
On Sun, Jan 19, 2025 at 09:55:32PM +0100, Michail Nikolaev wrote:
Stepping back, any variation in the total ordering of step-start and
step-complete events necessarily changes the expected output. Hence, wewant
to freeze that ordering. We could do that directly, by having
isolationtester
read a format that lists start and complete events. We could also have
the
isolationtester write the same format, so you could draft a permutation
without blocking conditions and just save the order that isolationtester
discovers by experiment. I'd find that less mentally taxing thanfollowing
rules to reach a frozen ordering via blocking conditions.
How well would that meet your test's needs?
It feels a bit strange to me to force isolationtest to output something in
a specific order, especially since we don't actually care about the order
in the test.But why do we need that output at all?
It's good to think of ways we can output differently for more stability.
However, ordered output helps the human studying what happened. Today's
output is well-optimized for that. Today's output resembles what a person
would see if running a collection of psql sessions. For example, if I'm
reviewing a patch, reviewing the expected spec output feels natural.
I think a simpler solution would be
to mark a step in a way that prevents isolationtester from outputting
detailed information about the completion of that step—just an overview of
the steps launched and completed at the end of the permutation.
My main concern about that is the difficulty of marking steps:
- Best: no markings
- Medium: easy rules for when you need a marking
- Worst: run on FreeBSD to determine marking requirements by experiment
I've not been able to identify an easy rule that avoids "run on FreeBSD" with
today's framework. (I have a non-easy list of four rules, but I'm not
confident even that list is complete.) I can't think of an easy rule for
"just an overview" markings, either. Even if there is one, not needing
markings is better, all else being equal. What other advantages may
compensate for this concern?
There's another, minor concern. The more "just an overview" markings in a
spec, the more reading its expected output wouldn't feel like reading psql
transcripts. I'd likely end up logging both today's ordered format and the
new format. We'd diff the new format only, and we'd write the old format for
human debugging.
In my tests, I evaluate how [CREATE|RE] INDEX CONCURRENTLY interferes with
UPSERTs. I’m not testing the exact ordering; I simply set the backends into
a specific state (with different index information) and let them proceed
one by one. I don’t care which one finishes first or last—I’m only
interested in identifying errors in the output.
That makes sense.
Thanks,
nm