refactor subscription tests to use PostgresNode's wait_for_catchup
Started by Peter Eisentrautabout 8 years ago5 messages
It appears that we have unwittingly created some duplicate and
copy-and-paste-prone code in src/test/subscription/ to wait for a
replication subscriber to catch up, when we already have
almost-sufficient code in PostgresNode to do that more compactly. So I
propose this patch to consolidate that.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Refactor-subscription-tests-to-use-PostgresNode-s-wa.patchtext/plain; charset=UTF-8; name=0001-Refactor-subscription-tests-to-use-PostgresNode-s-wa.patch; x-mac-creator=0; x-mac-type=0Download
From daf3c2d02a409a33bff349c558d79dcdd966982e Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Mon, 8 Jan 2018 17:32:09 -0500
Subject: [PATCH] Refactor subscription tests to use PostgresNode's
wait_for_catchup
---
src/test/perl/PostgresNode.pm | 16 +++++++++++++---
src/test/subscription/t/001_rep_changes.pl | 19 +++++--------------
src/test/subscription/t/002_types.pl | 15 ++++-----------
src/test/subscription/t/003_constraints.pl | 15 ++++-----------
src/test/subscription/t/004_sync.pl | 14 +++-----------
src/test/subscription/t/005_encoding.pl | 13 ++-----------
src/test/subscription/t/006_rewrite.pl | 17 ++++-------------
src/test/subscription/t/007_ddl.pl | 11 +----------
src/test/subscription/t/008_diff_schema.pl | 15 +++------------
9 files changed, 39 insertions(+), 96 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 80f68df246..6062f41480 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1465,7 +1465,8 @@ sub lsn
=item $node->wait_for_catchup(standby_name, mode, target_lsn)
-Wait for the node with application_name standby_name (usually from node->name)
+Wait for the node with application_name standby_name (usually from node->name,
+also works for logical subscriptions)
until its replication location in pg_stat_replication equals or passes the
upstream's WAL insert point at the time this function is called. By default
the replay_lsn is waited for, but 'mode' may be specified to wait for any of
@@ -1477,6 +1478,7 @@ poll_query_until timeout.
Requires that the 'postgres' db exists and is accessible.
target_lsn may be any arbitrary lsn, but is typically $master_node->lsn('insert').
+If omitted, pg_current_wal_lsn() is used.
This is not a test. It die()s on failure.
@@ -1497,7 +1499,15 @@ sub wait_for_catchup
{
$standby_name = $standby_name->name;
}
- die 'target_lsn must be specified' unless defined($target_lsn);
+ my $lsn_expr;
+ if (defined($target_lsn))
+ {
+ $lsn_expr = "'$target_lsn'";
+ }
+ else
+ {
+ $lsn_expr = 'pg_current_wal_lsn()'
+ }
print "Waiting for replication conn "
. $standby_name . "'s "
. $mode
@@ -1505,7 +1515,7 @@ sub wait_for_catchup
. $target_lsn . " on "
. $self->name . "\n";
my $query =
-qq[SELECT '$target_lsn' <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';];
+qq[SELECT $lsn_expr <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';];
$self->poll_query_until('postgres', $query)
or die "timed out waiting for catchup, current location is "
. ($self->safe_psql('postgres', $query) || '(unknown)');
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 0136c79d4b..e0104cd8d0 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -60,11 +60,7 @@
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub, tap_pub_ins_only"
);
-# Wait for subscriber to finish initialization
-my $caughtup_query =
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';";
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Also wait for initial table sync to finish
my $synced_query =
@@ -93,8 +89,7 @@
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_mixed VALUES (2, 'bar')");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), min(a), max(a) FROM tab_ins");
@@ -132,9 +127,7 @@
$node_publisher->safe_psql('postgres',
"UPDATE tab_full2 SET x = 'bb' WHERE x = 'b'");
-# Wait for subscription to catch up
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), min(a), max(a) FROM tab_full");
@@ -176,8 +169,7 @@
"INSERT INTO tab_ins SELECT generate_series(1001,1100)");
$node_publisher->safe_psql('postgres', "DELETE FROM tab_rep");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*), min(a), max(a) FROM tab_ins");
@@ -200,8 +192,7 @@
);
$node_publisher->safe_psql('postgres', "INSERT INTO tab_full VALUES(0)");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# note that data are different on provider and subscriber
$result = $node_subscriber->safe_psql('postgres',
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index 3ca027ecb4..80620416fa 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -106,11 +106,7 @@
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
);
-# Wait for subscriber to finish initialization
-my $caughtup_query =
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';";
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Wait for initial sync to finish as well
my $synced_query =
@@ -246,8 +242,7 @@
(4, '"yellow horse"=>"moaned"');
));
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Check the data on subscriber
my $result = $node_subscriber->safe_psql(
@@ -368,8 +363,7 @@
UPDATE tst_hstore SET b = '"also"=>"updated"' WHERE a = 3;
));
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Check the data on subscriber
$result = $node_subscriber->safe_psql(
@@ -489,8 +483,7 @@
DELETE FROM tst_hstore WHERE a = 1;
));
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Check the data on subscriber
$result = $node_subscriber->safe_psql(
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 06863aef84..6f6805b952 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -39,19 +39,14 @@
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (copy_data = false)"
);
-# Wait for subscriber to finish initialization
-my $caughtup_query =
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';";
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_fk (bid) VALUES (1);");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_fk_ref (id, bid) VALUES (1, 1);");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Check data on subscriber
my $result = $node_subscriber->safe_psql('postgres',
@@ -69,8 +64,7 @@
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_fk_ref (id, bid) VALUES (2, 2);");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# FK is not enforced on subscriber
$result = $node_subscriber->safe_psql('postgres',
@@ -104,8 +98,7 @@ BEGIN
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_fk_ref (id, bid) VALUES (10, 10);");
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# The row should be skipped on subscriber
$result = $node_subscriber->safe_psql('postgres',
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index 05fd2f0e6c..a9a223bdf7 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -37,11 +37,7 @@
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
);
-# Wait for subscriber to finish initialization
-my $caughtup_query =
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';";
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
# Also wait for initial table sync to finish
my $synced_query =
@@ -124,9 +120,7 @@
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_rep_next (a) AS SELECT generate_series(1,10)");
-# Wait for subscription to catch up
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM tab_rep_next");
@@ -149,9 +143,7 @@
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_rep_next SELECT generate_series(1,10)");
-# Wait for subscription to catch up
-$node_publisher->poll_query_until('postgres', $caughtup_query)
- or die "Timed out while waiting for subscriber to catch up";
+$node_publisher->wait_for_catchup($appname);
$result = $node_subscriber->safe_psql('postgres',
"SELECT count(*) FROM tab_rep_next");
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index 2b0c47c07d..65439f1b28 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -5,15 +5,6 @@
use TestLib;
use Test::More tests => 1;
-sub wait_for_caught_up
-{
- my ($node, $appname) = @_;
-
- $node->poll_query_until('postgres',
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
- ) or die "Timed out while waiting for subscriber to catch up";
-}
-
my $node_publisher = get_new_node('publisher');
$node_publisher->init(
allows_streaming => 'logical',
@@ -39,7 +30,7 @@ sub wait_for_caught_up
"CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION mypub;"
);
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
# Wait for initial sync to finish as well
my $synced_query =
@@ -50,7 +41,7 @@ sub wait_for_caught_up
$node_publisher->safe_psql('postgres',
q{INSERT INTO test1 VALUES (1, E'Mot\xc3\xb6rhead')}); # hand-rolled UTF-8
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
is( $node_subscriber->safe_psql(
'postgres', q{SELECT a FROM test1 WHERE b = E'Mot\xf6rhead'}
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index 5e3211aefa..aa1184c85f 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -5,15 +5,6 @@
use TestLib;
use Test::More tests => 2;
-sub wait_for_caught_up
-{
- my ($node, $appname) = @_;
-
- $node->poll_query_until('postgres',
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
- ) or die "Timed out while waiting for subscriber to catch up";
-}
-
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
$node_publisher->start;
@@ -35,7 +26,7 @@ sub wait_for_caught_up
"CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION mypub;"
);
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
# Wait for initial sync to finish as well
my $synced_query =
@@ -45,7 +36,7 @@ sub wait_for_caught_up
$node_publisher->safe_psql('postgres', q{INSERT INTO test1 (a, b) VALUES (1, 'one'), (2, 'two');});
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
is($node_subscriber->safe_psql('postgres', q{SELECT a, b FROM test1}),
qq(1|one
@@ -57,11 +48,11 @@ sub wait_for_caught_up
$node_subscriber->safe_psql('postgres', $ddl2);
$node_publisher->safe_psql('postgres', $ddl2);
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
$node_publisher->safe_psql('postgres', q{INSERT INTO test1 (a, b, c) VALUES (3, 'three', 33);});
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
is($node_subscriber->safe_psql('postgres', q{SELECT a, b, c FROM test1}),
qq(1|one|0
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 3f36238840..b219bf33dd 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -5,15 +5,6 @@
use TestLib;
use Test::More tests => 1;
-sub wait_for_caught_up
-{
- my ($node, $appname) = @_;
-
- $node->poll_query_until('postgres',
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
- ) or die "Timed out while waiting for subscriber to catch up";
-}
-
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
$node_publisher->start;
@@ -35,7 +26,7 @@ sub wait_for_caught_up
"CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION mypub;"
);
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
$node_subscriber->safe_psql('postgres', q{
BEGIN;
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index b71be6e487..ea31625402 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -5,15 +5,6 @@
use TestLib;
use Test::More tests => 3;
-sub wait_for_caught_up
-{
- my ($node, $appname) = @_;
-
- $node->poll_query_until('postgres',
-"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
- ) or die "Timed out while waiting for subscriber to catch up";
-}
-
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
@@ -42,7 +33,7 @@ sub wait_for_caught_up
"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
);
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
# Also wait for initial table sync to finish
my $synced_query =
@@ -58,7 +49,7 @@ sub wait_for_caught_up
# subscriber didn't change
$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(b)");
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
@@ -70,7 +61,7 @@ sub wait_for_caught_up
$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
-wait_for_caught_up($node_publisher, $appname);
+$node_publisher->wait_for_catchup($appname);
$result =
$node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
--
2.15.1
Re: refactor subscription tests to use PostgresNode's wait_for_catchup
On Mon, Jan 08, 2018 at 09:46:21PM -0500, Peter Eisentraut wrote:
It appears that we have unwittingly created some duplicate and
copy-and-paste-prone code in src/test/subscription/ to wait for a
replication subscriber to catch up, when we already have
almost-sufficient code in PostgresNode to do that more compactly. So I
propose this patch to consolidate that.
This looks sane to me. I have two comments while I read the
surroundings.
@@ -1505,7 +1515,7 @@ sub wait_for_catchup . $target_lsn . " on " . $self->name . "\n"; my $query = -qq[SELECT '$target_lsn' <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';]; +qq[SELECT $lsn_expr <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';]; $self->poll_query_until('postgres', $query) or die "timed out waiting for catchup, current location is " . ($self->safe_psql('postgres', $query) || '(unknown)');
This log is wrong from the beginning. Here $query returns a boolean
status and not a location. I think that when the poll dies because of a
timeout you should do a lookup at ${mode}_lsn from pg_stat_replication
when application_name matching $standby_name. Could you fix that as
well?
Could you also update promote_standby in RewindTest.pm? Your refactoring
to use pg_current_wal_lsn() if a target_lsn is not possible makes this
move possible. Using the generic APIs gives better logs as well.
--
Michael
Re: refactor subscription tests to use PostgresNode's wait_for_catchup
On 1/8/18 23:47, Michael Paquier wrote:
@@ -1505,7 +1515,7 @@ sub wait_for_catchup . $target_lsn . " on " . $self->name . "\n"; my $query = -qq[SELECT '$target_lsn' <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';]; +qq[SELECT $lsn_expr <= ${mode}_lsn FROM pg_catalog.pg_stat_replication WHERE application_name = '$standby_name';]; $self->poll_query_until('postgres', $query) or die "timed out waiting for catchup, current location is " . ($self->safe_psql('postgres', $query) || '(unknown)');This log is wrong from the beginning. Here $query returns a boolean
status and not a location. I think that when the poll dies because of a
timeout you should do a lookup at ${mode}_lsn from pg_stat_replication
when application_name matching $standby_name. Could you fix that as
well?
Should we just remove it? Apparently, it was never functional to begin
with. Otherwise, we'd have to write a second query to return the value
to print. wait_for_slot_catchup has the same issue. Seems like a lot
of overhead for something that has never been used.
Could you also update promote_standby in RewindTest.pm? Your refactoring
to use pg_current_wal_lsn() if a target_lsn is not possible makes this
move possible. Using the generic APIs gives better logs as well.
Right.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: refactor subscription tests to use PostgresNode's wait_for_catchup
On Wed, Jan 10, 2018 at 09:45:56PM -0500, Peter Eisentraut wrote:
On 1/8/18 23:47, Michael Paquier wrote:
Should we just remove it? Apparently, it was never functional to begin
with. Otherwise, we'd have to write a second query to return the value
to print. wait_for_slot_catchup has the same issue. Seems like a lot
of overhead for something that has never been used.
Fine for me to remove it.
--
Michael
Re: refactor subscription tests to use PostgresNode's wait_for_catchup
On 1/10/18 22:24, Michael Paquier wrote:
On Wed, Jan 10, 2018 at 09:45:56PM -0500, Peter Eisentraut wrote:
On 1/8/18 23:47, Michael Paquier wrote:
Should we just remove it? Apparently, it was never functional to begin
with. Otherwise, we'd have to write a second query to return the value
to print. wait_for_slot_catchup has the same issue. Seems like a lot
of overhead for something that has never been used.
committed
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services