Making background psql nicer to use in tap tests
Hi,
Plenty tap tests require a background psql. But they're pretty annoying to
use.
I think the biggest improvement would be an easy way to run a single query and
get the result of that query. Manually having to pump_until() is awkward and
often leads to hangs/timeouts, instead of test failures, because one needs to
specify a match pattern to pump_until(), which on mismatch leads to trying to
keep pumping forever.
It's annoyingly hard to wait for the result of a query in a generic way with
background_psql(), and more generally for psql. background_psql() uses -XAtq,
which means that we'll not get "status" output (like "BEGIN" or "(1 row)"),
and that queries not returning anything are completely invisible.
A second annoyance is that issuing a query requires a trailing newline,
otherwise psql won't process it.
The best way I can see is to have a helper that issues the query, followed by
a trailing newline, an \echo with a recognizable separator, and then uses
pump_until() to wait for that separator.
Another area worthy of improvement is that background_psql() requires passing
in many variables externally - without a recognizable benefit afaict. What's
the point in 'stdin', 'stdout', 'timer' being passed in? stdin/stdout need to
point to empty strings, so we know what's needed - in fact we'll even reset
them if they're passed in. The timer is always going to be
PostgreSQL::Test::Utils::timeout_default, so again, what's the point?
I think it'd be far more usable if we made background_psql() return a hash
with the relevant variables. The 031_recovery_conflict.pl test has:
my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
my %psql_standby = ('stdin' => '', 'stdout' => '');
$psql_standby{run} =
$node_standby->background_psql($test_db, \$psql_standby{stdin},
\$psql_standby{stdout},
$psql_timeout);
$psql_standby{stdout} = '';
How about just returning a reference to a hash like that? Except that I'd also
make stderr available, which one can't currently access.
The $psql_standby{stdout} = ''; is needed because background_psql() leaves a
banner in the output, which it shouldn't, but we probably should just fix
that.
Brought to you by: Trying to write a test for vacuum_defer_cleanup_age.
- Andres
Andres Freund <andres@anarazel.de> writes:
It's annoyingly hard to wait for the result of a query in a generic way with
background_psql(), and more generally for psql. background_psql() uses -XAtq,
which means that we'll not get "status" output (like "BEGIN" or "(1 row)"),
and that queries not returning anything are completely invisible.
Yeah, the empty-query-result problem was giving me fits recently.
+1 for wrapping this into something more convenient to use.
regards, tom lane
Hi,
On 2023-01-30 15:06:46 -0500, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:
It's annoyingly hard to wait for the result of a query in a generic way with
background_psql(), and more generally for psql. background_psql() uses -XAtq,
which means that we'll not get "status" output (like "BEGIN" or "(1 row)"),
and that queries not returning anything are completely invisible.Yeah, the empty-query-result problem was giving me fits recently.
+1 for wrapping this into something more convenient to use.
I've hacked some on this. I first tried to just introduce a few helper
functions in Cluster.pm, but that ended up being awkward. So I bit the bullet
and introduced a new class (in BackgroundPsql.pm), and made background_psql()
and interactive_psql() return an instance of it.
This is just a rough prototype. Several function names don't seem great, it
need POD documentation, etc.
The main convenience things it has over the old interface:
- $node->background_psql('dbname') is enough
- $psql->query(), which returns the query results as a string, is a lot easier
to use than having to pump, identify query boundaries via regex etc.
- $psql->query_safe(), which dies if any query fails (detected via stderr)
- $psql->query_until() is a helper that makes it a bit easier to start queries
that won't finish until a later point
I don't quite like the new interface yet:
- It's somewhat common to want to know if there was a failure, but also get
the query result, not sure what the best function signature for that is in
perl.
- query_until() sounds a bit too much like $node->poll_query_until(). Maybe
query_wait_until() is better? OTOH, the other function has poll in the name,
so maybe it's ok.
- right now there's a bit too much logic in background_psql() /
interactive_psql() for my taste
Those points aside, I think it already makes the tests a good bit more
readable. My WIP vacuum_defer_cleanup_age patch shrunk by half with it.
I think with a bit more polish it's easy enough to use that we could avoid a
good number of those one-off psql's that we do all over.
I didn't really know what this, insrc/test/subscription/t/015_stream.pl, is
about:
$h->finish; # errors make the next test fail, so ignore them here
There's no further test?
I'm somewhat surprised it doesn't cause problems in another ->finish later on,
where we then afterwards just use $h again. Apparently IPC::Run just
automagically restarts psql?
Greetings,
Andres Freund
Attachments:
v1-0001-WIP-test-Introduce-BackgroundPsql-class.patchtext/x-diff; charset=us-asciiDownload
From fb0e9fceead01aaee0bdd05c3ad6c67814f6b820 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 30 Jan 2023 15:39:08 -0800
Subject: [PATCH v1] WIP: test: Introduce BackgroundPsql class
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
src/bin/psql/t/010_tab_completion.pl | 27 +--
contrib/amcheck/t/003_cic_2pc.pl | 66 +++----
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 165 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 83 ++-------
src/test/recovery/t/031_recovery_conflict.pl | 101 +++--------
src/test/subscription/t/015_stream.pl | 52 ++----
6 files changed, 252 insertions(+), 242 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 7746c75e0c9..7c382b98f8f 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -92,14 +92,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +102,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -443,10 +433,7 @@ check_completion("blarg \t\t", qr//, "check completion failure path");
clear_query();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf964..5323ed11ae9 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+my $main_h = $node->background_psql('postgres');
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
+));
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 00000000000..9257075224d
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,165 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+# Start a new psql background session
+#
+# Parameters:
+# - "interactive" - should a PTY be used
+# - "psql" - psql command, parameters, including connection string
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => ''};
+ my $run;
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->wait_connect();
+
+ return $self;
+}
+
+sub wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->wait_connect()
+}
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ # feed the query to psql's stdin, follwed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $ret = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $ret =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+# Like query(), but errors out if the query failed.
+#
+# Query failure is determined by producing output on stderr.
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+# issue query, but only wait for $until, not query completion
+#
+# Note: Query needs newline and semicolon for psql to process the input.
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 04921ca3a3d..e16ff197cf9 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2029,45 +2024,18 @@ sub background_psql
$params{on_error_stop} = 1 unless defined $params{on_error_stop};
- push @psql_params, '-v', 'ON_ERROR_STOP=1' if $params{on_error_stop};
- push @psql_params, @{ $params{extra_params} }
- if defined $params{extra_params};
-
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2043,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2061,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2072,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 875afb8e3ce..b5a5e634f3a 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -68,14 +68,7 @@ $node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db);
my $expected_conflicts = 0;
@@ -104,15 +97,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -128,7 +120,7 @@ $primary_lsn = $node_primary->lsn('flush');
$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -142,15 +134,12 @@ $primary_lsn = $node_primary->lsn('flush');
$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -165,7 +154,7 @@ $node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -174,12 +163,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -188,7 +177,7 @@ $primary_lsn = $node_primary->lsn('flush');
$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -199,14 +188,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -218,7 +207,7 @@ $node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -234,7 +223,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -258,19 +247,15 @@ SELECT txid_current();
$primary_lsn = $node_primary->lsn('flush');
$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -286,7 +271,7 @@ $primary_lsn = $node_primary->lsn('flush');
$node_primary->wait_for_catchup($node_standby, 'replay', $primary_lsn);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -294,7 +279,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -319,8 +304,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -328,37 +312,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 91e8aa8c0a5..7d3c893df23 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.38.0
On 31 Jan 2023, at 01:00, Andres Freund <andres@anarazel.de> wrote:
I've hacked some on this. I first tried to just introduce a few helper
functions in Cluster.pm, but that ended up being awkward. So I bit the bullet
and introduced a new class (in BackgroundPsql.pm), and made background_psql()
and interactive_psql() return an instance of it.
Thanks for working on this!
This is just a rough prototype. Several function names don't seem great, it
need POD documentation, etc.
It might be rough around the edges but I don't think it's too far off a state
in which in can be committed, given that it's replacing something even rougher.
With documentation and some polish I think we can iterate on it in the tree.
I've played around a lot with it and it seems fairly robust.
I don't quite like the new interface yet:
- It's somewhat common to want to know if there was a failure, but also get
the query result, not sure what the best function signature for that is in
perl.
What if query() returns a list with the return value last? The caller will get
the return value when assigning a single var as the return, and can get both in
those cases when it's interesting. That would make for reasonably readable
code in most places?
$ret_val = $h->query("SELECT 1;");
($query_result, $ret_val) = $h->query("SELECT 1;");
Returning a hash seems like a worse option since it will complicate callsites
which only want to know success/failure.
- query_until() sounds a bit too much like $node->poll_query_until(). Maybe
query_wait_until() is better? OTOH, the other function has poll in the name,
so maybe it's ok.
query_until isn't great but query_wait_until is IMO worse since the function
may well be used for tests which aren't using longrunning waits. It's also
very useful for things which aren't queries at all, like psql backslash
commands. I don't have any better ideas though, so +1 for sticking with
query_until.
- right now there's a bit too much logic in background_psql() /
interactive_psql() for my taste
Not sure what you mean, I don't think they're especially heavy on logic?
Those points aside, I think it already makes the tests a good bit more
readable. My WIP vacuum_defer_cleanup_age patch shrunk by half with it.
The test for \password in the SCRAM iteration count patch shrunk to 1/3 of the
previous coding.
I think with a bit more polish it's easy enough to use that we could avoid a
good number of those one-off psql's that we do all over.
Agreed, and ideally implement tests which were left unwritten due to the old
API being clunky.
+ # feed the query to psql's stdin, follwed by \n (so psql processes the
s/follwed/followed/
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
This require a bit of knowledge about the internals which I think we should
hide in this new API. How about providing a function for defining the timeout?
Re timeouts: one thing I've done repeatedly is to use short timeouts and reset
them per query, and that turns pretty ugly fast. I hacked up your patch to
provide $h->reset_timer_before_query() which then injects a {timeout}->start
before running each query without the caller having to do it. Not sure if I'm
alone in doing that but if not I think it makes sense to add.
--
Daniel Gustafsson
Hi,
On 2023-03-14 21:24:32 +0100, Daniel Gustafsson wrote:
On 31 Jan 2023, at 01:00, Andres Freund <andres@anarazel.de> wrote:
I've hacked some on this. I first tried to just introduce a few helper
functions in Cluster.pm, but that ended up being awkward. So I bit the bullet
and introduced a new class (in BackgroundPsql.pm), and made background_psql()
and interactive_psql() return an instance of it.Thanks for working on this!
Thanks for helping it move along :)
This is just a rough prototype. Several function names don't seem great, it
need POD documentation, etc.It might be rough around the edges but I don't think it's too far off a state
in which in can be committed, given that it's replacing something even rougher.
With documentation and some polish I think we can iterate on it in the tree.
Cool.
I don't quite like the new interface yet:
- It's somewhat common to want to know if there was a failure, but also get
the query result, not sure what the best function signature for that is in
perl.What if query() returns a list with the return value last? The caller will get
the return value when assigning a single var as the return, and can get both in
those cases when it's interesting. That would make for reasonably readable
code in most places?
$ret_val = $h->query("SELECT 1;");
($query_result, $ret_val) = $h->query("SELECT 1;");
I hate perl.
Returning a hash seems like a worse option since it will complicate callsites
which only want to know success/failure.
Yea. Perhaps it's worth having a separate function for this? ->query_rc() or such?
- right now there's a bit too much logic in background_psql() /
interactive_psql() for my tasteNot sure what you mean, I don't think they're especially heavy on logic?
-EMISSINGWORD on my part. A bit too much duplicated logic.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up, +which can modified later.This require a bit of knowledge about the internals which I think we should
hide in this new API. How about providing a function for defining the timeout?
"definining" in the sense of accessing it? Or passing one in?
Re timeouts: one thing I've done repeatedly is to use short timeouts and reset
them per query, and that turns pretty ugly fast. I hacked up your patch to
provide $h->reset_timer_before_query() which then injects a {timeout}->start
before running each query without the caller having to do it. Not sure if I'm
alone in doing that but if not I think it makes sense to add.
I don't quite understand the use case, but I don't mind it as a functionality.
Greetings,
Andres Freund
On 2023-01-30 Mo 19:00, Andres Freund wrote:
Hi,
On 2023-01-30 15:06:46 -0500, Tom Lane wrote:
Andres Freund<andres@anarazel.de> writes:
It's annoyingly hard to wait for the result of a query in a generic way with
background_psql(), and more generally for psql. background_psql() uses -XAtq,
which means that we'll not get "status" output (like "BEGIN" or "(1 row)"),
and that queries not returning anything are completely invisible.Yeah, the empty-query-result problem was giving me fits recently.
+1 for wrapping this into something more convenient to use.I've hacked some on this. I first tried to just introduce a few helper
functions in Cluster.pm, but that ended up being awkward. So I bit the bullet
and introduced a new class (in BackgroundPsql.pm), and made background_psql()
and interactive_psql() return an instance of it.This is just a rough prototype. Several function names don't seem great, it
need POD documentation, etc.
Since this class is only intended to have instances created from
Cluster, I would be inclined just to put it at the end of Cluster.pm
instead of creating a new file. That makes it clearer that the new
package is not standalone. We already have instances of that.
The first param of the constructor is a bit opaque. If it were going to
be called from elsewhere I'd want something a bit more obvious, but I
guess we can live with it here. An alternative might be
multiple_constructors (e.g. new_background, new_interactive) which use a
common private routine.
Don't have comments yet on the other things, will continue looking.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On 15 Mar 2023, at 02:03, Andres Freund <andres@anarazel.de> wrote:
Returning a hash seems like a worse option since it will complicate callsites
which only want to know success/failure.Yea. Perhaps it's worth having a separate function for this? ->query_rc() or such?
If we are returning a hash then I agree it should be a separate function.
Maybe Andrew has input on which is the most Perl way of doing this.
- right now there's a bit too much logic in background_psql() /
interactive_psql() for my tasteNot sure what you mean, I don't think they're especially heavy on logic?
-EMISSINGWORD on my part. A bit too much duplicated logic.
That makes more sense, and I can kind of agree. I don't think it's too bad but
I agree there is room for improvement.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up, +which can modified later.This require a bit of knowledge about the internals which I think we should
hide in this new API. How about providing a function for defining the timeout?"definining" in the sense of accessing it? Or passing one in?
I meant passing one in.
Re timeouts: one thing I've done repeatedly is to use short timeouts and reset
them per query, and that turns pretty ugly fast. I hacked up your patch to
provide $h->reset_timer_before_query() which then injects a {timeout}->start
before running each query without the caller having to do it. Not sure if I'm
alone in doing that but if not I think it makes sense to add.I don't quite understand the use case, but I don't mind it as a functionality.
I've used it a lot when I want to run n command which each should finish
quickly or not at all. So one time budget per command rather than having a
longer timeout for a set of commands that comprise a test. It can be done
already today by calling ->start but it doesn't exactly make the code cleaner.
As mentioned off-list I did some small POD additions when reviewing, so I've
attached them here in a v2 in the hopes that it might be helpful. I've also
included the above POC for restarting the timeout per query to show what I
meant.
--
Daniel Gustafsson
Attachments:
v2-0001-Refactor-background-psql-TAP-functions.patchapplication/octet-stream; name=v2-0001-Refactor-background-psql-TAP-functions.patch; x-unix-mode=0644Download
From 6be886f8ec7c1540eaefe405aeb76549bfbcfbff Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Fri, 10 Mar 2023 14:38:35 +0100
Subject: [PATCH v2] Refactor background psql TAP functions
Author: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
contrib/amcheck/t/003_cic_2pc.pl | 70 ++---
src/bin/psql/t/010_tab_completion.pl | 27 +-
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 291 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 83 +----
src/test/recovery/t/031_recovery_conflict.pl | 101 ++----
src/test/subscription/t/015_stream.pl | 52 +---
6 files changed, 380 insertions(+), 244 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf96..5323ed11ae 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+my $main_h = $node->background_psql('postgres');
+
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
-
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+));
+
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 55a88f9812..ce609f450e 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -92,14 +92,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +102,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -451,10 +441,7 @@ check_completion(
clear_line();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 0000000000..1990b47a4e
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,291 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::BackgroundPsql - class for controlling background psql processes
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::Cluster;
+
+ my $node = PostgreSQL::Test::Cluster->new('mynode');
+
+ # Create a data directory with initdb
+ $node->init();
+
+ # Start the PostgreSQL server
+ $node->start();
+
+ # Create and start an interactive psql session
+ my $isession = $node->interactive_psql('postgres');
+
+ # Run a query and get the output as seen by psql
+ my $ret = $isession->query("SELECT 1");
+ # Run a backslash command and wait until the prompt returns
+ $isession->query_until(qr/postgres #/, "\\d foo\n");
+ # Close the session and exit psql
+ $isession->quit;
+
+ # Create and start a background psql session
+ my $bsession = $node->background_psql('postgres');
+
+ # Run a query which is guaranteed to not return in case it fails
+ $bsession->query_safe("SELECT 1");
+ # Initiate a command which can be expected to terminate at a later stage
+ $bsession->query_until(qr/start/, q(
+ \echo start
+ CREATE INDEX CONCURRENTLY idx ON t(a);
+ ));
+ # Close the session and exit psql
+ $bsession->quit;
+
+=head1 DESCRIPTION
+
+PostgreSQL::Test::BackgroundPsql contains functionality for controlling a
+background or interactive psql session operating on a PostgreSQL node
+initiated by PostgreSQL::Test::Cluster.
+
+=cut
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+our ($restart_before_query);
+
+INIT
+{
+ $restart_before_query = 0;
+}
+
+=pod
+
+=head1 METHODS
+
+=over
+
+=item PostgreSQL::Test::BackroundPsql->new(interactive, @params)
+
+Builds a new object of class C<PostgreSQL::Test::BackgroundPsql> for either
+an interactive or background session and starts it. If C<interactive> is
+true then a PTY will be attached. C<psql_params> should contain the full
+command to run psql with all desired parameters and a complete connection
+string.
+
+=cut
+
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => ''};
+ my $run;
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->wait_connect();
+
+ return $self;
+}
+
+# Internal routine for awaiting psql starting up and being ready to consume
+# input.
+sub wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+=pod
+
+=item quit
+
+Close the session and clean up resources. Each test run must be closed with
+C<quit>.
+
+=cut
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+=pod
+
+=item $session->reconnect_and_clear
+
+Terminate the current session and connect again.
+
+=cut
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->wait_connect()
+}
+
+=pod
+
+=item $session->query()
+
+Executes a query in the current session and returns the output.
+
+=cut
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ $self->{timeout}->start() if ($restart_before_query == 1);
+
+ # Feed the query to psql's stdin, followed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $ret = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $ret =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_safe()
+
+Wrapper around C<query> which errors out if the query failed to execute.
+Query failure is determined by it producing output on stderr.
+
+=cut
+
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_until(until, query)
+
+Issue C<query> and wait for C<until> appearing in the query output rather than
+waiting for query completion. C<query> needs to end with newline and semicolon
+(if applicable) for psql to process the input.
+
+=cut
+
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{timeout}->start() if ($restart_before_query == 1);
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->restart_timer_before_query()
+
+Configures the timer to be restarted before each query such that the defined
+timeout is valid per query rather than per test run.
+
+=cut
+
+sub restart_timer_before_query
+{
+ my ($self) = @_;
+
+ $restart_before_query = 1;
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 3e2a27fb71..29e5bdc9b2 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2029,45 +2024,18 @@ sub background_psql
$params{on_error_stop} = 1 unless defined $params{on_error_stop};
- push @psql_params, '-v', 'ON_ERROR_STOP=1' if $params{on_error_stop};
- push @psql_params, @{ $params{extra_params} }
- if defined $params{extra_params};
-
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2043,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2061,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2072,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 84375faccb..66673420c8 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -67,14 +67,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db);
my $expected_conflicts = 0;
@@ -102,15 +95,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -125,7 +117,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -138,15 +130,12 @@ $node_primary->safe_psql($test_db,
$node_primary->wait_for_replay_catchup($node_standby);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -160,7 +149,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -169,12 +158,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -182,7 +171,7 @@ $node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -193,14 +182,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -211,7 +200,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -227,7 +216,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -250,19 +239,15 @@ SELECT txid_current();
$node_primary->wait_for_replay_catchup($node_standby);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -277,7 +262,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -285,7 +270,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -309,8 +294,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -318,37 +302,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 0e0f27f14d..9762497146 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.32.1 (Apple Git-133)
On 2023-03-17 Fr 05:48, Daniel Gustafsson wrote:
On 15 Mar 2023, at 02:03, Andres Freund<andres@anarazel.de> wrote:
Returning a hash seems like a worse option since it will complicate callsites
which only want to know success/failure.Yea. Perhaps it's worth having a separate function for this? ->query_rc() or such?
If we are returning a hash then I agree it should be a separate function.
Maybe Andrew has input on which is the most Perl way of doing this.
I think the perlish way is use the `wantarray` function. Perl knows if
you're expecting a scalar return value or a list (which includes a hash).
return wantarray ? $retval : (list or hash);
A few more issues:
A common perl idiom is to start private routine names with an
underscore. so I'd rename wait_connect to _wait_connect;
Why is $restart_before_query a package/class level value instead of an
instance value? And why can we only ever set it to 1 but not back again?
Maybe we don't want to, but it looks odd.
If we are going to keep this as a separate package, then we should put
some code in the constructor to prevent it being called from elsewhere
than the Cluster package. e.g.
# this constructor should only be called from PostgreSQL::Test::Cluster
my ($package, $file, $line) = caller;
die "Forbidden caller of constructor: package: $package, file:
$file:$line"
unless $package eq 'PostgreSQL::Test::Cluster';
This should refer to the full class name:
+=item $node->background_psql($dbname, %params) => BackgroundPsql instance
Still reviewing ...
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On 17 Mar 2023, at 14:48, Andrew Dunstan <andrew@dunslane.net> wrote:
On 2023-03-17 Fr 05:48, Daniel Gustafsson wrote:On 15 Mar 2023, at 02:03, Andres Freund <andres@anarazel.de>
wrote:Returning a hash seems like a worse option since it will complicate callsites
which only want to know success/failure.Yea. Perhaps it's worth having a separate function for this? ->query_rc() or such?
If we are returning a hash then I agree it should be a separate function.
Maybe Andrew has input on which is the most Perl way of doing this.I think the perlish way is use the `wantarray` function. Perl knows if you're expecting a scalar return value or a list (which includes a hash).
return wantarray ? $retval : (list or hash);
Aha, TIL. That seems like precisely what we want.
A common perl idiom is to start private routine names with an underscore. so I'd rename wait_connect to _wait_connect;
There are quite a few routines documented as internal in Cluster.pm which don't
start with an underscore. Should we change them as well? I'm happy to prepare
a separate patch to address that if we want that.
Why is $restart_before_query a package/class level value instead of an instance value? And why can we only ever set it to 1 but not back again? Maybe we don't want to, but it looks odd.
It was mostly a POC to show what I meant with the functionality. I think there
should be a way to turn it off (set it to zero) even though I doubt it will be
used much.
If we are going to keep this as a separate package, then we should put some code in the constructor to prevent it being called from elsewhere than the Cluster package. e.g.
# this constructor should only be called from PostgreSQL::Test::Cluster
my ($package, $file, $line) = caller;die "Forbidden caller of constructor: package: $package, file: $file:$line"
unless $package eq 'PostgreSQL::Test::Cluster';
I don't have strong feelings about where to place this, but Cluster.pm is
already quite long so I see a small upside to keeping it separate to not make
that worse.
--
Daniel Gustafsson
On 2023-03-17 Fr 10:08, Daniel Gustafsson wrote:
A common perl idiom is to start private routine names with an underscore. so I'd rename wait_connect to _wait_connect;
There are quite a few routines documented as internal in Cluster.pm which don't
start with an underscore. Should we change them as well? I'm happy to prepare
a separate patch to address that if we want that.
Possibly. There are two concerns. First, make sure that they really are
private. Last time I looked I think I noticed at least one thing that
was alleged to be private but was called from a TAP script. Second,
unless we backpatch it there will be some drift between branches, which
can make backpatching things a bit harder. But by all means prep a patch
so we can see the scope of the issue.
Why is $restart_before_query a package/class level value instead of an instance value? And why can we only ever set it to 1 but not back again? Maybe we don't want to, but it looks odd.
It was mostly a POC to show what I meant with the functionality. I think there
should be a way to turn it off (set it to zero) even though I doubt it will be
used much.
A common idiom is to have a composite getter/setter method for object
properties something like this
sub settingname
{
my ($self, $arg) = @_;
$self->{settingname} = $arg if defined $arg;
return $self->{settingname};
}
If we are going to keep this as a separate package, then we should put some code in the constructor to prevent it being called from elsewhere than the Cluster package. e.g.
# this constructor should only be called from PostgreSQL::Test::Cluster
my ($package, $file, $line) = caller;die "Forbidden caller of constructor: package: $package, file: $file:$line"
unless $package eq 'PostgreSQL::Test::Cluster';I don't have strong feelings about where to place this, but Cluster.pm is
already quite long so I see a small upside to keeping it separate to not make
that worse.
Yeah, I can go along with that.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
On 2023-03-17 Fr 10:08, Daniel Gustafsson wrote:
Why is $restart_before_query a package/class level value instead of
an instance value? And why can we only ever set it to 1 but not back
again? Maybe we don't want to, but it looks odd.It was mostly a POC to show what I meant with the functionality. I think there
should be a way to turn it off (set it to zero) even though I doubt it will be
used much.A common idiom is to have a composite getter/setter method for object
properties something like thissub settingname
{
my ($self, $arg) = @_;
$self->{settingname} = $arg if defined $arg;
return $self->{settingname};
}
Or, if undef is a valid value:
sub settingname
{
my $self = shift;
$self->[settingname} = shift if @_;
return $self->{settingname};
}
- ilmari
On 2023-03-17 Fr 14:07, Dagfinn Ilmari Mannsåker wrote:
Andrew Dunstan<andrew@dunslane.net> writes:
On 2023-03-17 Fr 10:08, Daniel Gustafsson wrote:
Why is $restart_before_query a package/class level value instead of
an instance value? And why can we only ever set it to 1 but not back
again? Maybe we don't want to, but it looks odd.It was mostly a POC to show what I meant with the functionality. I think there
should be a way to turn it off (set it to zero) even though I doubt it will be
used much.A common idiom is to have a composite getter/setter method for object
properties something like thissub settingname
{
my ($self, $arg) = @_;
$self->{settingname} = $arg if defined $arg;
return $self->{settingname};
}Or, if undef is a valid value:
sub settingname
{
my $self = shift;
$self->[settingname} = shift if @_;
return $self->{settingname};
}
Yes, I agree that's better (modulo the bracket typo)
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Hi,
On 2023-03-17 12:25:14 -0400, Andrew Dunstan wrote:
On 2023-03-17 Fr 10:08, Daniel Gustafsson wrote:
If we are going to keep this as a separate package, then we should put some code in the constructor to prevent it being called from elsewhere than the Cluster package. e.g.
# this constructor should only be called from PostgreSQL::Test::Cluster
my ($package, $file, $line) = caller;
die "Forbidden caller of constructor: package: $package, file: $file:$line"
unless $package eq 'PostgreSQL::Test::Cluster';I don't have strong feelings about where to place this, but Cluster.pm is
already quite long so I see a small upside to keeping it separate to not make
that worse.Yeah, I can go along with that.
Cool - I'd prefer a separate file. I do find Cluster.pm somewhat unwieldy at
this point, and I susect that we'll end up with additional helpers around
BackgroundPsql.
Greetings,
Andres Freund
On 2023-03-17 Fr 18:58, Andres Freund wrote:
Hi,
On 2023-03-17 12:25:14 -0400, Andrew Dunstan wrote:
On 2023-03-17 Fr 10:08, Daniel Gustafsson wrote:
If we are going to keep this as a separate package, then we should put some code in the constructor to prevent it being called from elsewhere than the Cluster package. e.g.
# this constructor should only be called from PostgreSQL::Test::Cluster
my ($package, $file, $line) = caller;
die "Forbidden caller of constructor: package: $package, file: $file:$line"
unless $package eq 'PostgreSQL::Test::Cluster';I don't have strong feelings about where to place this, but Cluster.pm is
already quite long so I see a small upside to keeping it separate to not make
that worse.Yeah, I can go along with that.
Cool - I'd prefer a separate file. I do find Cluster.pm somewhat unwieldy at
this point, and I susect that we'll end up with additional helpers around
BackgroundPsql.
Yeah. BTW, a better test than the one above would be
$package->isa("PostgreSQL::Test::Cluster")
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On 18 Mar 2023, at 23:07, Andrew Dunstan <andrew@dunslane.net> wrote:
BTW, a better test than the one above would be
$package->isa("PostgreSQL::Test::Cluster")
Attached is a quick updated v3 of the patch which, to the best of my Perl
abilities, tries to address the comments raised here.
--
Daniel Gustafsson
Attachments:
v3-0001-Refactor-background-psql-TAP-functions.patchapplication/octet-stream; name=v3-0001-Refactor-background-psql-TAP-functions.patch; x-unix-mode=0644Download
From 28cd64a0a70d8859b0764844cc02e4b3bace4e9f Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 23 Mar 2023 23:32:28 +0100
Subject: [PATCH v3] Refactor background psql TAP functions
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
contrib/amcheck/t/003_cic_2pc.pl | 70 ++---
src/bin/psql/t/010_tab_completion.pl | 27 +-
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 295 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 83 +----
src/test/recovery/t/031_recovery_conflict.pl | 101 ++----
src/test/subscription/t/015_stream.pl | 52 +--
6 files changed, 384 insertions(+), 244 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf96..5323ed11ae 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+my $main_h = $node->background_psql('postgres');
+
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
-
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+));
+
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 55a88f9812..ce609f450e 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -92,14 +92,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +102,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -451,10 +441,7 @@ check_completion(
clear_line();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 0000000000..742571b515
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,295 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::BackgroundPsql - class for controlling background psql processes
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::Cluster;
+
+ my $node = PostgreSQL::Test::Cluster->new('mynode');
+
+ # Create a data directory with initdb
+ $node->init();
+
+ # Start the PostgreSQL server
+ $node->start();
+
+ # Create and start an interactive psql session
+ my $isession = $node->interactive_psql('postgres');
+
+ # Run a query and get the output as seen by psql
+ my $ret = $isession->query("SELECT 1");
+ # Run a backslash command and wait until the prompt returns
+ $isession->query_until(qr/postgres #/, "\\d foo\n");
+ # Close the session and exit psql
+ $isession->quit;
+
+ # Create and start a background psql session
+ my $bsession = $node->background_psql('postgres');
+
+ # Run a query which is guaranteed to not return in case it fails
+ $bsession->query_safe("SELECT 1");
+ # Initiate a command which can be expected to terminate at a later stage
+ $bsession->query_until(qr/start/, q(
+ \echo start
+ CREATE INDEX CONCURRENTLY idx ON t(a);
+ ));
+ # Close the session and exit psql
+ $bsession->quit;
+
+=head1 DESCRIPTION
+
+PostgreSQL::Test::BackgroundPsql contains functionality for controlling a
+background or interactive psql session operating on a PostgreSQL node
+initiated by PostgreSQL::Test::Cluster.
+
+=cut
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+=pod
+
+=head1 METHODS
+
+=over
+
+=item PostgreSQL::Test::BackroundPsql->new(interactive, @params)
+
+Builds a new object of class C<PostgreSQL::Test::BackgroundPsql> for either
+an interactive or background session and starts it. If C<interactive> is
+true then a PTY will be attached. C<psql_params> should contain the full
+command to run psql with all desired parameters and a complete connection
+string.
+
+=cut
+
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => '', 'query_timer_restart' => undef};
+ my $run;
+
+ # This constructor should only be called from PostgreSQL::Test::Cluster
+ my ($package, $file, $line) = caller;
+ die "Forbidden caller of constructor: package: $package, file: $file:$line"
+ unless $package->isa('PostgreSQL::Test::Cluster');
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->_wait_connect();
+
+ return $self;
+}
+
+# Internal routine for awaiting psql starting up and being ready to consume
+# input.
+sub _wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+=pod
+
+=item quit
+
+Close the session and clean up resources. Each test run must be closed with
+C<quit>.
+
+=cut
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+=pod
+
+=item $session->reconnect_and_clear
+
+Terminate the current session and connect again.
+
+=cut
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->_wait_connect()
+}
+
+=pod
+
+=item $session->query()
+
+Executes a query in the current session and returns the output in scalar
+context and (output, error) in list context where error is 1 in case there
+was output generated on stderr when executing the query.
+
+=cut
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ my $output;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+
+ # Feed the query to psql's stdin, followed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $output = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $output =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ $ret = $self->{stderr} eq "" ? 0 : 1;
+
+ return wantarray ? ( $output, $ret ) : $output;
+}
+
+=pod
+
+=item $session->query_safe()
+
+Wrapper around C<query> which errors out if the query failed to execute.
+Query failure is determined by it producing output on stderr.
+
+=cut
+
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_until(until, query)
+
+Issue C<query> and wait for C<until> appearing in the query output rather than
+waiting for query completion. C<query> needs to end with newline and semicolon
+(if applicable) for psql to process the input.
+
+=cut
+
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->restart_timer_before_query()
+
+Configures the timer to be restarted before each query such that the defined
+timeout is valid per query rather than per test run.
+
+=cut
+
+sub set_query_timer_restart
+{
+ my $self = shift;
+
+ $self->{query_timer_restart} = shift if @_;
+ return $self->{query_timer_restart};
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 3e2a27fb71..6f1047e187 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => PostgreSQL::Test::BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2029,45 +2024,18 @@ sub background_psql
$params{on_error_stop} = 1 unless defined $params{on_error_stop};
- push @psql_params, '-v', 'ON_ERROR_STOP=1' if $params{on_error_stop};
- push @psql_params, @{ $params{extra_params} }
- if defined $params{extra_params};
-
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2043,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2061,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2072,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 84375faccb..66673420c8 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -67,14 +67,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db);
my $expected_conflicts = 0;
@@ -102,15 +95,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -125,7 +117,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -138,15 +130,12 @@ $node_primary->safe_psql($test_db,
$node_primary->wait_for_replay_catchup($node_standby);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -160,7 +149,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -169,12 +158,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -182,7 +171,7 @@ $node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -193,14 +182,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -211,7 +200,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -227,7 +216,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -250,19 +239,15 @@ SELECT txid_current();
$node_primary->wait_for_replay_catchup($node_standby);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -277,7 +262,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -285,7 +270,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -309,8 +294,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -318,37 +302,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 0e0f27f14d..9762497146 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.32.1 (Apple Git-133)
The attached v4 fixes some incorrect documentation (added by me in v3), and
fixes that background_psql() didn't honor on_error_stop and extraparams passed
by the user. I've also added a commit which implements the \password test from
the SCRAM iteration count patchset as well as cleaned up a few IPC::Run
includes from test scripts.
--
Daniel Gustafsson
Attachments:
v4-0002-Add-test-SCRAM-iteration-changes-with-psql-passwo.patchapplication/octet-stream; name=v4-0002-Add-test-SCRAM-iteration-changes-with-psql-passwo.patch; x-unix-mode=0644Download
From e35dee9bee53dbfb22b7bfede63ca077fad20bd2 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Fri, 31 Mar 2023 14:52:41 +0200
Subject: [PATCH v4 2/2] Add test SCRAM iteration changes with psql \password
A version of this test was included in the original patch for altering
SCRAM iteration count, but was omitted due to how interactive psql TAP
sessions worked before being refactored.
Discussion: https://postgr.es/m/xxx
---
src/test/authentication/t/001_password.pl | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 00857fdae5..4e0eb5fdbe 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -101,6 +101,25 @@ my $res = $node->safe_psql('postgres',
WHERE rolname = 'scram_role_iter'");
is($res, 'SCRAM-SHA-256$1024:', 'scram_iterations in server side ROLE');
+# Alter the password on the created role using \password in psql to ensure
+# that clientside password changes use the scram_iterations value when
+# calculating SCRAM secrets.
+my $session = $node->interactive_psql('postgres');
+
+$session->set_query_timer_restart();
+$session->query("SET password_encryption='scram-sha-256';");
+$session->query("SET scram_iterations=42;");
+$session->query_until(qr/Enter new password/, "\\password scram_role_iter\n");
+$session->query_until(qr/Enter it again/, "pass\n");
+$session->query_until(qr/postgres=# /, "pass\n");
+$session->quit;
+
+$res = $node->safe_psql('postgres',
+ "SELECT substr(rolpassword,1,17)
+ FROM pg_authid
+ WHERE rolname = 'scram_role_iter'");
+is($res, 'SCRAM-SHA-256$42:', 'scram_iterations in psql \password command');
+
# Create a database to test regular expression.
$node->safe_psql('postgres', "CREATE database regex_testdb;");
--
2.32.1 (Apple Git-133)
v4-0001-Refactor-background-psql-TAP-functions.patchapplication/octet-stream; name=v4-0001-Refactor-background-psql-TAP-functions.patch; x-unix-mode=0644Download
From 5fd32a62974732cf396c6118619e0cfa548d22b0 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 23 Mar 2023 23:32:28 +0100
Subject: [PATCH v4 1/2] Refactor background psql TAP functions
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
contrib/amcheck/t/003_cic_2pc.pl | 70 ++---
src/bin/psql/t/010_tab_completion.pl | 28 +-
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 296 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 79 +----
.../t/010_logical_decoding_timelines.pl | 1 -
src/test/recovery/t/031_recovery_conflict.pl | 101 ++----
src/test/subscription/t/015_stream.pl | 52 +--
7 files changed, 385 insertions(+), 242 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf96..5323ed11ae 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+my $main_h = $node->background_psql('postgres');
+
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
-
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+));
+
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 55a88f9812..576b81958e 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -7,7 +7,6 @@ use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-use IPC::Run qw(pump finish timer);
use Data::Dumper;
# Do nothing unless Makefile has told us that the build is --with-readline.
@@ -92,14 +91,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +101,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -451,10 +440,7 @@ check_completion(
clear_line();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 0000000000..24d9f6c5cd
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,296 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::BackgroundPsql - class for controlling background psql processes
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::Cluster;
+
+ my $node = PostgreSQL::Test::Cluster->new('mynode');
+
+ # Create a data directory with initdb
+ $node->init();
+
+ # Start the PostgreSQL server
+ $node->start();
+
+ # Create and start an interactive psql session
+ my $isession = $node->interactive_psql('postgres');
+ # Apply timeout per query rather than per session
+ $isession->set_query_timer_restart();
+ # Run a query and get the output as seen by psql
+ my $ret = $isession->query("SELECT 1");
+ # Run a backslash command and wait until the prompt returns
+ $isession->query_until(qr/postgres #/, "\\d foo\n");
+ # Close the session and exit psql
+ $isession->quit;
+
+ # Create and start a background psql session
+ my $bsession = $node->background_psql('postgres');
+
+ # Run a query which is guaranteed to not return in case it fails
+ $bsession->query_safe("SELECT 1");
+ # Initiate a command which can be expected to terminate at a later stage
+ $bsession->query_until(qr/start/, q(
+ \echo start
+ CREATE INDEX CONCURRENTLY idx ON t(a);
+ ));
+ # Close the session and exit psql
+ $bsession->quit;
+
+=head1 DESCRIPTION
+
+PostgreSQL::Test::BackgroundPsql contains functionality for controlling a
+background or interactive psql session operating on a PostgreSQL node
+initiated by PostgreSQL::Test::Cluster.
+
+=cut
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+=pod
+
+=head1 METHODS
+
+=over
+
+=item PostgreSQL::Test::BackroundPsql->new(interactive, @params)
+
+Builds a new object of class C<PostgreSQL::Test::BackgroundPsql> for either
+an interactive or background session and starts it. If C<interactive> is
+true then a PTY will be attached. C<psql_params> should contain the full
+command to run psql with all desired parameters and a complete connection
+string.
+
+=cut
+
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => '', 'query_timer_restart' => undef};
+ my $run;
+
+ # This constructor should only be called from PostgreSQL::Test::Cluster
+ my ($package, $file, $line) = caller;
+ die "Forbidden caller of constructor: package: $package, file: $file:$line"
+ unless $package->isa('PostgreSQL::Test::Cluster');
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->_wait_connect();
+
+ return $self;
+}
+
+# Internal routine for awaiting psql starting up and being ready to consume
+# input.
+sub _wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+=pod
+
+=item quit
+
+Close the session and clean up resources. Each test run must be closed with
+C<quit>.
+
+=cut
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+=pod
+
+=item $session->reconnect_and_clear
+
+Terminate the current session and connect again.
+
+=cut
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->_wait_connect()
+}
+
+=pod
+
+=item $session->query()
+
+Executes a query in the current session and returns the output in scalar
+context and (output, error) in list context where error is 1 in case there
+was output generated on stderr when executing the query.
+
+=cut
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ my $output;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+
+ # Feed the query to psql's stdin, followed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $output = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $output =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ $ret = $self->{stderr} eq "" ? 0 : 1;
+
+ return wantarray ? ( $output, $ret ) : $output;
+}
+
+=pod
+
+=item $session->query_safe()
+
+Wrapper around C<query> which errors out if the query failed to execute.
+Query failure is determined by it producing output on stderr.
+
+=cut
+
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_until(until, query)
+
+Issue C<query> and wait for C<until> appearing in the query output rather than
+waiting for query completion. C<query> needs to end with newline and semicolon
+(if applicable) for psql to process the input.
+
+=cut
+
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->set_query_timer_restart()
+
+Configures the timer to be restarted before each query such that the defined
+timeout is valid per query rather than per test run.
+
+=cut
+
+sub set_query_timer_restart
+{
+ my $self = shift;
+
+ $self->{query_timer_restart} = shift if @_;
+ return $self->{query_timer_restart};
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index a3aef8b5e9..50e0961511 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => PostgreSQL::Test::BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2033,41 +2028,18 @@ sub background_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2047,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2065,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2076,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/010_logical_decoding_timelines.pl b/src/test/recovery/t/010_logical_decoding_timelines.pl
index eb1a3b6ef8..993f654a9b 100644
--- a/src/test/recovery/t/010_logical_decoding_timelines.pl
+++ b/src/test/recovery/t/010_logical_decoding_timelines.pl
@@ -28,7 +28,6 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use File::Copy;
-use IPC::Run ();
use Scalar::Util qw(blessed);
my ($stdout, $stderr, $ret);
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 84375faccb..66673420c8 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -67,14 +67,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db);
my $expected_conflicts = 0;
@@ -102,15 +95,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -125,7 +117,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -138,15 +130,12 @@ $node_primary->safe_psql($test_db,
$node_primary->wait_for_replay_catchup($node_standby);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -160,7 +149,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -169,12 +158,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -182,7 +171,7 @@ $node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -193,14 +182,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -211,7 +200,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -227,7 +216,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -250,19 +239,15 @@ SELECT txid_current();
$node_primary->wait_for_replay_catchup($node_standby);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -277,7 +262,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -285,7 +270,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -309,8 +294,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -318,37 +302,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 0e0f27f14d..9762497146 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.32.1 (Apple Git-133)
On 31 Mar 2023, at 22:33, Daniel Gustafsson <daniel@yesql.se> wrote:
The attached v4 fixes some incorrect documentation (added by me in v3), and
fixes that background_psql() didn't honor on_error_stop and extraparams passed
by the user. I've also added a commit which implements the \password test from
the SCRAM iteration count patchset as well as cleaned up a few IPC::Run
includes from test scripts.
And a v5 to fix a test failure in recovery tests.
--
Daniel Gustafsson
Attachments:
v5-0001-Refactor-background-psql-TAP-functions.patchapplication/octet-stream; name=v5-0001-Refactor-background-psql-TAP-functions.patch; x-unix-mode=0644Download
From e108e4568eb2ce8443bce52b971da22f6f5e3615 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 23 Mar 2023 23:32:28 +0100
Subject: [PATCH v5 1/2] Refactor background psql TAP functions
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
contrib/amcheck/t/003_cic_2pc.pl | 70 ++---
src/bin/psql/t/010_tab_completion.pl | 28 +-
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 296 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 79 +----
.../t/010_logical_decoding_timelines.pl | 1 -
src/test/recovery/t/031_recovery_conflict.pl | 102 ++----
src/test/subscription/t/015_stream.pl | 52 +--
7 files changed, 386 insertions(+), 242 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf96..5323ed11ae 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+my $main_h = $node->background_psql('postgres');
+
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
-
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+));
+
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 55a88f9812..576b81958e 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -7,7 +7,6 @@ use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-use IPC::Run qw(pump finish timer);
use Data::Dumper;
# Do nothing unless Makefile has told us that the build is --with-readline.
@@ -92,14 +91,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +101,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -451,10 +440,7 @@ check_completion(
clear_line();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 0000000000..24d9f6c5cd
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,296 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::BackgroundPsql - class for controlling background psql processes
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::Cluster;
+
+ my $node = PostgreSQL::Test::Cluster->new('mynode');
+
+ # Create a data directory with initdb
+ $node->init();
+
+ # Start the PostgreSQL server
+ $node->start();
+
+ # Create and start an interactive psql session
+ my $isession = $node->interactive_psql('postgres');
+ # Apply timeout per query rather than per session
+ $isession->set_query_timer_restart();
+ # Run a query and get the output as seen by psql
+ my $ret = $isession->query("SELECT 1");
+ # Run a backslash command and wait until the prompt returns
+ $isession->query_until(qr/postgres #/, "\\d foo\n");
+ # Close the session and exit psql
+ $isession->quit;
+
+ # Create and start a background psql session
+ my $bsession = $node->background_psql('postgres');
+
+ # Run a query which is guaranteed to not return in case it fails
+ $bsession->query_safe("SELECT 1");
+ # Initiate a command which can be expected to terminate at a later stage
+ $bsession->query_until(qr/start/, q(
+ \echo start
+ CREATE INDEX CONCURRENTLY idx ON t(a);
+ ));
+ # Close the session and exit psql
+ $bsession->quit;
+
+=head1 DESCRIPTION
+
+PostgreSQL::Test::BackgroundPsql contains functionality for controlling a
+background or interactive psql session operating on a PostgreSQL node
+initiated by PostgreSQL::Test::Cluster.
+
+=cut
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+=pod
+
+=head1 METHODS
+
+=over
+
+=item PostgreSQL::Test::BackroundPsql->new(interactive, @params)
+
+Builds a new object of class C<PostgreSQL::Test::BackgroundPsql> for either
+an interactive or background session and starts it. If C<interactive> is
+true then a PTY will be attached. C<psql_params> should contain the full
+command to run psql with all desired parameters and a complete connection
+string.
+
+=cut
+
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => '', 'query_timer_restart' => undef};
+ my $run;
+
+ # This constructor should only be called from PostgreSQL::Test::Cluster
+ my ($package, $file, $line) = caller;
+ die "Forbidden caller of constructor: package: $package, file: $file:$line"
+ unless $package->isa('PostgreSQL::Test::Cluster');
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->_wait_connect();
+
+ return $self;
+}
+
+# Internal routine for awaiting psql starting up and being ready to consume
+# input.
+sub _wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+=pod
+
+=item quit
+
+Close the session and clean up resources. Each test run must be closed with
+C<quit>.
+
+=cut
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+=pod
+
+=item $session->reconnect_and_clear
+
+Terminate the current session and connect again.
+
+=cut
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->_wait_connect()
+}
+
+=pod
+
+=item $session->query()
+
+Executes a query in the current session and returns the output in scalar
+context and (output, error) in list context where error is 1 in case there
+was output generated on stderr when executing the query.
+
+=cut
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ my $output;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+
+ # Feed the query to psql's stdin, followed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $output = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $output =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ $ret = $self->{stderr} eq "" ? 0 : 1;
+
+ return wantarray ? ( $output, $ret ) : $output;
+}
+
+=pod
+
+=item $session->query_safe()
+
+Wrapper around C<query> which errors out if the query failed to execute.
+Query failure is determined by it producing output on stderr.
+
+=cut
+
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_until(until, query)
+
+Issue C<query> and wait for C<until> appearing in the query output rather than
+waiting for query completion. C<query> needs to end with newline and semicolon
+(if applicable) for psql to process the input.
+
+=cut
+
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->set_query_timer_restart()
+
+Configures the timer to be restarted before each query such that the defined
+timeout is valid per query rather than per test run.
+
+=cut
+
+sub set_query_timer_restart
+{
+ my $self = shift;
+
+ $self->{query_timer_restart} = shift if @_;
+ return $self->{query_timer_restart};
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index a3aef8b5e9..50e0961511 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => PostgreSQL::Test::BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2033,41 +2028,18 @@ sub background_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2047,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2065,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2076,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/010_logical_decoding_timelines.pl b/src/test/recovery/t/010_logical_decoding_timelines.pl
index eb1a3b6ef8..993f654a9b 100644
--- a/src/test/recovery/t/010_logical_decoding_timelines.pl
+++ b/src/test/recovery/t/010_logical_decoding_timelines.pl
@@ -28,7 +28,6 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use File::Copy;
-use IPC::Run ();
use Scalar::Util qw(blessed);
my ($stdout, $stderr, $ret);
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 84375faccb..e29bc6c181 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -67,14 +67,8 @@ $node_primary->wait_for_replay_catchup($node_standby);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db,
+ on_error_stop => 0);
my $expected_conflicts = 0;
@@ -102,15 +96,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -125,7 +118,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -138,15 +131,12 @@ $node_primary->safe_psql($test_db,
$node_primary->wait_for_replay_catchup($node_standby);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -160,7 +150,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -169,12 +159,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -182,7 +172,7 @@ $node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -193,14 +183,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -211,7 +201,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -227,7 +217,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -250,19 +240,15 @@ SELECT txid_current();
$node_primary->wait_for_replay_catchup($node_standby);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -277,7 +263,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -285,7 +271,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -309,8 +295,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -318,37 +303,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 0e0f27f14d..9762497146 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.32.1 (Apple Git-133)
v5-0002-Add-test-SCRAM-iteration-changes-with-psql-passwo.patchapplication/octet-stream; name=v5-0002-Add-test-SCRAM-iteration-changes-with-psql-passwo.patch; x-unix-mode=0644Download
From cf6634891d73653fbb97b55045101c309f913ceb Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Fri, 31 Mar 2023 14:52:41 +0200
Subject: [PATCH v5 2/2] Add test SCRAM iteration changes with psql \password
A version of this test was included in the original patch for altering
SCRAM iteration count, but was omitted due to how interactive psql TAP
sessions worked before being refactored.
Discussion: https://postgr.es/m/xxx
---
src/test/authentication/t/001_password.pl | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 00857fdae5..4e0eb5fdbe 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -101,6 +101,25 @@ my $res = $node->safe_psql('postgres',
WHERE rolname = 'scram_role_iter'");
is($res, 'SCRAM-SHA-256$1024:', 'scram_iterations in server side ROLE');
+# Alter the password on the created role using \password in psql to ensure
+# that clientside password changes use the scram_iterations value when
+# calculating SCRAM secrets.
+my $session = $node->interactive_psql('postgres');
+
+$session->set_query_timer_restart();
+$session->query("SET password_encryption='scram-sha-256';");
+$session->query("SET scram_iterations=42;");
+$session->query_until(qr/Enter new password/, "\\password scram_role_iter\n");
+$session->query_until(qr/Enter it again/, "pass\n");
+$session->query_until(qr/postgres=# /, "pass\n");
+$session->quit;
+
+$res = $node->safe_psql('postgres',
+ "SELECT substr(rolpassword,1,17)
+ FROM pg_authid
+ WHERE rolname = 'scram_role_iter'");
+is($res, 'SCRAM-SHA-256$42:', 'scram_iterations in psql \password command');
+
# Create a database to test regular expression.
$node->safe_psql('postgres', "CREATE database regex_testdb;");
--
2.32.1 (Apple Git-133)
Hi,
On 2023-04-02 22:24:16 +0200, Daniel Gustafsson wrote:
And a v5 to fix a test failure in recovery tests.
Thanks for workin gon this!
There's this XXX that I added:
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});- $in .= q{ - COMMIT; - \q - }; - $h->finish; # errors make the next test fail, so ignore them here + $h->query_safe('COMMIT'); + $h->quit; + # XXX: Not sure what this means + # errors make the next test fail, so ignore them here$node_publisher->wait_for_catchup($appname);
I still don't know what that comment is supposed to mean, unfortunately.
Greetings,
Andres Freund
On 2 Apr 2023, at 23:37, Andres Freund <andres@anarazel.de> wrote:
There's this XXX that I added:
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});- $in .= q{ - COMMIT; - \q - }; - $h->finish; # errors make the next test fail, so ignore them here + $h->query_safe('COMMIT'); + $h->quit; + # XXX: Not sure what this means + # errors make the next test fail, so ignore them here$node_publisher->wait_for_catchup($appname);
I still don't know what that comment is supposed to mean, unfortunately.
My reading of it is that it's ignoring any croak errors which IPC::Run might
throw if ->finish() isn't able to reap the psql process which had the \q.
I've added Amit who committed it in 216a784829c on cc: to see if he remembers
the comment in question and can shed some light. Skimming the linked thread
yields no immediate clues.
--
Daniel Gustafsson
Unless there are objections I plan to get this in before the freeze, in order
to have better interactive tests starting with 16. With a little bit of
documentation polish I think it's ready.
--
Daniel Gustafsson
On 5 Apr 2023, at 23:44, Daniel Gustafsson <daniel@yesql.se> wrote:
Unless there are objections I plan to get this in before the freeze, in order
to have better interactive tests starting with 16. With a little bit of
documentation polish I think it's ready.
When looking at the CFBot failure on Linux and Windows (not on macOS) I noticed
that it was down to the instance lacking IO::Pty.
[19:59:12.609](1.606s) ok 1 - scram_iterations in server side ROLE
Can't locate IO/Pty.pm in @INC (you may need to install the IO::Pty module) (@INC contains: /tmp/cirrus-ci-build/src/test/perl /tmp/cirrus-ci-build/src/test/authentication /etc/perl /usr/local/lib/i386-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/i386-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/i386-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl) at /usr/share/perl5/IPC/Run.pm line 1828.
Skimming the VM creation [0]https://github.com/anarazel/pg-vm-images/blob/main/scripts/linux_debian_install_deps.sh it seems like it should be though? On macOS the
module is installed inside Cirrus and the test runs fine.
I don't think we should go ahead with a patch that refactors interactive_psql
only to SKIP over it in CI (which is what the tab_completion test does now), so
let's wait until we have that sorted before going ahead.
--
Daniel Gustafsson
[0]: https://github.com/anarazel/pg-vm-images/blob/main/scripts/linux_debian_install_deps.sh
On 2023-04-07 Fr 09:32, Daniel Gustafsson wrote:
On 5 Apr 2023, at 23:44, Daniel Gustafsson<daniel@yesql.se> wrote:
Unless there are objections I plan to get this in before the freeze, in order
to have better interactive tests starting with 16. With a little bit of
documentation polish I think it's ready.When looking at the CFBot failure on Linux and Windows (not on macOS) I noticed
that it was down to the instance lacking IO::Pty.[19:59:12.609](1.606s) ok 1 - scram_iterations in server side ROLE
Can't locate IO/Pty.pm in @INC (you may need to install the IO::Pty module) (@INC contains: /tmp/cirrus-ci-build/src/test/perl /tmp/cirrus-ci-build/src/test/authentication /etc/perl /usr/local/lib/i386-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/i386-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/i386-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl) at /usr/share/perl5/IPC/Run.pm line 1828.Skimming the VM creation [0] it seems like it should be though? On macOS the
module is installed inside Cirrus and the test runs fine.I don't think we should go ahead with a patch that refactors interactive_psql
only to SKIP over it in CI (which is what the tab_completion test does now), so
let's wait until we have that sorted before going ahead.
It should probably be added to config/check_modules.pl if we're going to
use it, but it seems to be missing for Strawberry Perl and msys/ucrt64
perl and I'm not sure how easy it will be to add there. It would
certainly add an installation burden for test instances at the very least.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Hi,
On 2023-04-07 15:32:12 +0200, Daniel Gustafsson wrote:
On 5 Apr 2023, at 23:44, Daniel Gustafsson <daniel@yesql.se> wrote:
Unless there are objections I plan to get this in before the freeze, in order
to have better interactive tests starting with 16. With a little bit of
documentation polish I think it's ready.When looking at the CFBot failure on Linux and Windows (not on macOS) I noticed
that it was down to the instance lacking IO::Pty.[19:59:12.609](1.606s) ok 1 - scram_iterations in server side ROLE
Can't locate IO/Pty.pm in @INC (you may need to install the IO::Pty module) (@INC contains: /tmp/cirrus-ci-build/src/test/perl /tmp/cirrus-ci-build/src/test/authentication /etc/perl /usr/local/lib/i386-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/i386-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/i386-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl) at /usr/share/perl5/IPC/Run.pm line 1828.Skimming the VM creation [0] it seems like it should be though?
Note it just fails on the 32build, not the 64bit build. Unfortunately I don't
think debian's multiarch in bullseye support installing enough of perl in
32bit and 64bit.
We can't have a hard dependency on non-default modules like IO::Pty anyway, so
the test needs to skip if it's not available.
On windows IO::Pty can't be installed, IIRC.
I don't think we should go ahead with a patch that refactors interactive_psql
only to SKIP over it in CI (which is what the tab_completion test does now), so
let's wait until we have that sorted before going ahead.
Maybe I am a bit confused, but isn't that just an existing requirement? Why
would we expect this patchset to change what dependencies use of
interactive_psql() has?
Greetings,
Andres Freund
Hi,
On 2023-04-07 10:55:19 -0400, Andrew Dunstan wrote:
It should probably be added to config/check_modules.pl if we're going to use
it, but it seems to be missing for Strawberry Perl and msys/ucrt64 perl and
I'm not sure how easy it will be to add there. It would certainly add an
installation burden for test instances at the very least.
The last time I tried, it can't be installed on windows with cpan either, the
module simply doesn't have the necessary windows bits - likely because
traditionally windows didn't really have ptys. I think some stuff has been
added, but it probably would still require a bunch of portability work.
Note that we normally don't even build with readline support on windows - so
there's not really much point in using IO::Pty there. While I've gotten that
to work manually not too long ago, it's still manual and not documented etc.
Afaict the failures are purely about patch 2, not 1, right?
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
On 2023-04-07 15:32:12 +0200, Daniel Gustafsson wrote:
I don't think we should go ahead with a patch that refactors interactive_psql
only to SKIP over it in CI (which is what the tab_completion test does now), so
let's wait until we have that sorted before going ahead.
Maybe I am a bit confused, but isn't that just an existing requirement? Why
would we expect this patchset to change what dependencies use of
interactive_psql() has?
It is an existing requirement, but only for a test that's not too
critical. If interactive_psql starts getting used for more interesting
things, we might be sad that the coverage is weak.
Having said that, weak coverage is better than no coverage. I don't
think this point should be a show-stopper for committing.
regards, tom lane
On 7 Apr 2023, at 16:58, Andres Freund <andres@anarazel.de> wrote:
Note it just fails on the 32build, not the 64bit build. Unfortunately I don't
think debian's multiarch in bullseye support installing enough of perl in
32bit and 64bit.
I should probably avoid parsing logfiles with fever-induced brainfog, I
confused myself to think it was both =(
Maybe I am a bit confused, but isn't that just an existing requirement? Why
would we expect this patchset to change what dependencies use of
interactive_psql() has?
Correct, there is no change from the current implementation.
--
Daniel Gustafsson
Hi,
On 2023-04-07 11:52:37 -0400, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:
On 2023-04-07 15:32:12 +0200, Daniel Gustafsson wrote:
I don't think we should go ahead with a patch that refactors interactive_psql
only to SKIP over it in CI (which is what the tab_completion test does now), so
let's wait until we have that sorted before going ahead.Maybe I am a bit confused, but isn't that just an existing requirement? Why
would we expect this patchset to change what dependencies use of
interactive_psql() has?It is an existing requirement, but only for a test that's not too
critical. If interactive_psql starts getting used for more interesting
things, we might be sad that the coverage is weak.
I don't really expect it to be used for non-critical things - after all,
interactive_psql() also depends on psql being built with readline support,
which we traditionally don't have on windows... For most tasks background_psql
should suffice...
Having said that, weak coverage is better than no coverage. I don't
think this point should be a show-stopper for committing.
Yea.
One thing I wonder is whether we should have a central function for checking
if interactive_psql() is available, instead of copying 010_tab_completion.pl's
logic for it into multiple tests. But that could come later too.
Greetings,
Andres Freund
On 7 Apr 2023, at 17:04, Andres Freund <andres@anarazel.de> wrote:
Afaict the failures are purely about patch 2, not 1, right?
Correct. The attached v6 wraps the interactive_psql test in a SKIP block with
a conditional on IO::Pty being available.
--
Daniel Gustafsson
Attachments:
v6-0002-Test-SCRAM-iteration-changes-with-psql-password.patchapplication/octet-stream; name=v6-0002-Test-SCRAM-iteration-changes-with-psql-password.patch; x-unix-mode=0644Download
From 737d852e477c049194ecad28ecbc73dc5fc58518 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Fri, 31 Mar 2023 14:52:41 +0200
Subject: [PATCH v6 2/2] Test SCRAM iteration changes with psql \password
A version of this test was included in the original patch for altering
SCRAM iteration count, but was omitted due to how interactive psql TAP
sessions worked before being refactored.
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/F72E7BC7-189F-4B17-BF47-9735EB72C364@yesql.se
---
src/test/authentication/t/001_password.pl | 26 +++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 00857fdae5..f414a8ba90 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -101,6 +101,32 @@ my $res = $node->safe_psql('postgres',
WHERE rolname = 'scram_role_iter'");
is($res, 'SCRAM-SHA-256$1024:', 'scram_iterations in server side ROLE');
+# If we don't have IO::Pty, forget it, because IPC::Run depends on that
+# to support pty connections
+SKIP:
+{
+ skip "IO::Pty required", 1 unless eval { require IO::Pty; };
+
+ # Alter the password on the created role using \password in psql to ensure
+ # that clientside password changes use the scram_iterations value when
+ # calculating SCRAM secrets.
+ my $session = $node->interactive_psql('postgres');
+
+ $session->set_query_timer_restart();
+ $session->query("SET password_encryption='scram-sha-256';");
+ $session->query("SET scram_iterations=42;");
+ $session->query_until(qr/Enter new password/, "\\password scram_role_iter\n");
+ $session->query_until(qr/Enter it again/, "pass\n");
+ $session->query_until(qr/postgres=# /, "pass\n");
+ $session->quit;
+
+ $res = $node->safe_psql('postgres',
+ "SELECT substr(rolpassword,1,17)
+ FROM pg_authid
+ WHERE rolname = 'scram_role_iter'");
+ is($res, 'SCRAM-SHA-256$42:', 'scram_iterations in psql \password command');
+}
+
# Create a database to test regular expression.
$node->safe_psql('postgres', "CREATE database regex_testdb;");
--
2.32.1 (Apple Git-133)
v6-0001-Refactor-background-psql-TAP-functions.patchapplication/octet-stream; name=v6-0001-Refactor-background-psql-TAP-functions.patch; x-unix-mode=0644Download
From 6a6983135afe43e45ad18f6e5a6c11bef215a28f Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <dgustafsson@postgresql.org>
Date: Thu, 23 Mar 2023 23:32:28 +0100
Subject: [PATCH v6 1/2] Refactor background psql TAP functions
Author: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Discussion: https://postgr.es/m/20230130194350.zj5v467x4jgqt3d6@awork3.anarazel.de
Discussion: https://postgr.es/m/882122.1675109206@sss.pgh.pa.us
---
contrib/amcheck/t/003_cic_2pc.pl | 70 ++---
src/bin/psql/t/010_tab_completion.pl | 28 +-
.../perl/PostgreSQL/Test/BackgroundPsql.pm | 296 ++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 79 +----
.../t/010_logical_decoding_timelines.pl | 1 -
src/test/recovery/t/031_recovery_conflict.pl | 102 ++----
src/test/subscription/t/015_stream.pl | 52 +--
7 files changed, 386 insertions(+), 242 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index eabe6fcf96..5323ed11ae 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -36,63 +36,46 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
# statements.
#
-my $main_in = '';
-my $main_out = '';
-my $main_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $main_h =
- $node->background_psql('postgres', \$main_in, \$main_out,
- $main_timer, on_error_stop => 1);
-$main_in .= q(
+my $main_h = $node->background_psql('postgres');
+
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint1
-);
-pump $main_h until $main_out =~ /syncpoint1/ || $main_timer->is_expired;
-
-my $cic_in = '';
-my $cic_out = '';
-my $cic_timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $cic_h =
- $node->background_psql('postgres', \$cic_in, \$cic_out,
- $cic_timer, on_error_stop => 1);
-$cic_in .= q(
+));
+
+my $cic_h = $node->background_psql('postgres');
+
+$cic_h->query_until(qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $cic_h until $cic_out =~ /start/ || $cic_timer->is_expired;
+));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'a';
-);
+));
-$main_in .= q(
+$main_h->query_safe(q(
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint2
-);
-pump $main_h until $main_out =~ /syncpoint2/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
-\echo syncpoint3
-);
-pump $main_h until $main_out =~ /syncpoint3/ || $main_timer->is_expired;
+));
$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
-$main_in .= q(
+$main_h->query_safe(q(
PREPARE TRANSACTION 'c';
COMMIT PREPARED 'c';
-);
-$main_h->pump_nb;
+));
-$main_h->finish;
-$cic_h->finish;
+$main_h->quit;
+$cic_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');
@@ -113,22 +96,15 @@ PREPARE TRANSACTION 'persists_forever';
));
$node->restart;
-my $reindex_in = '';
-my $reindex_out = '';
-my $reindex_timer =
- IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-my $reindex_h =
- $node->background_psql('postgres', \$reindex_in, \$reindex_out,
- $reindex_timer, on_error_stop => 1);
-$reindex_in .= q(
+my $reindex_h = $node->background_psql('postgres');
+$reindex_h->query_until(qr/start/, q(
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
-);
-pump $reindex_h until $reindex_out =~ /start/ || $reindex_timer->is_expired;
+));
$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
-$reindex_h->finish;
+$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
diff --git a/src/bin/psql/t/010_tab_completion.pl b/src/bin/psql/t/010_tab_completion.pl
index 55a88f9812..576b81958e 100644
--- a/src/bin/psql/t/010_tab_completion.pl
+++ b/src/bin/psql/t/010_tab_completion.pl
@@ -7,7 +7,6 @@ use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
-use IPC::Run qw(pump finish timer);
use Data::Dumper;
# Do nothing unless Makefile has told us that the build is --with-readline.
@@ -92,14 +91,7 @@ print $FH "other stuff\n";
close $FH;
# fire up an interactive psql session
-my $in = '';
-my $out = '';
-
-my $timer = timer($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node->interactive_psql('postgres', \$in, \$out, $timer);
-
-like($out, qr/psql/, "print startup banner");
+my $h = $node->interactive_psql('postgres');
# Simple test case: type something and see if psql responds as expected
sub check_completion
@@ -109,15 +101,12 @@ sub check_completion
# report test failures from caller location
local $Test::Builder::Level = $Test::Builder::Level + 1;
- # reset output collector
- $out = "";
# restart per-command timer
- $timer->start($PostgreSQL::Test::Utils::timeout_default);
- # send the data to be sent
- $in .= $send;
- # wait ...
- pump $h until ($out =~ $pattern || $timer->is_expired);
- my $okay = ($out =~ $pattern && !$timer->is_expired);
+ $h->{timeout}->start($PostgreSQL::Test::Utils::timeout_default);
+
+ # send the data to be sent and wait for its result
+ my $out = $h->query_until($pattern, $send);
+ my $okay = ($out =~ $pattern && !$h->{timeout}->is_expired);
ok($okay, $annotation);
# for debugging, log actual output if it didn't match
local $Data::Dumper::Terse = 1;
@@ -451,10 +440,7 @@ check_completion(
clear_line();
# send psql an explicit \q to shut it down, else pty won't close properly
-$timer->start($PostgreSQL::Test::Utils::timeout_default);
-$in .= "\\q\n";
-finish $h or die "psql returned $?";
-$timer->reset;
+$h->quit or die "psql returned $?";
# done
$node->stop;
diff --git a/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
new file mode 100644
index 0000000000..24d9f6c5cd
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/BackgroundPsql.pm
@@ -0,0 +1,296 @@
+
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::BackgroundPsql - class for controlling background psql processes
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::Cluster;
+
+ my $node = PostgreSQL::Test::Cluster->new('mynode');
+
+ # Create a data directory with initdb
+ $node->init();
+
+ # Start the PostgreSQL server
+ $node->start();
+
+ # Create and start an interactive psql session
+ my $isession = $node->interactive_psql('postgres');
+ # Apply timeout per query rather than per session
+ $isession->set_query_timer_restart();
+ # Run a query and get the output as seen by psql
+ my $ret = $isession->query("SELECT 1");
+ # Run a backslash command and wait until the prompt returns
+ $isession->query_until(qr/postgres #/, "\\d foo\n");
+ # Close the session and exit psql
+ $isession->quit;
+
+ # Create and start a background psql session
+ my $bsession = $node->background_psql('postgres');
+
+ # Run a query which is guaranteed to not return in case it fails
+ $bsession->query_safe("SELECT 1");
+ # Initiate a command which can be expected to terminate at a later stage
+ $bsession->query_until(qr/start/, q(
+ \echo start
+ CREATE INDEX CONCURRENTLY idx ON t(a);
+ ));
+ # Close the session and exit psql
+ $bsession->quit;
+
+=head1 DESCRIPTION
+
+PostgreSQL::Test::BackgroundPsql contains functionality for controlling a
+background or interactive psql session operating on a PostgreSQL node
+initiated by PostgreSQL::Test::Cluster.
+
+=cut
+
+package PostgreSQL::Test::BackgroundPsql;
+
+use strict;
+use warnings;
+
+use Carp;
+use Config;
+use IPC::Run;
+use PostgreSQL::Test::Utils qw(pump_until);
+use Test::More;
+
+=pod
+
+=head1 METHODS
+
+=over
+
+=item PostgreSQL::Test::BackroundPsql->new(interactive, @params)
+
+Builds a new object of class C<PostgreSQL::Test::BackgroundPsql> for either
+an interactive or background session and starts it. If C<interactive> is
+true then a PTY will be attached. C<psql_params> should contain the full
+command to run psql with all desired parameters and a complete connection
+string.
+
+=cut
+
+sub new
+{
+ my $class = shift;
+ my ($interactive, $psql_params) = @_;
+ my $psql = {'stdin' => '', 'stdout' => '', 'stderr' => '', 'query_timer_restart' => undef};
+ my $run;
+
+ # This constructor should only be called from PostgreSQL::Test::Cluster
+ my ($package, $file, $line) = caller;
+ die "Forbidden caller of constructor: package: $package, file: $file:$line"
+ unless $package->isa('PostgreSQL::Test::Cluster');
+
+ $psql->{timeout} = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
+
+ if ($interactive)
+ {
+ $run = IPC::Run::start $psql_params,
+ '<pty<', \$psql->{stdin}, '>pty>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+ else
+ {
+ $run = IPC::Run::start $psql_params,
+ '<', \$psql->{stdin}, '>', \$psql->{stdout}, '2>', \$psql->{stderr},
+ $psql->{timeout};
+ }
+
+ $psql->{run} = $run;
+
+ my $self = bless $psql, $class;
+
+ $self->_wait_connect();
+
+ return $self;
+}
+
+# Internal routine for awaiting psql starting up and being ready to consume
+# input.
+sub _wait_connect
+{
+ my ($self) = @_;
+
+ # Request some output, and pump until we see it. This means that psql
+ # connection failures are caught here, relieving callers of the need to
+ # handle those. (Right now, we have no particularly good handling for
+ # errors anyway, but that might be added later.)
+ my $banner = "background_psql: ready";
+ $self->{stdin} .= "\\echo $banner\n";
+ $self->{run}->pump() until $self->{stdout} =~ /$banner/ || $self->{timeout}->is_expired;
+ $self->{stdout} = ''; # clear out banner
+
+ die "psql startup timed out" if $self->{timeout}->is_expired;
+}
+
+=pod
+
+=item quit
+
+Close the session and clean up resources. Each test run must be closed with
+C<quit>.
+
+=cut
+
+sub quit
+{
+ my ($self) = @_;
+
+ $self->{stdin} .= "\\q\n";
+
+ return $self->{run}->finish;
+}
+
+=pod
+
+=item $session->reconnect_and_clear
+
+Terminate the current session and connect again.
+
+=cut
+
+sub reconnect_and_clear
+{
+ my ($self) = @_;
+
+ # If psql isn't dead already, tell it to quit as \q, when already dead,
+ # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
+ $self->{run}->pump_nb();
+ if ($self->{run}->pumpable())
+ {
+ $self->{stdin} .= "\\q\n";
+ }
+ $self->{run}->finish;
+
+ # restart
+ $self->{run}->run();
+ $self->{stdin} = '';
+ $self->{stdout} = '';
+
+ $self->_wait_connect()
+}
+
+=pod
+
+=item $session->query()
+
+Executes a query in the current session and returns the output in scalar
+context and (output, error) in list context where error is 1 in case there
+was output generated on stderr when executing the query.
+
+=cut
+
+sub query
+{
+ my ($self, $query) = @_;
+ my $ret;
+ my $output;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ note "issuing query via background psql: $query";
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+
+ # Feed the query to psql's stdin, followed by \n (so psql processes the
+ # line), by a ; (so that psql issues the query, if it doesnt't include a ;
+ # itself), and a separator echoed with \echo, that we can wait on.
+ my $banner = "background_psql: QUERY_SEPARATOR";
+ $self->{stdin} .= "$query\n;\n\\echo $banner\n";
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, qr/$banner/);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+ $output = $self->{stdout};
+
+ # remove banner again, our caller doesn't care
+ $output =~ s/\n$banner$//s;
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ $ret = $self->{stderr} eq "" ? 0 : 1;
+
+ return wantarray ? ( $output, $ret ) : $output;
+}
+
+=pod
+
+=item $session->query_safe()
+
+Wrapper around C<query> which errors out if the query failed to execute.
+Query failure is determined by it producing output on stderr.
+
+=cut
+
+sub query_safe
+{
+ my ($self, $query) = @_;
+
+ my $ret = $self->query($query);
+
+ if ($self->{stderr} ne "")
+ {
+ die "query failed: $self->{stderr}";
+ }
+
+ return $ret;
+}
+
+=pod
+
+=item $session->query_until(until, query)
+
+Issue C<query> and wait for C<until> appearing in the query output rather than
+waiting for query completion. C<query> needs to end with newline and semicolon
+(if applicable) for psql to process the input.
+
+=cut
+
+sub query_until
+{
+ my ($self, $until, $query) = @_;
+ my $ret;
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ $self->{timeout}->start() if (defined($self->{query_timer_restart}));
+ $self->{stdin} .= $query;
+
+ pump_until($self->{run}, $self->{timeout}, \$self->{stdout}, $until);
+
+ die "psql query timed out" if $self->{timeout}->is_expired;
+
+ $ret = $self->{stdout};
+
+ # clear out output for the next query
+ $self->{stdout} = '';
+
+ return $ret;
+}
+
+=pod
+
+=item $session->set_query_timer_restart()
+
+Configures the timer to be restarted before each query such that the defined
+timeout is valid per query rather than per test run.
+
+=cut
+
+sub set_query_timer_restart
+{
+ my $self = shift;
+
+ $self->{query_timer_restart} = shift if @_;
+ return $self->{query_timer_restart};
+}
+
+1;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index a3aef8b5e9..50e0961511 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -113,6 +113,7 @@ use PostgreSQL::Test::RecursiveCopy;
use Socket;
use Test::More;
use PostgreSQL::Test::Utils ();
+use PostgreSQL::Test::BackgroundPsql ();
use Time::HiRes qw(usleep);
use Scalar::Util qw(blessed);
@@ -1966,18 +1967,12 @@ sub psql
=pod
-=item $node->background_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->background_psql($dbname, %params) => PostgreSQL::Test::BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object, which the
-caller may use to send input to B<psql>. The process's stdin is sourced from
-the $stdin scalar reference, and its stdout and stderr go to the $stdout
-scalar reference. This allows the caller to act on other parts of the system
-while idling this backend.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -1986,7 +1981,7 @@ Dies on failure to invoke psql, or if psql fails to connect. Errors occurring
later are the caller's problem. psql runs with on_error_stop by default so
that it will stop running sql and return 3 if passed SQL results in an error.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
=over
@@ -2012,7 +2007,7 @@ If given, it must be an array reference containing additional parameters to B<ps
sub background_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2033,41 +2028,18 @@ sub background_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<', $stdin, '>', $stdout, $timer;
-
- # Request some output, and pump until we see it. This means that psql
- # connection failures are caught here, relieving callers of the need to
- # handle those. (Right now, we have no particularly good handling for
- # errors anyway, but that might be added later.)
- my $banner = "background_psql: ready";
- $$stdin = "\\echo $banner\n";
- pump $harness until $$stdout =~ /$banner/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(0, \@psql_params);
}
=pod
-=item $node->interactive_psql($dbname, \$stdin, \$stdout, $timer, %params) => harness
+=item $node->interactive_psql($dbname, %params) => BackgroundPsql instance
-Invoke B<psql> on B<$dbname> and return an IPC::Run harness object,
-which the caller may use to send interactive input to B<psql>.
-The process's stdin is sourced from the $stdin scalar reference,
-and its stdout and stderr go to the $stdout scalar reference.
-ptys are used so that psql thinks it's being called interactively.
+Invoke B<psql> on B<$dbname> and return a BackgroundPsql object, which the
+caller may use to send interactive input to B<psql>.
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to set the timeout length (usually
-$PostgreSQL::Test::Utils::timeout_default), and to restart the timer after
-each command if the timeout is per-command.
+A default timeout of $PostgreSQL::Test::Utils::timeout_default is set up,
+which can modified later.
psql is invoked in tuples-only unaligned mode with reading of B<.psqlrc>
disabled. That may be overridden by passing extra psql parameters.
@@ -2075,7 +2047,7 @@ disabled. That may be overridden by passing extra psql parameters.
Dies on failure to invoke psql, or if psql fails to connect.
Errors occurring later are the caller's problem.
-Be sure to "finish" the harness when done with it.
+Be sure to "quit" the returned object when done with it.
The only extra parameter currently accepted is
@@ -2093,7 +2065,7 @@ This requires IO::Pty in addition to IPC::Run.
sub interactive_psql
{
- my ($self, $dbname, $stdin, $stdout, $timer, %params) = @_;
+ my ($self, $dbname, %params) = @_;
local %ENV = $self->_get_env();
@@ -2104,26 +2076,7 @@ sub interactive_psql
push @psql_params, @{ $params{extra_params} }
if defined $params{extra_params};
- # Ensure there is no data waiting to be sent:
- $$stdin = "" if ref($stdin);
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@psql_params,
- '<pty<', $stdin, '>pty>', $stdout, $timer;
-
- # Pump until we see psql's help banner. This ensures that callers
- # won't write anything to the pty before it's ready, avoiding an
- # implementation issue in IPC::Run. Also, it means that psql
- # connection failures are caught here, relieving callers of
- # the need to handle those. (Right now, we have no particularly
- # good handling for errors anyway, but that might be added later.)
- pump $harness
- until $$stdout =~ /Type "help" for help/ || $timer->is_expired;
-
- die "psql startup timed out" if $timer->is_expired;
-
- return $harness;
+ return PostgreSQL::Test::BackgroundPsql->new(1, \@psql_params);
}
# Common sub of pgbench-invoking interfaces. Makes any requested script files
diff --git a/src/test/recovery/t/010_logical_decoding_timelines.pl b/src/test/recovery/t/010_logical_decoding_timelines.pl
index eb1a3b6ef8..993f654a9b 100644
--- a/src/test/recovery/t/010_logical_decoding_timelines.pl
+++ b/src/test/recovery/t/010_logical_decoding_timelines.pl
@@ -28,7 +28,6 @@ use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use File::Copy;
-use IPC::Run ();
use Scalar::Util qw(blessed);
my ($stdout, $stderr, $ret);
diff --git a/src/test/recovery/t/031_recovery_conflict.pl b/src/test/recovery/t/031_recovery_conflict.pl
index 84375faccb..e29bc6c181 100644
--- a/src/test/recovery/t/031_recovery_conflict.pl
+++ b/src/test/recovery/t/031_recovery_conflict.pl
@@ -67,14 +67,8 @@ $node_primary->wait_for_replay_catchup($node_standby);
# a longrunning psql that we can use to trigger conflicts
-my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
-my %psql_standby = ('stdin' => '', 'stdout' => '');
-$psql_standby{run} =
- $node_standby->background_psql($test_db, \$psql_standby{stdin},
- \$psql_standby{stdout},
- $psql_timeout);
-$psql_standby{stdout} = '';
-
+my $psql_standby = $node_standby->background_psql($test_db,
+ on_error_stop => 0);
my $expected_conflicts = 0;
@@ -102,15 +96,14 @@ my $cursor1 = "test_recovery_conflict_cursor";
# DECLARE and use a cursor on standby, causing buffer with the only block of
# the relation to be pinned on the standby
-$psql_standby{stdin} .= qq[
- BEGIN;
- DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
- FETCH FORWARD FROM $cursor1;
- ];
+my $res = $psql_standby->query_safe(qq[
+ BEGIN;
+ DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
+ FETCH FORWARD FROM $cursor1;
+]);
# FETCH FORWARD should have returned a 0 since all values of b in the table
# are 0
-ok(pump_until_standby(qr/^0$/m),
- "$sect: cursor with conflicting pin established");
+like($res, qr/^0$/m, "$sect: cursor with conflicting pin established");
# to check the log starting now for recovery conflict messages
my $log_location = -s $node_standby->logfile;
@@ -125,7 +118,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding shared buffer pin for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("bufferpin");
@@ -138,15 +131,12 @@ $node_primary->safe_psql($test_db,
$node_primary->wait_for_replay_catchup($node_standby);
# DECLARE and FETCH from cursor on the standby
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
DECLARE $cursor1 CURSOR FOR SELECT b FROM $table1;
FETCH FORWARD FROM $cursor1;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^0$/m,),
- "$sect: cursor with conflicting snapshot established");
+ ]);
+like($res, qr/^0$/m, "$sect: cursor with conflicting snapshot established");
# Do some HOT updates
$node_primary->safe_psql($test_db,
@@ -160,7 +150,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User query might have needed to see row versions that must be removed");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("snapshot");
@@ -169,12 +159,12 @@ $sect = "lock conflict";
$expected_conflicts++;
# acquire lock to conflict with
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
LOCK TABLE $table1 IN ACCESS SHARE MODE;
SELECT 1;
- ];
-ok(pump_until_standby(qr/^1$/m), "$sect: conflicting lock acquired");
+ ]);
+like($res, qr/^1$/m, "$sect: conflicting lock acquired");
# DROP TABLE containing block which standby has in a pinned buffer
$node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
@@ -182,7 +172,7 @@ $node_primary->safe_psql($test_db, qq[DROP TABLE $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User was holding a relation lock for too long");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("lock");
@@ -193,14 +183,14 @@ $expected_conflicts++;
# DECLARE a cursor for a query which, with sufficiently low work_mem, will
# spill tuples into temp files in the temporary tablespace created during
# setup.
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_safe(qq[
BEGIN;
SET work_mem = '64kB';
DECLARE $cursor1 CURSOR FOR
SELECT count(*) FROM generate_series(1,6000);
FETCH FORWARD FROM $cursor1;
- ];
-ok(pump_until_standby(qr/^6000$/m),
+ ]);
+like($res, qr/^6000$/m,
"$sect: cursor with conflicting temp file established");
# Drop the tablespace currently containing spill files for the query on the
@@ -211,7 +201,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log(
"User was or might have been using tablespace that must be dropped");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("tablespace");
@@ -227,7 +217,7 @@ $node_standby->adjust_conf(
'max_standby_streaming_delay',
"${PostgreSQL::Test::Utils::timeout_default}s");
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Generate a few dead rows, to later be cleaned up by vacuum. Then acquire a
# lock on another relation in a prepared xact, so it's held continuously by
@@ -250,19 +240,15 @@ SELECT txid_current();
$node_primary->wait_for_replay_catchup($node_standby);
-$psql_standby{stdin} .= qq[
+$res = $psql_standby->query_until(qr/^1$/m, qq[
BEGIN;
-- hold pin
DECLARE $cursor1 CURSOR FOR SELECT a FROM $table1;
FETCH FORWARD FROM $cursor1;
-- wait for lock held by prepared transaction
SELECT * FROM $table2;
- ];
-ok( pump_until(
- $psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, qr/^1$/m,),
- "$sect: cursor holding conflicting pin, also waiting for lock, established"
-);
+ ]);
+ok( 1, "$sect: cursor holding conflicting pin, also waiting for lock, established");
# just to make sure we're waiting for lock already
ok( $node_standby->poll_query_until(
@@ -277,7 +263,7 @@ $node_primary->safe_psql($test_db, qq[VACUUM $table1;]);
$node_primary->wait_for_replay_catchup($node_standby);
check_conflict_log("User transaction caused buffer deadlock with recovery.");
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
check_conflict_stat("deadlock");
# clean up for next tests
@@ -285,7 +271,7 @@ $node_primary->safe_psql($test_db, qq[ROLLBACK PREPARED 'lock';]);
$node_standby->adjust_conf('postgresql.conf', 'max_standby_streaming_delay',
'50ms');
$node_standby->restart();
-reconnect_and_clear();
+$psql_standby->reconnect_and_clear();
# Check that expected number of conflicts show in pg_stat_database. Needs to
@@ -309,8 +295,7 @@ check_conflict_log("User was connected to a database that must be dropped");
# explicitly shut down psql instances gracefully - to avoid hangs or worse on
# windows
-$psql_standby{stdin} .= "\\q\n";
-$psql_standby{run}->finish;
+$psql_standby->quit;
$node_standby->stop();
$node_primary->stop();
@@ -318,37 +303,6 @@ $node_primary->stop();
done_testing();
-
-sub pump_until_standby
-{
- my $match = shift;
-
- return pump_until($psql_standby{run}, $psql_timeout,
- \$psql_standby{stdout}, $match);
-}
-
-sub reconnect_and_clear
-{
- # If psql isn't dead already, tell it to quit as \q, when already dead,
- # causes IPC::Run to unhelpfully error out with "ack Broken pipe:".
- $psql_standby{run}->pump_nb();
- if ($psql_standby{run}->pumpable())
- {
- $psql_standby{stdin} .= "\\q\n";
- }
- $psql_standby{run}->finish;
-
- # restart
- $psql_standby{run}->run();
- $psql_standby{stdin} = '';
- $psql_standby{stdout} = '';
-
- # Run query to ensure connection has finished re-establishing
- $psql_standby{stdin} .= qq[SELECT 1;\n];
- die unless pump_until_standby(qr/^1$/m);
- $psql_standby{stdout} = '';
-}
-
sub check_conflict_log
{
my $message = shift;
diff --git a/src/test/subscription/t/015_stream.pl b/src/test/subscription/t/015_stream.pl
index 0e0f27f14d..9762497146 100644
--- a/src/test/subscription/t/015_stream.pl
+++ b/src/test/subscription/t/015_stream.pl
@@ -28,26 +28,20 @@ sub test_streaming
my ($node_publisher, $node_subscriber, $appname, $is_parallel) = @_;
# Interleave a pair of transactions, each exceeding the 64kB limit.
- my $in = '';
- my $out = '';
-
my $offset = 0;
- my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
- my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+ my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
- $in .= q{
+ $h->query_safe(q{
BEGIN;
INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
DELETE FROM test_tab WHERE mod(a,3) = 0;
- };
- $h->pump_nb;
+ });
$node_publisher->safe_psql(
'postgres', q{
@@ -57,11 +51,10 @@ sub test_streaming
COMMIT;
});
- $in .= q{
- COMMIT;
- \q
- };
- $h->finish; # errors make the next test fail, so ignore them here
+ $h->query_safe('COMMIT');
+ $h->quit;
+ # XXX: Not sure what this means
+ # errors make the next test fail, so ignore them here
$node_publisher->wait_for_catchup($appname);
@@ -219,12 +212,7 @@ $node_subscriber->reload;
$node_subscriber->safe_psql('postgres', q{SELECT 1});
# Interleave a pair of transactions, each exceeding the 64kB limit.
-my $in = '';
-my $out = '';
-
-my $timer = IPC::Run::timeout($PostgreSQL::Test::Utils::timeout_default);
-
-my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
+my $h = $node_publisher->background_psql('postgres',
on_error_stop => 0);
# Confirm if a deadlock between the leader apply worker and the parallel apply
@@ -232,11 +220,10 @@ my $h = $node_publisher->background_psql('postgres', \$in, \$out, $timer,
my $offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the parallel apply worker executes the insert command before the
# leader worker.
@@ -246,11 +233,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres', "INSERT INTO test_tab_2 values(1)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
@@ -277,11 +261,10 @@ $node_subscriber->safe_psql('postgres',
# Check the subscriber log from now on.
$offset = -s $node_subscriber->logfile;
-$in .= q{
+$h->query_safe(q{
BEGIN;
INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i);
-};
-$h->pump_nb;
+});
# Ensure that the first parallel apply worker executes the insert command
# before the second one.
@@ -292,11 +275,8 @@ $node_subscriber->wait_for_log(
$node_publisher->safe_psql('postgres',
"INSERT INTO test_tab_2 SELECT i FROM generate_series(1, 5000) s(i)");
-$in .= q{
-COMMIT;
-\q
-};
-$h->finish;
+$h->query_safe('COMMIT');
+$h->quit;
$node_subscriber->wait_for_log(qr/ERROR: ( [A-Z0-9]+:)? deadlock detected/,
$offset);
--
2.32.1 (Apple Git-133)
On 7 Apr 2023, at 18:14, Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Apr 2023, at 17:04, Andres Freund <andres@anarazel.de> wrote:
Afaict the failures are purely about patch 2, not 1, right?
Correct. The attached v6 wraps the interactive_psql test in a SKIP block with
a conditional on IO::Pty being available.
This version was green in the CFBot, so I ended up pushing it after some
documentation fixups and polish.
--
Daniel Gustafsson
On 7 Apr 2023, at 22:24, Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Apr 2023, at 18:14, Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Apr 2023, at 17:04, Andres Freund <andres@anarazel.de> wrote:
Afaict the failures are purely about patch 2, not 1, right?
Correct. The attached v6 wraps the interactive_psql test in a SKIP block with
a conditional on IO::Pty being available.This version was green in the CFBot, so I ended up pushing it after some
documentation fixups and polish.
Looks like morepork wasn't happy with the interactive \password test.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=morepork&dt=2023-04-07%2020%3A30%3A29
Looking into why the timer timed out in that test.
--
Daniel Gustafsson
On 7 Apr 2023, at 23:01, Daniel Gustafsson <daniel@yesql.se> wrote:
Looks like morepork wasn't happy with the interactive \password test.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=morepork&dt=2023-04-07%2020%3A30%3A29
Looking into why the timer timed out in that test.
Staring at this I've been unable to figure out if there an underlying problem
here or a flaky testrun, since I can't reproduce it. Maybe the animal owner
(on cc) have any insights?
The test has passed on several different platforms in the buildfarm, including
Linux, Solaris, macOS, NetBSD, FreeBSD and other OpenBSD animals. It also
passed in an OpenBSD VM running with our Cirrus framework.
Unless there are objections raised I propose leaving it in for now, and I will
return to it tomorrow after some sleep, and install OpenBSD 6.9 to see if it's
reproducible.
--
Daniel Gustafsson
Daniel Gustafsson <daniel@yesql.se> writes:
On 7 Apr 2023, at 23:01, Daniel Gustafsson <daniel@yesql.se> wrote:
Staring at this I've been unable to figure out if there an underlying problem
here or a flaky testrun, since I can't reproduce it. Maybe the animal owner
(on cc) have any insights?
The test has passed on several different platforms in the buildfarm, including
Linux, Solaris, macOS, NetBSD, FreeBSD and other OpenBSD animals. It also
passed in an OpenBSD VM running with our Cirrus framework.
prion and mantid have now failed with the same symptom. I don't
see a pattern, but it's not OpenBSD-only. It will be interesting
to see if the failure is intermittent or not on those animals.
Unless there are objections raised I propose leaving it in for now, and I will
return to it tomorrow after some sleep, and install OpenBSD 6.9 to see if it's
reproducible.
Agreed, we don't need a hasty revert here. Better to gather data.
regards, tom lane
On 8 Apr 2023, at 00:35, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
On 7 Apr 2023, at 23:01, Daniel Gustafsson <daniel@yesql.se> wrote:
Staring at this I've been unable to figure out if there an underlying problem
here or a flaky testrun, since I can't reproduce it. Maybe the animal owner
(on cc) have any insights?The test has passed on several different platforms in the buildfarm, including
Linux, Solaris, macOS, NetBSD, FreeBSD and other OpenBSD animals. It also
passed in an OpenBSD VM running with our Cirrus framework.prion and mantid have now failed with the same symptom. I don't
see a pattern, but it's not OpenBSD-only. It will be interesting
to see if the failure is intermittent or not on those animals.
It would be interesting to know how far in the pumped input they get, if they
time out on the first one with nothing going through? Will investigate further
tomorrow to see.
--
Daniel Gustafsson
On 8 Apr 2023, at 00:59, Daniel Gustafsson <daniel@yesql.se> wrote:
On 8 Apr 2023, at 00:35, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
On 7 Apr 2023, at 23:01, Daniel Gustafsson <daniel@yesql.se> wrote:
Staring at this I've been unable to figure out if there an underlying problem
here or a flaky testrun, since I can't reproduce it. Maybe the animal owner
(on cc) have any insights?The test has passed on several different platforms in the buildfarm, including
Linux, Solaris, macOS, NetBSD, FreeBSD and other OpenBSD animals. It also
passed in an OpenBSD VM running with our Cirrus framework.prion and mantid have now failed with the same symptom. I don't
see a pattern, but it's not OpenBSD-only. It will be interesting
to see if the failure is intermittent or not on those animals.
morepork has failed again, which is good, since intermittent failures are
harder to track down.
It would be interesting to know how far in the pumped input they get, if they
time out on the first one with nothing going through? Will investigate further
tomorrow to see.
Actually, one quick datapoint. prion and mantid report running IPC::Run
version 0.92, and morepork 0.96. Animals that pass are running 20180523.0,
20200505.0, 20220807.0 or similar versions. We don't print the IO::Pty version
during configure, but maybe this is related to older versions of the modules
and this test (not all of them apparently) need to SKIP if IO::Pty is missing
or too old? Somewhere to start looking at the very least.
--
Daniel Gustafsson
On 2023-04-07 Fr 19:14, Daniel Gustafsson wrote:
On 8 Apr 2023, at 00:59, Daniel Gustafsson<daniel@yesql.se> wrote:
On 8 Apr 2023, at 00:35, Tom Lane<tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson<daniel@yesql.se> writes:
On 7 Apr 2023, at 23:01, Daniel Gustafsson<daniel@yesql.se> wrote:
Staring at this I've been unable to figure out if there an underlying problem
here or a flaky testrun, since I can't reproduce it. Maybe the animal owner
(on cc) have any insights?
The test has passed on several different platforms in the buildfarm, including
Linux, Solaris, macOS, NetBSD, FreeBSD and other OpenBSD animals. It also
passed in an OpenBSD VM running with our Cirrus framework.prion and mantid have now failed with the same symptom. I don't
see a pattern, but it's not OpenBSD-only. It will be interesting
to see if the failure is intermittent or not on those animals.morepork has failed again, which is good, since intermittent failures are
harder to track down.It would be interesting to know how far in the pumped input they get, if they
time out on the first one with nothing going through? Will investigate further
tomorrow to see.Actually, one quick datapoint. prion and mantid report running IPC::Run
version 0.92, and morepork 0.96. Animals that pass are running 20180523.0,
20200505.0, 20220807.0 or similar versions. We don't print the IO::Pty version
during configure, but maybe this is related to older versions of the modules
and this test (not all of them apparently) need to SKIP if IO::Pty is missing
or too old? Somewhere to start looking at the very least.
Those aren't CPAN version numbers. See <https://metacpan.org/pod/IO::Pty>
prion was running 1.10 (dated to 2010). I have just updated it to 1.17
(the CPAN latest). We'll see if that makes a difference.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
Actually, one quick datapoint. prion and mantid report running IPC::Run
version 0.92, and morepork 0.96. Animals that pass are running 20180523.0,
20200505.0, 20220807.0 or similar versions. We don't print the IO::Pty version
during configure, but maybe this is related to older versions of the modules
and this test (not all of them apparently) need to SKIP if IO::Pty is missing
or too old? Somewhere to start looking at the very least.
prion was running 1.10 (dated to 2010). I have just updated it to 1.17
(the CPAN latest). We'll see if that makes a difference.
I've been doing some checking with perlbrew locally. It appears to not
be about IO::Pty so much as IPC::Run: it works with IPC::Run 0.99 but
not 0.79. Still bisecting to identify exactly what's the minimum
okay version.
regards, tom lane
I wrote:
I've been doing some checking with perlbrew locally. It appears to not
be about IO::Pty so much as IPC::Run: it works with IPC::Run 0.99 but
not 0.79. Still bisecting to identify exactly what's the minimum
okay version.
The answer is: it works with IPC::Run >= 0.98. The version of IO::Pty
doesn't appear significant; it works at least back to 1.00 from early
2002.
IPC::Run 0.98 is relatively new (2018), so I don't think it'd fly
to make that our new minimum version across-the-board. I recommend
just setting up this one test to SKIP if IPC::Run is too old.
regards, tom lane
Hi,
On 2023-04-07 20:49:39 -0400, Tom Lane wrote:
I wrote:
I've been doing some checking with perlbrew locally. It appears to not
be about IO::Pty so much as IPC::Run: it works with IPC::Run 0.99 but
not 0.79. Still bisecting to identify exactly what's the minimum
okay version.The answer is: it works with IPC::Run >= 0.98. The version of IO::Pty
doesn't appear significant; it works at least back to 1.00 from early
2002.IPC::Run 0.98 is relatively new (2018), so I don't think it'd fly
to make that our new minimum version across-the-board. I recommend
just setting up this one test to SKIP if IPC::Run is too old.
Does the test actually take a while before it fails, or is it quick? It's
possible the failure is caused by 001_password.pl's use of
set_query_timer_restart(). I don't think other tests do something quite
comparable.
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
On 2023-04-07 20:49:39 -0400, Tom Lane wrote:
IPC::Run 0.98 is relatively new (2018), so I don't think it'd fly
to make that our new minimum version across-the-board. I recommend
just setting up this one test to SKIP if IPC::Run is too old.
Does the test actually take a while before it fails, or is it quick?
It times out at whatever your PG_TEST_TIMEOUT_DEFAULT is. I waited
3 minutes the first time, and then reduced that to 20sec for the
rest of the tries ...
regards, tom lane
On 8 Apr 2023, at 02:49, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I wrote:
I've been doing some checking with perlbrew locally. It appears to not
be about IO::Pty so much as IPC::Run: it works with IPC::Run 0.99 but
not 0.79. Still bisecting to identify exactly what's the minimum
okay version.The answer is: it works with IPC::Run >= 0.98. The version of IO::Pty
doesn't appear significant; it works at least back to 1.00 from early
2002.
Thanks for investigating this!
IPC::Run 0.98 is relatively new (2018), so I don't think it'd fly
to make that our new minimum version across-the-board.
Absolutely, that's not an option.
I recommend
just setting up this one test to SKIP if IPC::Run is too old.
Yes, will do that when I have a little more time at hand for monitoring the BF
later today.
--
Daniel Gustafsson
On 2023-04-08 09:53, Daniel Gustafsson wrote:
I recommend
just setting up this one test to SKIP if IPC::Run is too old.Yes, will do that when I have a little more time at hand for monitoring the BF
later today.
So what do you want me to do about grison and morepork? I guess I could
try to install a newer version of IPC::Run from CPAN or should I just
leave it be?
/Mikael
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@gmail.com> writes:
So what do you want me to do about grison and morepork? I guess I could
try to install a newer version of IPC::Run from CPAN or should I just
leave it be?
I think "leave it be" is fine.
regards, tom lane
While fixing a recent bug on visibility on a standby [1]/messages/by-id/6b852e98-2d49-4ca1-9e95-db419a2696e0@iki.fi, I wrote a
regression test that uses BackgroundPsql to run some queries in a
long-running psql session. The problem is that that was refactored in
v17, commit 664d757531. The test I wrote for v17 doesn't work as it is
on backbranches. Options:
1. Write the new test differently on backbranches. Before 664d757531,
the test needs to work a lot harder to use the background psql session,
calling pump() etc. That's doable, but as noted in the discussion that
led to 664d757531, it's laborious and error-prone.
2. Backport commit 664d757531. This might break out-of-tree perl tests
that use the background_psql() function. I don't know if any such tests
exist, and they would need to be changed for v17 anyway, so that seems
acceptable. Anyone aware of any extensions using the perl test modules?
3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".
I'm leaning towards 3. We might need to backport more perl tests that
use background_psql() in the future, backporting the test module will
make that easier.
Thoughts?
[1]: /messages/by-id/6b852e98-2d49-4ca1-9e95-db419a2696e0@iki.fi
/messages/by-id/6b852e98-2d49-4ca1-9e95-db419a2696e0@iki.fi
--
Heikki Linnakangas
Neon (https://neon.tech)
On 25/06/2024 13:26, Heikki Linnakangas wrote:
While fixing a recent bug on visibility on a standby [1], I wrote a
regression test that uses BackgroundPsql to run some queries in a
long-running psql session. The problem is that that was refactored in
v17, commit 664d757531. The test I wrote for v17 doesn't work as it is
on backbranches. Options:
Sorry, I meant v16. The BackgroundPsql refactorings went into v16. The
backporting question remains for v15 and below.
--
Heikki Linnakangas
Neon (https://neon.tech)
Hi,
On 2024-06-25 13:26:23 +0300, Heikki Linnakangas wrote:
While fixing a recent bug on visibility on a standby [1], I wrote a
regression test that uses BackgroundPsql to run some queries in a
long-running psql session. The problem is that that was refactored in v17,
commit 664d757531. The test I wrote for v17 doesn't work as it is on
backbranches. Options:1. Write the new test differently on backbranches. Before 664d757531, the
test needs to work a lot harder to use the background psql session, calling
pump() etc. That's doable, but as noted in the discussion that led to
664d757531, it's laborious and error-prone.2. Backport commit 664d757531. This might break out-of-tree perl tests that
use the background_psql() function. I don't know if any such tests exist,
and they would need to be changed for v17 anyway, so that seems acceptable.
Anyone aware of any extensions using the perl test modules?3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".I'm leaning towards 3. We might need to backport more perl tests that use
background_psql() in the future, backporting the test module will make that
easier.Thoughts?
Yes, I've wished for this a couple times. I think 2 or 3 would be reasonable.
I think 1) often just leads to either tests not being written or being
fragile...
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
On 2024-06-25 13:26:23 +0300, Heikki Linnakangas wrote:
1. Write the new test differently on backbranches. Before 664d757531, the
test needs to work a lot harder to use the background psql session, calling
pump() etc. That's doable, but as noted in the discussion that led to
664d757531, it's laborious and error-prone.2. Backport commit 664d757531. This might break out-of-tree perl tests that
use the background_psql() function. I don't know if any such tests exist,
and they would need to be changed for v17 anyway, so that seems acceptable.
Anyone aware of any extensions using the perl test modules?3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".
Yes, I've wished for this a couple times. I think 2 or 3 would be reasonable.
I think 1) often just leads to either tests not being written or being
fragile...
I'd vote for (2). (3) is just leaving a foot-gun for people to
hurt themselves with.
regards, tom lane
On Tue, Jun 25, 2024 at 7:40 AM Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2024-06-25 13:26:23 +0300, Heikki Linnakangas wrote:
While fixing a recent bug on visibility on a standby [1], I wrote a
regression test that uses BackgroundPsql to run some queries in a
long-running psql session. The problem is that that was refactored in v17,
commit 664d757531. The test I wrote for v17 doesn't work as it is on
backbranches. Options:1. Write the new test differently on backbranches. Before 664d757531, the
test needs to work a lot harder to use the background psql session, calling
pump() etc. That's doable, but as noted in the discussion that led to
664d757531, it's laborious and error-prone.2. Backport commit 664d757531. This might break out-of-tree perl tests that
use the background_psql() function. I don't know if any such tests exist,
and they would need to be changed for v17 anyway, so that seems acceptable.
Anyone aware of any extensions using the perl test modules?3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".I'm leaning towards 3. We might need to backport more perl tests that use
background_psql() in the future, backporting the test module will make that
easier.Thoughts?
Yes, I've wished for this a couple times. I think 2 or 3 would be reasonable.
I think 1) often just leads to either tests not being written or being
fragile...
+1 to backporting background psql!
I'm also okay with 2 or 3. But note that for 2, there are several
tests with skip_all and at least one of them uses background_psql
(031_recovery_conflict.pl), so we'll just have to remember to update
those. I assume that is easy enough to do if you grep for
background_psql -- but, just in case you were going to be 100%
test-driven :)
- Melanie
On 2024-06-25 Tu 10:26 AM, Tom Lane wrote:
Andres Freund<andres@anarazel.de> writes:
On 2024-06-25 13:26:23 +0300, Heikki Linnakangas wrote:
1. Write the new test differently on backbranches. Before 664d757531, the
test needs to work a lot harder to use the background psql session, calling
pump() etc. That's doable, but as noted in the discussion that led to
664d757531, it's laborious and error-prone.2. Backport commit 664d757531. This might break out-of-tree perl tests that
use the background_psql() function. I don't know if any such tests exist,
and they would need to be changed for v17 anyway, so that seems acceptable.
Anyone aware of any extensions using the perl test modules?3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".Yes, I've wished for this a couple times. I think 2 or 3 would be reasonable.
I think 1) often just leads to either tests not being written or being
fragile...I'd vote for (2). (3) is just leaving a foot-gun for people to
hurt themselves with.
+1
I'd like to get rid of it in its current form at least. Just about all
the uses I'm aware of could be transformed to use the Session object
I've been working on, based either on FFI or a small XS wrapper for some
of libpq.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
On 25 Jun 2024, at 16:26, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andres Freund <andres@anarazel.de> writes:
On 2024-06-25 13:26:23 +0300, Heikki Linnakangas wrote:
1. Write the new test differently on backbranches. Before 664d757531, the
test needs to work a lot harder to use the background psql session, calling
pump() etc. That's doable, but as noted in the discussion that led to
664d757531, it's laborious and error-prone.2. Backport commit 664d757531. This might break out-of-tree perl tests that
use the background_psql() function. I don't know if any such tests exist,
and they would need to be changed for v17 anyway, so that seems acceptable.
Anyone aware of any extensions using the perl test modules?3. Backport commit 664d757531, but keep the existing background_psql()
function unchanged. Add a different constructor to get the v17-style
BackgroundPsql session, something like "$node->background_psql_new()".Yes, I've wished for this a couple times. I think 2 or 3 would be reasonable.
I think 1) often just leads to either tests not being written or being
fragile...I'd vote for (2). (3) is just leaving a foot-gun for people to
hurt themselves with.
I agree with this, if we're backporting we should opt for 2. The only out of
tree user of background_psql() that I could find was check_pgactivity but they
seem to have vendored an old copy of our lib rather than use anything in a our
tree so we should be fine there.
Before pulling any triggers I think https://commitfest.postgresql.org/48/4959/
should be considered, since Tom found some flaws in the current code around how
timers and timeouts are used.
However, since Andrew is actively aiming to replace all of this shortly, should
we wait a see where that lands to avoid having to backport another library
change?
--
Daniel Gustafsson
Daniel Gustafsson <daniel@yesql.se> writes:
Before pulling any triggers I think https://commitfest.postgresql.org/48/4959/
should be considered, since Tom found some flaws in the current code around how
timers and timeouts are used.
That's certainly another issue to consider, but is it really a blocker
for this one?
However, since Andrew is actively aiming to replace all of this shortly, should
we wait a see where that lands to avoid having to backport another library
change?
I would like to see what he comes up with ... but is it likely to
be something we'd risk back-patching?
regards, tom lane
On 25 Jun 2024, at 22:57, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
Before pulling any triggers I think https://commitfest.postgresql.org/48/4959/
should be considered, since Tom found some flaws in the current code around how
timers and timeouts are used.That's certainly another issue to consider, but is it really a blocker
for this one?
It's not a blocker, but when poking at the code it seems useful to consider the
open items around it.
However, since Andrew is actively aiming to replace all of this shortly, should
we wait a see where that lands to avoid having to backport another library
change?I would like to see what he comes up with ... but is it likely to
be something we'd risk back-patching?
Maybe it'll be a stretch given that it's likely to introduce new dependencies.
--
Daniel Gustafsson
On 2024-Jun-25, Tom Lane wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
However, since Andrew is actively aiming to replace all of this shortly, should
we wait a see where that lands to avoid having to backport another library
change?I would like to see what he comes up with ... but is it likely to
be something we'd risk back-patching?
FWIW I successfully used the preliminary PqFFI stuff Andrew posted to
write a test program for bug #18377, which I think ended up being better
than with BackgroundPsql, so I think it's a good way forward. As for
back-patching it, I suspect we're going to end up backpatching the
framework anyway just because we'll want to have it available for
backpatching future tests, even if we keep a backpatch minimal by doing
only the framework and not existing tests.
I also backpatched the PqFFI and PostgreSQL::Session modules to older PG
branches, to run my test program there. This required only removing
some lines from PqFFI.pm that were about importing libpq functions that
older libpq didn't have.
Of course, the PostgreSQL::Session stuff is not ready yet, so if we want
this test in the tree soon, I don't think we should wait.
I'll note, though, that Test::More doesn't work terribly nicely with
perl threads, but that only relates to my test program and not to PqFFI.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"La gente vulgar sólo piensa en pasar el tiempo;
el que tiene talento, en aprovecharlo"
On Wed, Jun 26, 2024 at 02:12:42AM +0200, Alvaro Herrera wrote:
FWIW I successfully used the preliminary PqFFI stuff Andrew posted to
write a test program for bug #18377, which I think ended up being better
than with BackgroundPsql, so I think it's a good way forward. As for
back-patching it, I suspect we're going to end up backpatching the
framework anyway just because we'll want to have it available for
backpatching future tests, even if we keep a backpatch minimal by doing
only the framework and not existing tests.I also backpatched the PqFFI and PostgreSQL::Session modules to older PG
branches, to run my test program there. This required only removing
some lines from PqFFI.pm that were about importing libpq functions that
older libpq didn't have.
Nice! I definitely +1 the backpatching of the testing bits. This
stuff can make validating bugs so much easier, particularly when there
are conflicting parts in the backend after a cherry-pick.
--
Michael
On 26/06/2024 03:25, Michael Paquier wrote:
On Wed, Jun 26, 2024 at 02:12:42AM +0200, Alvaro Herrera wrote:
FWIW I successfully used the preliminary PqFFI stuff Andrew posted to
write a test program for bug #18377, which I think ended up being better
than with BackgroundPsql, so I think it's a good way forward. As for
back-patching it, I suspect we're going to end up backpatching the
framework anyway just because we'll want to have it available for
backpatching future tests, even if we keep a backpatch minimal by doing
only the framework and not existing tests.I also backpatched the PqFFI and PostgreSQL::Session modules to older PG
branches, to run my test program there. This required only removing
some lines from PqFFI.pm that were about importing libpq functions that
older libpq didn't have.Nice! I definitely +1 the backpatching of the testing bits. This
stuff can make validating bugs so much easier, particularly when there
are conflicting parts in the backend after a cherry-pick.
I haven't looked closely at the new PgFFI stuff but +1 on that in
general, and it makes sense to backport that once it lands on master. In
the meanwhile, I think we should backport BackgroundPsql as it is, to
make it possible to backport tests using it right now, even if it is
short-lived.
--
Heikki Linnakangas
Neon (https://neon.tech)
On Wed, Jun 26, 2024 at 3:34 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
I haven't looked closely at the new PgFFI stuff but +1 on that in
general, and it makes sense to backport that once it lands on master. In
the meanwhile, I think we should backport BackgroundPsql as it is, to
make it possible to backport tests using it right now, even if it is
short-lived.
+1. The fact that PgFFI may be coming isn't a reason to not back-patch
this. The risk of back-patching testing infrastructure is also very
low as compared with code; in fact, there's a lot of risk from NOT
back-patching popular testing infrastructure.
--
Robert Haas
EDB: http://www.enterprisedb.com
On 26/06/2024 14:54, Robert Haas wrote:
On Wed, Jun 26, 2024 at 3:34 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
I haven't looked closely at the new PgFFI stuff but +1 on that in
general, and it makes sense to backport that once it lands on master. In
the meanwhile, I think we should backport BackgroundPsql as it is, to
make it possible to backport tests using it right now, even if it is
short-lived.+1. The fact that PgFFI may be coming isn't a reason to not back-patch
this. The risk of back-patching testing infrastructure is also very
low as compared with code; in fact, there's a lot of risk from NOT
back-patching popular testing infrastructure.
Ok, I pushed commits to backport BackgroundPsql down to v12. I used
"option 2", i.e. I changed background_psql() to return the new
BackgroundPsql object.
--
Heikki Linnakangas
Neon (https://neon.tech)
On Thu, Jun 27, 2024 at 10:05 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
Ok, I pushed commits to backport BackgroundPsql down to v12. I used
"option 2", i.e. I changed background_psql() to return the new
BackgroundPsql object.
Don't we need to add install and uninstall rules for the new module, like
we did in
https://git.postgresql.org/pg/commitdiff/a4c17c86176cfa712f541b81b2a026ae054b275e
and
https://git.postgresql.org/pg/commitdiff/7039c7cff6736780c3bbb41a90a6dfea0f581ad2
?
Thanks,
Pavan
On 29 Jun 2024, at 06:38, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
Don't we need to add install and uninstall rules for the new module, like we did in https://git.postgresql.org/pg/commitdiff/a4c17c86176cfa712f541b81b2a026ae054b275e and https://git.postgresql.org/pg/commitdiff/7039c7cff6736780c3bbb41a90a6dfea0f581ad2 ?
Thats correct, we should backport those as well.
--
Daniel Gustafsson
H i Daniel,
On Mon, Jul 1, 2024 at 1:09 PM Daniel Gustafsson <daniel@yesql.se> wrote:
On 29 Jun 2024, at 06:38, Pavan Deolasee <pavan.deolasee@gmail.com>
wrote:
Don't we need to add install and uninstall rules for the new module,
like we did in
https://git.postgresql.org/pg/commitdiff/a4c17c86176cfa712f541b81b2a026ae054b275e
and
https://git.postgresql.org/pg/commitdiff/7039c7cff6736780c3bbb41a90a6dfea0f581ad2
?Thats correct, we should backport those as well.
Thanks for confirming. Attaching patches for PG15 and PG14, but this will
need backporting all the way up to PG12.
Thanks,
Pavan
Attachments:
0001-Fix-missing-installation-uninstallation-rules-for-Ba-PG14.patchapplication/octet-stream; name=0001-Fix-missing-installation-uninstallation-rules-for-Ba-PG14.patchDownload
From e553d48f2ba9eae9dcc14423c7368045615dc154 Mon Sep 17 00:00:00 2001
From: Pavan Deolasee <pavan.deolasee@gmail.com>
Date: Mon, 1 Jul 2024 19:37:16 +0530
Subject: [PATCH] Fix missing installation/uninstallation rules for
BackgroundPsql.pm
Commit 31877cd8 backported BackgroundPsql perl module module with helper
functions for tests running interactive or background psql tasks. to
PG 12 to 15, but did not add installation/uninstallation rules of the build
system, causing problems running TAP tests for the extensions.
---
src/test/perl/Makefile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index 811acf7cd59..2adb9e91f39 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgresVersion.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgresVersion.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Utils.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Utils.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/TestLib.pm'
@@ -35,5 +36,6 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgresVersion.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Utils.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
endif
--
2.38.1
0001-Fix-missing-installation-uninstallation-rules-for-Ba-PG15.patchapplication/octet-stream; name=0001-Fix-missing-installation-uninstallation-rules-for-Ba-PG15.patchDownload
From 1725622b7fb31613b2af32e1b1ff5777cc432317 Mon Sep 17 00:00:00 2001
From: Pavan Deolasee <pavan.deolasee@gmail.com>
Date: Mon, 1 Jul 2024 19:31:51 +0530
Subject: [PATCH] Fix missing installation/uninstallation rules for
BackgroundPsql.pm
Commit d5fd7865 backported BackgroundPsql perl module module with helper
functions for tests running interactive or background psql tasks. to
PG 12 to 15, but did not add installation/uninstallation rules of the build
system, causing problems running TAP tests for the extensions.
---
src/test/perl/Makefile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index ffa736a54cd..4e2ea91ec54 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -24,6 +24,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/RecursiveCopy.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/RecursiveCopy.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Utils.pm'
@@ -31,5 +32,6 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/RecursiveCopy.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
endif
--
2.38.1
On 01/07/2024 17:11, Pavan Deolasee wrote:
H i Daniel,
On Mon, Jul 1, 2024 at 1:09 PM Daniel Gustafsson <daniel@yesql.se
<mailto:daniel@yesql.se>> wrote:On 29 Jun 2024, at 06:38, Pavan Deolasee
<pavan.deolasee@gmail.com <mailto:pavan.deolasee@gmail.com>> wrote:
Don't we need to add install and uninstall rules for the new
module, like we did in
https://git.postgresql.org/pg/commitdiff/a4c17c86176cfa712f541b81b2a026ae054b275e <https://git.postgresql.org/pg/commitdiff/a4c17c86176cfa712f541b81b2a026ae054b275e> and https://git.postgresql.org/pg/commitdiff/7039c7cff6736780c3bbb41a90a6dfea0f581ad2 <https://git.postgresql.org/pg/commitdiff/7039c7cff6736780c3bbb41a90a6dfea0f581ad2> ?Thats correct, we should backport those as well.
Thanks for confirming. Attaching patches for PG15 and PG14, but this
will need backporting all the way up to PG12.
Thanks! Pushed to v12-v15.
--
Heikki Linnakangas
Neon (https://neon.tech)