possibility to specify template database for pg_regress
Hi
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.
Regards
Pavel
On Wed, Feb 8, 2017 at 12:43 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.
Not directly, all the databases created by pg_regress are enforced
with template0.. Having a switch sounds useful though without seeing
in details your use case.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Michael Paquier <michael.paquier@gmail.com> writes:
On Wed, Feb 8, 2017 at 12:43 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.
Not directly, all the databases created by pg_regress are enforced
with template0.. Having a switch sounds useful though without seeing
in details your use case.
I ran into a use-case just today: I wanted to run one particular
regression test script under CLOBBER_CACHE_ALWAYS, but it needed
stuff created by earlier scripts, and I didn't especially want to
run all of those scripts under CCA. With a way to select a template,
I could've run the earlier scripts in a normal build, renamed the
ending-state regression database to something else, and then installed
a CCA-enabled executable and run a test with just the script of
interest. The way I actually got it done was considerably hackier :-(
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-02-07 19:23:45 -0500, Tom Lane wrote:
Michael Paquier <michael.paquier@gmail.com> writes:
On Wed, Feb 8, 2017 at 12:43 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.Not directly, all the databases created by pg_regress are enforced
with template0.. Having a switch sounds useful though without seeing
in details your use case.I ran into a use-case just today: I wanted to run one particular
regression test script under CLOBBER_CACHE_ALWAYS, but it needed
stuff created by earlier scripts, and I didn't especially want to
run all of those scripts under CCA. With a way to select a template,
I could've run the earlier scripts in a normal build, renamed the
ending-state regression database to something else, and then installed
a CCA-enabled executable and run a test with just the script of
interest. The way I actually got it done was considerably hackier :-(
Can't you do that with --use-existing? I'm pretty sure I used it for
very similar issues before. And yes, the --help text for that is
misleading.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, Feb 8, 2017 at 9:23 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I ran into a use-case just today: I wanted to run one particular
regression test script under CLOBBER_CACHE_ALWAYS, but it needed
stuff created by earlier scripts, and I didn't especially want to
run all of those scripts under CCA. With a way to select a template,
I could've run the earlier scripts in a normal build, renamed the
ending-state regression database to something else, and then installed
a CCA-enabled executable and run a test with just the script of
interest. The way I actually got it done was considerably hackier :-(
Looking at the code, --dbname can actually accept a list of databases.
Perhaps we could just have the equivalent for templates? I think that
we just need to be sure that the template list matches the length of
the database list if the template list is longer than one.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2017-02-08 1:30 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Wed, Feb 8, 2017 at 9:23 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I ran into a use-case just today: I wanted to run one particular
regression test script under CLOBBER_CACHE_ALWAYS, but it needed
stuff created by earlier scripts, and I didn't especially want to
run all of those scripts under CCA. With a way to select a template,
I could've run the earlier scripts in a normal build, renamed the
ending-state regression database to something else, and then installed
a CCA-enabled executable and run a test with just the script of
interest. The way I actually got it done was considerably hackier :-(Looking at the code, --dbname can actually accept a list of databases.
Perhaps we could just have the equivalent for templates? I think that
we just need to be sure that the template list matches the length of
the database list if the template list is longer than one.
What is sense for list of databases?
Some option --template can be great - with backpatch if it is possible.
Regards
Pavel
Show quoted text
--
Michael
On Wed, Feb 8, 2017 at 4:24 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
What is sense for list of databases?
ECPG uses it for example, see 0992259.
Some option --template can be great - with backpatch if it is possible.
That's not really complicated to patch... That could be a nice task
for a starter.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2017-02-08 8:30 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Wed, Feb 8, 2017 at 4:24 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:What is sense for list of databases?
ECPG uses it for example, see 0992259.
Some option --template can be great - with backpatch if it is possible.
That's not really complicated to patch... That could be a nice task
for a starter.
Today I am doing some training - I can look on it at evening
Regards
Pavel
Show quoted text
--
Michael
Hi
2017-02-08 8:33 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:
2017-02-08 8:30 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Wed, Feb 8, 2017 at 4:24 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:What is sense for list of databases?
ECPG uses it for example, see 0992259.
Some option --template can be great - with backpatch if it is possible.
That's not really complicated to patch... That could be a nice task
for a starter.Today I am doing some training - I can look on it at evening
here is a patch
Regards
Pavel
Show quoted text
Regards
Pavel
--
Michael
Attachments:
pg_regress-template-option.patchtext/x-patch; charset=US-ASCII; name=pg_regress-template-option.patchDownload
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index d4d00d9..354b918 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -68,6 +68,7 @@ const char *pretty_diff_opts = "-w -C3";
/* options settable from command line */
_stringlist *dblist = NULL;
+_stringlist *templatelist = NULL;
bool debug = false;
char *inputdir = ".";
char *outputdir = ".";
@@ -1907,7 +1908,7 @@ drop_database_if_exists(const char *dbname)
}
static void
-create_database(const char *dbname)
+create_database(const char *dbname, const char *template)
{
_stringlist *sl;
@@ -1917,10 +1918,12 @@ create_database(const char *dbname)
*/
header(_("creating database \"%s\""), dbname);
if (encoding)
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0 ENCODING='%s'%s", dbname, encoding,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\" ENCODING='%s'%s",
+ dbname, template, encoding,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
else
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0%s", dbname,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\"%s",
+ dbname, template,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
psql_command(dbname,
"ALTER DATABASE \"%s\" SET lc_messages TO 'C';"
@@ -1995,6 +1998,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --template=DB use template DB (default \"template0\")\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_("\n"));
@@ -2041,6 +2045,7 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
{"launcher", required_argument, NULL, 21},
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
+ {"template", required_argument, NULL, 25},
{NULL, 0, NULL, 0}
};
@@ -2154,6 +2159,16 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
case 24:
config_auth_datadir = pg_strdup(optarg);
break;
+ case 25:
+
+ /*
+ * If a default template was specified, we need to remove it
+ * before we add the specified one.
+ */
+ free_stringlist(&templatelist);
+ split_to_stringlist(optarg, ",", &templatelist);
+ break;
+
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2454,8 +2469,25 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
*/
if (!use_existing)
{
- for (sl = dblist; sl; sl = sl->next)
- create_database(sl->str);
+ if (templatelist != NULL)
+ {
+ _stringlist *tl;
+
+ for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl = tl->next)
+ {
+ if (tl != NULL)
+ create_database(sl->str, tl->str);
+ else
+ {
+ fprintf(stderr, _("%s: the template list is shorter than database list\n"),
+ progname);
+ exit(2);
+ }
+ }
+ }
+ else
+ for (sl = dblist; sl; sl = sl->next)
+ create_database(sl->str, "template0");
for (sl = extraroles; sl; sl = sl->next)
create_role(sl->str, dblist);
}
On Thu, Feb 9, 2017 at 5:13 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
here is a patch
Thanks.
- for (sl = dblist; sl; sl = sl->next)
- create_database(sl->str);
+ if (templatelist != NULL)
+ {
+ _stringlist *tl;
+
+ for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl
= tl->next)
+ {
+ if (tl != NULL)
+ create_database(sl->str, tl->str);
+ else
+ {
+ fprintf(stderr, _("%s: the template list is
shorter than database list\n"),
+ progname);
+ exit(2);
+ }
+ }
+ }
+ else
+ for (sl = dblist; sl; sl = sl->next)
+ create_database(sl->str, "template0");
There is one problem here: if the length of the template list is
shorter than the database list, databases get halfly created, then
pg_regress complains, letting the instance in a half-way state. I
think that you had better do any sanity checks before creating or even
dropping existing databases.
--
Michael--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi
2017-02-10 6:00 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Thu, Feb 9, 2017 at 5:13 AM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:here is a patch
Thanks.
- for (sl = dblist; sl; sl = sl->next) - create_database(sl->str); + if (templatelist != NULL) + { + _stringlist *tl; + + for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl = tl->next) + { + if (tl != NULL) + create_database(sl->str, tl->str); + else + { + fprintf(stderr, _("%s: the template list is shorter than database list\n"), + progname); + exit(2); + } + } + } + else + for (sl = dblist; sl; sl = sl->next) + create_database(sl->str, "template0"); There is one problem here: if the length of the template list is shorter than the database list, databases get halfly created, then pg_regress complains, letting the instance in a half-way state. I think that you had better do any sanity checks before creating or even dropping existing databases.
here is new update - check is done before any creating
Regards
Pavel
Show quoted text
--
Michael
Attachments:
pg_regress-template-option-02.patchtext/x-patch; charset=US-ASCII; name=pg_regress-template-option-02.patchDownload
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index d4d00d9..b5f5c2f 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -68,6 +68,7 @@ const char *pretty_diff_opts = "-w -C3";
/* options settable from command line */
_stringlist *dblist = NULL;
+_stringlist *templatelist = NULL;
bool debug = false;
char *inputdir = ".";
char *outputdir = ".";
@@ -1907,7 +1908,7 @@ drop_database_if_exists(const char *dbname)
}
static void
-create_database(const char *dbname)
+create_database(const char *dbname, const char *template)
{
_stringlist *sl;
@@ -1917,10 +1918,12 @@ create_database(const char *dbname)
*/
header(_("creating database \"%s\""), dbname);
if (encoding)
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0 ENCODING='%s'%s", dbname, encoding,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\" ENCODING='%s'%s",
+ dbname, template, encoding,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
else
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0%s", dbname,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\"%s",
+ dbname, template,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
psql_command(dbname,
"ALTER DATABASE \"%s\" SET lc_messages TO 'C';"
@@ -1995,6 +1998,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --template=DB use template DB (default \"template0\")\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_("\n"));
@@ -2041,6 +2045,7 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
{"launcher", required_argument, NULL, 21},
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
+ {"template", required_argument, NULL, 25},
{NULL, 0, NULL, 0}
};
@@ -2154,6 +2159,16 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
case 24:
config_auth_datadir = pg_strdup(optarg);
break;
+ case 25:
+
+ /*
+ * If a default template was specified, we need to remove it
+ * before we add the specified one.
+ */
+ free_stringlist(&templatelist);
+ split_to_stringlist(optarg, ",", &templatelist);
+ break;
+
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2454,8 +2469,35 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
*/
if (!use_existing)
{
- for (sl = dblist; sl; sl = sl->next)
- create_database(sl->str);
+ if (templatelist != NULL)
+ {
+ _stringlist *tl;
+
+ /*
+ * The template list should to have same length as database list.
+ * Check it before any database creation.
+ */
+ for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl = tl->next)
+ if (tl == NULL)
+ {
+ fprintf(stderr, _("%s: the template list is shorter than database list\n"),
+ progname);
+ exit(2);
+ }
+ if (tl != NULL)
+ {
+ fprintf(stderr, _("%s: the template list is longer than database list\n"),
+ progname);
+ exit(2);
+ }
+
+ for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl = tl->next)
+ create_database(sl->str, tl->str);
+ }
+ else
+ for (sl = dblist; sl; sl = sl->next)
+ create_database(sl->str, "template0");
+
for (sl = extraroles; sl; sl = sl->next)
create_role(sl->str, dblist);
}
On Sat, Feb 11, 2017 at 3:03 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
here is new update - check is done before any creating
It may be better to do any checks before dropping existing databases
as well... It would be as well just simpler to complain with a single
error message like "database and template list lengths do not match".
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi
2017-02-13 6:46 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Sat, Feb 11, 2017 at 3:03 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:here is new update - check is done before any creating
It may be better to do any checks before dropping existing databases
as well... It would be as well just simpler to complain with a single
error message like "database and template list lengths do not match".
next step
Regards
Pavel
Show quoted text
--
Michael
Attachments:
pg_regress-template-option-03.patchtext/x-patch; charset=US-ASCII; name=pg_regress-template-option-03.patchDownload
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index d4d00d9c66..ef0542ad0c 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -68,6 +68,7 @@ const char *pretty_diff_opts = "-w -C3";
/* options settable from command line */
_stringlist *dblist = NULL;
+_stringlist *templatelist = NULL;
bool debug = false;
char *inputdir = ".";
char *outputdir = ".";
@@ -1907,7 +1908,7 @@ drop_database_if_exists(const char *dbname)
}
static void
-create_database(const char *dbname)
+create_database(const char *dbname, const char *template)
{
_stringlist *sl;
@@ -1917,10 +1918,12 @@ create_database(const char *dbname)
*/
header(_("creating database \"%s\""), dbname);
if (encoding)
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0 ENCODING='%s'%s", dbname, encoding,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\" ENCODING='%s'%s",
+ dbname, template, encoding,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
else
- psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=template0%s", dbname,
+ psql_command("postgres", "CREATE DATABASE \"%s\" TEMPLATE=\"%s\"%s",
+ dbname, template,
(nolocale) ? " LC_COLLATE='C' LC_CTYPE='C'" : "");
psql_command(dbname,
"ALTER DATABASE \"%s\" SET lc_messages TO 'C';"
@@ -1995,6 +1998,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --template=DB use template DB (default \"template0\")\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_("\n"));
@@ -2041,10 +2045,12 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
{"launcher", required_argument, NULL, 21},
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
+ {"template", required_argument, NULL, 25},
{NULL, 0, NULL, 0}
};
_stringlist *sl;
+ _stringlist *tl;
int c;
int i;
int option_index;
@@ -2154,6 +2160,16 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
case 24:
config_auth_datadir = pg_strdup(optarg);
break;
+ case 25:
+
+ /*
+ * If a default template was specified, we need to remove it
+ * before we add the specified one.
+ */
+ free_stringlist(&templatelist);
+ split_to_stringlist(optarg, ",", &templatelist);
+ break;
+
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2205,6 +2221,18 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
unlimit_core_size();
#endif
+ /* The length of template list should be same like db list */
+ if (templatelist != NULL)
+ {
+ for (sl = dblist, tl = templatelist; sl && tl; sl = sl->next, tl = tl->next);
+ if (sl || tl)
+ {
+ fprintf(stderr, _("%s: database and template list lengths do not match\n"),
+ progname);
+ exit(2);
+ }
+ }
+
if (temp_instance)
{
FILE *pg_conf;
@@ -2454,8 +2482,17 @@ regression_main(int argc, char *argv[], init_function ifunc, test_function tfunc
*/
if (!use_existing)
{
- for (sl = dblist; sl; sl = sl->next)
- create_database(sl->str);
+ if (templatelist != NULL)
+ {
+ for (sl = dblist, tl = templatelist; sl; sl = sl->next, tl = tl->next)
+ create_database(sl->str, tl->str);
+ }
+ else
+ {
+ for (sl = dblist; sl; sl = sl->next)
+ create_database(sl->str, "template0");
+ }
+
for (sl = extraroles; sl; sl = sl->next)
create_role(sl->str, dblist);
}
On 2017-02-13 20:59:43 +0100, Pavel Stehule wrote:
Hi
2017-02-13 6:46 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Sat, Feb 11, 2017 at 3:03 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:here is new update - check is done before any creating
It may be better to do any checks before dropping existing databases
as well... It would be as well just simpler to complain with a single
error message like "database and template list lengths do not match".next step
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, Feb 14, 2017 at 11:36 AM, Andres Freund <andres@anarazel.de> wrote:
On 2017-02-13 20:59:43 +0100, Pavel Stehule wrote:
Hi
2017-02-13 6:46 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Sat, Feb 11, 2017 at 3:03 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:here is new update - check is done before any creating
It may be better to do any checks before dropping existing databases
as well... It would be as well just simpler to complain with a single
error message like "database and template list lengths do not match".next step
This looks fine to me.
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.
Some tests create objects without removing them, meaning that
continuous runs would fail with only --use-existing. This patch brings
value in such cases.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-02-14 11:46:52 +0900, Michael Paquier wrote:
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.Some tests create objects without removing them, meaning that
continuous runs would fail with only --use-existing. This patch brings
value in such cases.
You can trivially script the CREATE/DROP DB outside with
--use-existing. Which seems a lot more flexible than adding more and
more options to pg_regress.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
2017-02-14 3:50 GMT+01:00 Andres Freund <andres@anarazel.de>:
On 2017-02-14 11:46:52 +0900, Michael Paquier wrote:
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.
Some tests create objects without removing them, meaning that
continuous runs would fail with only --use-existing. This patch brings
value in such cases.You can trivially script the CREATE/DROP DB outside with
--use-existing. Which seems a lot more flexible than adding more and
more options to pg_regress.
Using template is natural and very simply solution - more it doesn't need
any outer scripts - so infrastructure for test can be pretty simply in this
case.
Regards
Pavel
2017-02-14 3:36 GMT+01:00 Andres Freund <andres@anarazel.de>:
On 2017-02-13 20:59:43 +0100, Pavel Stehule wrote:
Hi
2017-02-13 6:46 GMT+01:00 Michael Paquier <michael.paquier@gmail.com>:
On Sat, Feb 11, 2017 at 3:03 PM, Pavel Stehule <
pavel.stehule@gmail.com>
wrote:
here is new update - check is done before any creating
It may be better to do any checks before dropping existing databases
as well... It would be as well just simpler to complain with a single
error message like "database and template list lengths do not match".next step
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.
vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.
I checked it - and it is not hard - but you have to overwrite some makefile
rules - and then you spend some time with makefile hacking
Possibility to set template removes all this dirty work. Setting
"REGRESS_OPTS += --template=mytests-template" is simple, clean and readable
Regards
Pavel
Show quoted text
- Andres
On 2/13/17 8:50 PM, Andres Freund wrote:
On 2017-02-14 11:46:52 +0900, Michael Paquier wrote:
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.Some tests create objects without removing them, meaning that
continuous runs would fail with only --use-existing. This patch brings
value in such cases.You can trivially script the CREATE/DROP DB outside with
--use-existing. Which seems a lot more flexible than adding more and
more options to pg_regress.
AFAIK if you're doing make check (as opposed to installcheck) it's
significantly more complicated than that since you'd have to create a
temp cluster/install yourself.
As an extension author, I'd *love* to have the cluster management stuff
in pg_regress broken out: it's the only reason I use pg_regress, and
pg_regress's idea of what a test failure is just gets in my way. But
breaking that out is far more invasive than allowing a template database.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-02-14 12:33:35 -0600, Jim Nasby wrote:
On 2/13/17 8:50 PM, Andres Freund wrote:
On 2017-02-14 11:46:52 +0900, Michael Paquier wrote:
I still fail to see why --use-existing as suggested in
/messages/by-id/20170208002900.vkldujzfkwbvqqq7@alap3.anarazel.de
isn't sufficient.Some tests create objects without removing them, meaning that
continuous runs would fail with only --use-existing. This patch brings
value in such cases.You can trivially script the CREATE/DROP DB outside with
--use-existing. Which seems a lot more flexible than adding more and
more options to pg_regress.AFAIK if you're doing make check (as opposed to installcheck) it's
significantly more complicated than that since you'd have to create a temp
cluster/install yourself.
But in that case you can't have useful templates in the regression test
either, so the whole discussion is moot?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2/14/17 1:59 PM, Andres Freund wrote:
AFAIK if you're doing make check (as opposed to installcheck) it's
significantly more complicated than that since you'd have to create a temp
cluster/install yourself.But in that case you can't have useful templates in the regression test
either, so the whole discussion is moot?
At that point it depends on what you're trying to do. Presumably
separating cluster control would make it much easier to script
createdb/dropdb as you suggested. Tom's use case might be more easily
served by specifying a template database. I don't think Pavel ever
posted his use case.
Speaking for myself, my normal pattern is to have a number of separate
pg_regress suites, each of which ends up loading the extension under
test. Loading a large extension can end up being very time consuming;
enough so that I'd expect it to be much faster to create the temp
cluster, load all the prereq's once in some template database, and then
use that template for most/all of the tests. In that scenario separating
cluster create/drop would certainly be useful, but the template option
would probably be helpful as well (though since pg_regress' diff-based
methodology just gets in my way I'd likely use some other harness to
actually run the tests).
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-02-14 14:29:56 -0600, Jim Nasby wrote:
On 2/14/17 1:59 PM, Andres Freund wrote:
AFAIK if you're doing make check (as opposed to installcheck) it's
significantly more complicated than that since you'd have to create a temp
cluster/install yourself.But in that case you can't have useful templates in the regression test
either, so the whole discussion is moot?At that point it depends on what you're trying to do. Presumably separating
cluster control would make it much easier to script createdb/dropdb as you
suggested.
That's not what I responded to...
Tom's use case might be more easily served by specifying a
template database. I don't think Pavel ever posted his use case.
Wait, that's precisely what Pavel asked?
On 2017-02-07 16:43:47 +0100, Pavel Stehule wrote:
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.
Obviously that only makes sense with installcheck.
Speaking for myself, my normal pattern is to have a number of separate
pg_regress suites, each of which ends up loading the extension under test.
Loading a large extension can end up being very time consuming; enough so
that I'd expect it to be much faster to create the temp cluster, load all
the prereq's once in some template database, and then use that template for
most/all of the tests.
I seriously doubt that. CREATE DATABASE is ridiculously expensive,
copies everything on the file-level and requires checkpoints. If your
extension is more expensive than that, I'd say you're likely doing
something wrong.
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Dne 14. 2. 2017 21:35 napsal uživatel "Andres Freund" <andres@anarazel.de>:
On 2017-02-14 14:29:56 -0600, Jim Nasby wrote:
On 2/14/17 1:59 PM, Andres Freund wrote:
AFAIK if you're doing make check (as opposed to installcheck) it's
significantly more complicated than that since you'd have to create a
temp
cluster/install yourself.
But in that case you can't have useful templates in the regression test
either, so the whole discussion is moot?At that point it depends on what you're trying to do. Presumably
separating
cluster control would make it much easier to script createdb/dropdb as you
suggested.
That's not what I responded to...
Tom's use case might be more easily served by specifying a
template database. I don't think Pavel ever posted his use case.
Wait, that's precisely what Pavel asked?
I would to use regress test environment in my current case. 99% code in
plpgsql, but there is pretty complex schema. About 300 tables. 1k views. 2k
functions. Import schema is slow. Database clonning is much faster.
On 2017-02-07 16:43:47 +0100, Pavel Stehule wrote:
Is possible to specify template database for pg_regress?
I have to run tests on database with thousands database objects. Using
template is much faster than import these objects.
Obviously that only makes sense with installcheck.
Speaking for myself, my normal pattern is to have a number of separate
pg_regress suites, each of which ends up loading the extension under test.
Loading a large extension can end up being very time consuming; enough so
that I'd expect it to be much faster to create the temp cluster, load all
the prereq's once in some template database, and then use that template
for
most/all of the tests.
I seriously doubt that. CREATE DATABASE is ridiculously expensive,
copies everything on the file-level and requires checkpoints. If your
extension is more expensive than that, I'd say you're likely doing
something wrong.
- Andres
On 2/14/17 2:49 PM, Pavel Stehule wrote:
Tom's use case might be more easily served by specifying a
template database. I don't think Pavel ever posted his use case.Wait, that's precisely what Pavel asked?
I would to use regress test environment in my current case. 99% code in
plpgsql, but there is pretty complex schema. About 300 tables. 1k views.
2k functions. Import schema is slow. Database clonning is much faster.
FWIW, for actual production environments (which I assume this is), I
find pg_regress to be completely useless. Some simple shell scripts to
build the database (possibly using sqitch) and then a script around
pg_prove is what I normally use. https://github.com/BlueTreble/db_tools
gives you the general idea.
Speaking for myself, my normal pattern is to have a number of separate
pg_regress suites, each of which ends up loading the extension under test.
Loading a large extension can end up being very time consuming; enough so
that I'd expect it to be much faster to create the temp cluster, load all
the prereq's once in some template database, and then use that template for
most/all of the tests.I seriously doubt that. CREATE DATABASE is ridiculously expensive,
copies everything on the file-level and requires checkpoints. If your
extension is more expensive than that, I'd say you're likely doing
something wrong.
That depends on the extension. pgTap for example contains over 900
functions. A quick test on my laptop shows it's faster to create a
database from a template containing the extension than it is to create
the extension itself.
decibel@decina:[12:33]~$time createdb t
real 0m0.433s
user 0m0.004s
sys 0m0.009s
decibel@decina:[12:34]~$time psql -c 'create extension pgtap' t
CREATE EXTENSION
real 0m0.559s
user 0m0.002s
sys 0m0.007s
decibel@decina:[12:34]~$time createdb -T t t2
real 0m0.441s
user 0m0.002s
sys 0m0.005s
decibel@decina:[12:34]~$time psql -c 'drop extension pgtap' t
DROP EXTENSION
real 0m0.197s
user 0m0.002s
sys 0m0.006s
decibel@decina:[12:34]~$time dropdb t
real 0m0.189s
user 0m0.003s
sys 0m0.007s
decibel@decina:[12:34]~$time dropdb t2
real 0m0.154s
user 0m0.002s
sys 0m0.005s
decibel@decina:[12:34]~$
Interestingly, CREATE EXTENSION is 2x faster than simply running the file:
decibel@decina:[12:38]~$time psql -qtf
$PGDATA/../share/extension/pgtap--0.97.0.sql t
real 0m1.225s
user 0m0.044s
sys 0m0.028s
decibel@decina:[12:39]~$
decibel@decina:[12:41]~$(echo 'begin;'; echo "\i
$PGDATA/../share/extension/pgtap--0.97.0.sql"; echo 'commit;') | time
psql -qt t
1.12 real 0.04 user 0.02 sys
decibel@decina:[12:41]~$
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers