pgbench: Skipping the creating primary keys after initialization

Started by Masahiko Sawadaover 8 years ago70 messages
#1Masahiko Sawada
sawada.mshk@gmail.com
1 attachment(s)

Hi all,

I'd like to propose a new option -I for pgbench command which skips
the creating primary keys after initialized tables. This option is
useful for users who want to do bench marking with no index or indexes
other than btree primary index. If we initialize pgbench tables at a
large number scale factor the primary key index creation takes a long
time even if we're going to use other types of indexes. With this
option, the initialization time is reduced and you can create indexes
as you want.

Feedback is very welcome. I'll add this patch to the next CF.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

skip_building_pkeys_after_initialized.patchapplication/octet-stream; name=skip_building_pkeys_after_initialized.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..ec63992 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -248,6 +248,15 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
       </listitem>
      </varlistentry>
 
+     <varlistentry>
+      <term><option>--no-primary-keys</option></term>
+      <listitem>
+       <para>
+        Create no primary keys after initialization.
+       </para>
+      </listitem>
+     </varlistentry>
+     
     </variablelist>
    </para>
 
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 4d364a1..c475d43 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -477,6 +477,7 @@ usage(void)
 		   "  -i, --initialize         invokes initialization mode\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
+		   "  -I, --no-primary-keys    skip building primary keys after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
 		   "  -s, --scale=NUM          scaling factor\n"
 		   "  --foreign-keys           create foreign key constraints between tables\n"
@@ -2568,7 +2569,7 @@ disconnect_all(CState *state, int length)
 
 /* create tables and setup data */
 static void
-init(bool is_no_vacuum)
+init(bool is_no_vacuum, bool is_no_pkeys)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2806,25 +2807,28 @@ init(bool is_no_vacuum)
 	/*
 	 * create indexes
 	 */
-	fprintf(stderr, "set primary keys...\n");
-	for (i = 0; i < lengthof(DDLINDEXes); i++)
+	if (!is_no_pkeys)
 	{
-		char		buffer[256];
+		fprintf(stderr, "set primary keys...\n");
+		for (i = 0; i < lengthof(DDLINDEXes); i++)
+		{
+			char		buffer[256];
 
-		strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
+			strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
 
-		if (index_tablespace != NULL)
-		{
-			char	   *escape_tablespace;
+			if (index_tablespace != NULL)
+			{
+				char	   *escape_tablespace;
 
-			escape_tablespace = PQescapeIdentifier(con, index_tablespace,
-												   strlen(index_tablespace));
-			snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
-					 " using index tablespace %s", escape_tablespace);
-			PQfreemem(escape_tablespace);
-		}
+				escape_tablespace = PQescapeIdentifier(con, index_tablespace,
+													   strlen(index_tablespace));
+				snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
+						 " using index tablespace %s", escape_tablespace);
+				PQfreemem(escape_tablespace);
+			}
 
-		executeStatement(con, buffer);
+			executeStatement(con, buffer);
+		}
 	}
 
 	/*
@@ -3623,6 +3627,7 @@ main(int argc, char **argv)
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
 		{"no-vacuum", no_argument, NULL, 'n'},
+		{"no-primary-keys", no_argument, NULL, 'I'},
 		{"port", required_argument, NULL, 'p'},
 		{"progress", required_argument, NULL, 'P'},
 		{"protocol", required_argument, NULL, 'M'},
@@ -3651,6 +3656,7 @@ main(int argc, char **argv)
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
 	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	int			is_no_pkeys = 0;	/* no primary keys build at after initialization */
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3711,7 +3717,7 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	while ((c = getopt_long(argc, argv, "iIh:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3862,6 +3868,9 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				internal_script_used = true;
 				break;
+			case 'I':
+				is_no_pkeys++;
+				break;
 			case 'f':
 				weight = parseScriptWeight(optarg, &script);
 				process_file(script, weight);
@@ -4052,7 +4061,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(is_no_vacuum, is_no_pkeys);
 		exit(0);
 	}
 	else
#2Robert Haas
robertmhaas@gmail.com
In reply to: Masahiko Sawada (#1)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Aug 1, 2017 at 9:49 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I'd like to propose a new option -I for pgbench command which skips
the creating primary keys after initialized tables. This option is
useful for users who want to do bench marking with no index or indexes
other than btree primary index. If we initialize pgbench tables at a
large number scale factor the primary key index creation takes a long
time even if we're going to use other types of indexes. With this
option, the initialization time is reduced and you can create indexes
as you want.

Feedback is very welcome. I'll add this patch to the next CF.

I support adding an option for this, but I propose that we just make
it a long-form option, similar to --log-prefix or --index-tablespace.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Robert Haas (#2)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 10:25 PM, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 1, 2017 at 9:49 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I'd like to propose a new option -I for pgbench command which skips
the creating primary keys after initialized tables. This option is
useful for users who want to do bench marking with no index or indexes
other than btree primary index. If we initialize pgbench tables at a
large number scale factor the primary key index creation takes a long
time even if we're going to use other types of indexes. With this
option, the initialization time is reduced and you can create indexes
as you want.

Feedback is very welcome. I'll add this patch to the next CF.

I support adding an option for this, but I propose that we just make
it a long-form option, similar to --log-prefix or --index-tablespace.

Yeah, that's better. I'll update the patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#2)
Re: pgbench: Skipping the creating primary keys after initialization

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Aug 1, 2017 at 9:49 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I'd like to propose a new option -I for pgbench command which skips
the creating primary keys after initialized tables.

I support adding an option for this, but I propose that we just make
it a long-form option, similar to --log-prefix or --index-tablespace.

I think we could probably do without this ... if you want a non-default
test setup, why do you need to use "pgbench -i" to create it?

It's not that there's anything greatly wrong with this particular idea,
it's just that pgbench has too many switches already, and omitting random
subsets of the initialization actions doesn't seem like it contributes
fundamental new benchmarking capability.

I could get behind a proposal that generalized pgbench's "-i" behavior
in some meaningful way. I wonder whether it would be possible to convert
that behavior into a script. Some of what it does is just SQL commands
with injected parameters, which pgbench does already. There's also
data-loading actions, which could be converted to backslash commands
perhaps. Then desires like this could be addressed by invoking a
customized script instead of complicating pgbench's option set.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Tatsuo Ishii
ishii@sraoss.co.jp
In reply to: Tom Lane (#4)
Re: pgbench: Skipping the creating primary keys after initialization

I think we could probably do without this ... if you want a non-default
test setup, why do you need to use "pgbench -i" to create it?

It's not that there's anything greatly wrong with this particular idea,
it's just that pgbench has too many switches already, and omitting random
subsets of the initialization actions doesn't seem like it contributes
fundamental new benchmarking capability.

I could get behind a proposal that generalized pgbench's "-i" behavior
in some meaningful way. I wonder whether it would be possible to convert
that behavior into a script. Some of what it does is just SQL commands
with injected parameters, which pgbench does already. There's also
data-loading actions, which could be converted to backslash commands
perhaps. Then desires like this could be addressed by invoking a
customized script instead of complicating pgbench's option set.

+1.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#4)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 9:41 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Aug 1, 2017 at 9:49 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I'd like to propose a new option -I for pgbench command which skips
the creating primary keys after initialized tables.

I support adding an option for this, but I propose that we just make
it a long-form option, similar to --log-prefix or --index-tablespace.

I think we could probably do without this ... if you want a non-default
test setup, why do you need to use "pgbench -i" to create it?

It's not that there's anything greatly wrong with this particular idea,
it's just that pgbench has too many switches already, and omitting random
subsets of the initialization actions doesn't seem like it contributes
fundamental new benchmarking capability.

I could get behind a proposal that generalized pgbench's "-i" behavior
in some meaningful way. I wonder whether it would be possible to convert
that behavior into a script. Some of what it does is just SQL commands
with injected parameters, which pgbench does already. There's also
data-loading actions, which could be converted to backslash commands
perhaps. Then desires like this could be addressed by invoking a
customized script instead of complicating pgbench's option set.

I've actually wanted this exact thing multiple times: most recently,
to make a non-unique btree index instead of a unique one, and to make
a hash index instead of a btree one. I don't object to a modest
effort at coming up with a more general mechanism here, but I also
think the switch as proposed is something that would have met my real
needs on multiple occasions. I've probably had 10 different occasions
when I wanted all of the standard pgbench initialization *except for*
something different about the indexes.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#6)
Re: pgbench: Skipping the creating primary keys after initialization

Robert Haas <robertmhaas@gmail.com> writes:

I've actually wanted this exact thing multiple times: most recently,
to make a non-unique btree index instead of a unique one, and to make
a hash index instead of a btree one. I don't object to a modest
effort at coming up with a more general mechanism here, but I also
think the switch as proposed is something that would have met my real
needs on multiple occasions. I've probably had 10 different occasions
when I wanted all of the standard pgbench initialization *except for*
something different about the indexes.

Sure, but "no indexes at all" is hardly ever the real goal, is it?
So the switch as proposed is only solving part of your problem.
I'd rather see a solution that addresses a larger range of desires.

Or in other words, this looks to me quite a bit like the hackery
that resulted in pgbench's -S and -N options, before we figured out
that making it scriptable was a better answer.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#7)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 10:17 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Sure, but "no indexes at all" is hardly ever the real goal, is it?

Right.

So the switch as proposed is only solving part of your problem.
I'd rather see a solution that addresses a larger range of desires.

That's reasonable.

Or in other words, this looks to me quite a bit like the hackery
that resulted in pgbench's -S and -N options, before we figured out
that making it scriptable was a better answer.

But it's not very clear to me how we could make this case scriptable,
and it would probably not be much different from just using the
proposed option and then running the script afterwards yourself via
psql. The thing about -N and -S is that those scripts are being run
repeatedly, so pgbench has to be involved. If you just want to create
different/extra indexes, you can do that yourself.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#8)
Re: pgbench: Skipping the creating primary keys after initialization

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Aug 2, 2017 at 10:17 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Or in other words, this looks to me quite a bit like the hackery
that resulted in pgbench's -S and -N options, before we figured out
that making it scriptable was a better answer.

But it's not very clear to me how we could make this case scriptable,

Well, I'm imagining that "-i" would essentially become a short form
of "-b initialize", as already happened for -S and -N, where the script
looks something like

drop table if exists pgbench_branches;
create table pgbench_branches (
bid int not null,bbalance int,filler char(88)
);
\load_data pgbench_branches [ other parameters to-be-determined ]
alter table pgbench_branches add primary key (bid);
... repeat for other tables ...

and we'd document that the same way we do for the existing built-in
scripts. Then, if there's something you don't like about it, you
just paste the script into a file and edit to taste.

I'm sure there's complexities that would only become apparent when
someone tries to write the patch, but that seems to me like a better
foundation for this class of desires than extending the option set
with various one-off options having no discernible architecture.

If you just want to create
different/extra indexes, you can do that yourself.

Sure, but there's no end to the number of small variations on this
theme that somebody might want. For example, we realized years ago
that the "filler" fields as-implemented don't really meet the intent
of the TPC-B spec (cf comment in the init() function). If someone
comes along with a patch adding a "--really-tpc-b" option to change
the table declarations and/or data loading code to fix that, will we
take that patch? What about one that wants all the id fields (not
just accounts.aid) to be bigint, or one that wants the balance fields
to be numeric?

You can say "let 'em set up the tables manually if they want that",
but I don't see why a nonstandard set of indexes is much different.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#10Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#9)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 11:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Aug 2, 2017 at 10:17 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Or in other words, this looks to me quite a bit like the hackery
that resulted in pgbench's -S and -N options, before we figured out
that making it scriptable was a better answer.

But it's not very clear to me how we could make this case scriptable,

Well, I'm imagining that "-i" would essentially become a short form
of "-b initialize", as already happened for -S and -N, where the script
looks something like

drop table if exists pgbench_branches;
create table pgbench_branches (
bid int not null,bbalance int,filler char(88)
);
\load_data pgbench_branches [ other parameters to-be-determined ]
alter table pgbench_branches add primary key (bid);
... repeat for other tables ...

and we'd document that the same way we do for the existing built-in
scripts. Then, if there's something you don't like about it, you
just paste the script into a file and edit to taste.

I imagine that would be useful for some use cases, but it's a heck of
a lot more work than just writing --no-indexes-please.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#10)
Re: pgbench: Skipping the creating primary keys after initialization

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Aug 2, 2017 at 11:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Well, I'm imagining that "-i" would essentially become a short form
of "-b initialize", as already happened for -S and -N, where the script
looks something like ...

I imagine that would be useful for some use cases, but it's a heck of
a lot more work than just writing --no-indexes-please.

Of course. It's also a heck of a lot more flexible. Adding on another
ad-hoc option that does the minimum possible amount of work needed to
address one specific problem is always going to be less work; but after
we repeat that process five or ten times, we're going to have a mess.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#11)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 12:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Of course. It's also a heck of a lot more flexible. Adding on another
ad-hoc option that does the minimum possible amount of work needed to
address one specific problem is always going to be less work; but after
we repeat that process five or ten times, we're going to have a mess.

Well, I still like Masahiko-san's proposal, but I'm not prepared to
keep arguing about it right now. Maybe some other people will weigh
in with an opinion.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#13Michael Paquier
michael.paquier@gmail.com
In reply to: Tom Lane (#9)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 2, 2017 at 5:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Aug 2, 2017 at 10:17 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Or in other words, this looks to me quite a bit like the hackery
that resulted in pgbench's -S and -N options, before we figured out
that making it scriptable was a better answer.

But it's not very clear to me how we could make this case scriptable,

Well, I'm imagining that "-i" would essentially become a short form
of "-b initialize", as already happened for -S and -N, where the script
looks something like

Yes, I would imagine a facility where one could do pgbench $script and
issue a complete test set. Here is for example a funky idea: let's
separate each script with a set of meta-commands, \init being what is
used just for initialization, and then use \script to define a set of
commands with a custom weight. Say:
\init
CREATE TABLE foo(a int);
\script select_query [weight N]
SELECT count(*) FROM foo;
\script insert_query [weight N]
INSERT INTO foo VALUES ('1');

That may be over-engineering things, but personally I don't like much
having just a switch to remove indexes. Next time we will come with
another option that only selects a portion of the indexes created.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Robert Haas (#12)
Re: pgbench: Skipping the creating primary keys after initialization

On Thu, Aug 3, 2017 at 2:00 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Aug 2, 2017 at 12:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Of course. It's also a heck of a lot more flexible. Adding on another
ad-hoc option that does the minimum possible amount of work needed to
address one specific problem is always going to be less work; but after
we repeat that process five or ten times, we're going to have a mess.

Well, I still like Masahiko-san's proposal, but I'm not prepared to
keep arguing about it right now. Maybe some other people will weigh
in with an opinion.

My motivation of this proposal is same as what Robert has. I
understand that ad-hoc option can solve only the part of big problem
and it could be cause of mess. However It seems me that the script
especially for table initialization will not be flexible than we
expected. I mean, even if we provide some meta commands for table
initialization or data loading, these meta commands work for only
pgbench tables (i.g., pgbench_accounts, pgbench_branches and so on).
If we want to create other tables and load data to them as we want we
can do that using psql -f. So an alternative ways is having a flexible
style option for example --custom-initialize = { [load, create_pkey,
create_fkey, vacuum], ... }. That would solve this in a better way.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#15Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#14)
Re: pgbench: Skipping the creating primary keys after initialization

Hello,

My motivation of this proposal is same as what Robert has. I
understand that ad-hoc option can solve only the part of big problem
and it could be cause of mess. However It seems me that the script
especially for table initialization will not be flexible than we
expected. I mean, even if we provide some meta commands for table
initialization or data loading, these meta commands work for only
pgbench tables (i.g., pgbench_accounts, pgbench_branches and so on).
If we want to create other tables and load data to them as we want we
can do that using psql -f. So an alternative ways is having a flexible
style option for example --custom-initialize = { [load, create_pkey,
create_fkey, vacuum], ... }. That would solve this in a better way.

Personnaly, I could be fine with a limited number of long options to
adjust pgbench initialization to various needs, eg --use-hash-index,
--skip-whetever-index, etc.

The flexible --custom-init idea outlined above looks nice as well.

As for a more generic solution, the easy part are the "CREATE" stuff and
the transaction script stuff (existing pgbench scripts).

For the CREATE stuff, the script language is SQL, the command to use it is
"psql"...

The real and hard part is to fill tables with meaningful pseudo-random
test data which do not violate constraints for any non trivial schema
involving foreign keys and various unique constraints.

The solution for this is SQL for trivial cases, think of:

"INSERT INTO Foo() SELECT ... FROM generate_series(...);"

For instance the pgbench initialization is really close to:

psql -Dscale=10 <<EOF
CREATE TABLE ... ;
INSERT INTO pgbench_account(...)
SELECT ... FROM generate_series(1, 100000 * :scale) AS i;
INSERT ...
CREATE INDEX ...;
VACUUM FULL ANALYZE;
EOF

And all existing options could probably be implemented easilly with the
recently added conditional (\if).

So my 0.02ᅵ is that if something is to be done, I would suggest to turn
the creation and initialization stuff into a standard "psql" script that
could be called from pgbench instead of integrating much more ad-hoc stuff
into pgbench.

Note that non trivial schema initialization requires more general
programming, so I do not believe in doing a lot at pgbench or psql levels.
The best I could come with is a data generator which takes as input the
schema with added directives on how to generate the various attributes
(tool named "datafiller", that some people use:-).

--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Tom Lane
tgl@sss.pgh.pa.us
In reply to: Fabien COELHO (#15)
Re: pgbench: Skipping the creating primary keys after initialization

Fabien COELHO <coelho@cri.ensmp.fr> writes:

As for a more generic solution, the easy part are the "CREATE" stuff and
the transaction script stuff (existing pgbench scripts).

For the CREATE stuff, the script language is SQL, the command to use it is
"psql"...

The real and hard part is to fill tables with meaningful pseudo-random
test data which do not violate constraints for any non trivial schema
involving foreign keys and various unique constraints.

The solution for this is SQL for trivial cases, think of:
"INSERT INTO Foo() SELECT ... FROM generate_series(...);"

Yeah. I was also thinking that complicated data-generation requirements
could be handled with plpgsql DO blocks, avoiding the need for hard-wired
code inside pgbench.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#17Tom Lane
tgl@sss.pgh.pa.us
In reply to: Masahiko Sawada (#14)
Re: pgbench: Skipping the creating primary keys after initialization

Masahiko Sawada <sawada.mshk@gmail.com> writes:

If we want to create other tables and load data to them as we want we
can do that using psql -f. So an alternative ways is having a flexible
style option for example --custom-initialize = { [load, create_pkey,
create_fkey, vacuum], ... }. That would solve this in a better way.

FWIW, I like that significantly better than your original proposal.
It'd allow people to execute parts of pgbench's standard initialization
sequence and then do other things in between (in psql). Realistically,
that's probably about as much win as we need here --- if you're veering
far enough away from the standard scenario that that doesn't do it for
you, you might as well just write an all-custom setup script in psql.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Tom Lane (#16)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

For the CREATE stuff, the script language is SQL, the command to use it is
"psql"...

The real and hard part is to fill tables with meaningful pseudo-random
test data which do not violate constraints for any non trivial schema
involving foreign keys and various unique constraints.

The solution for this is SQL for trivial cases, think of:
"INSERT INTO Foo() SELECT ... FROM generate_series(...);"

Yeah. I was also thinking that complicated data-generation requirements
could be handled with plpgsql DO blocks, avoiding the need for hard-wired
code inside pgbench.

I do not thing that it is really be needed for what pgbench does, though.
See attached attempt, including a no_foreign_keys option.

The only tricky thing is to have the elapsed/remaining advancement report
on stdout, maybe with some PL/pgSQL.

Timings are very similar compared to "pgbench -i".

--
Fabien.

Attachments:

pgbench_init.sqlapplication/x-sql; name=pgbench_init.sqlDownload
#19Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#18)
Re: pgbench: Skipping the creating primary keys after initialization

On Fri, Aug 4, 2017 at 3:24 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

For the CREATE stuff, the script language is SQL, the command to use it
is
"psql"...

The real and hard part is to fill tables with meaningful pseudo-random
test data which do not violate constraints for any non trivial schema
involving foreign keys and various unique constraints.

The solution for this is SQL for trivial cases, think of:
"INSERT INTO Foo() SELECT ... FROM generate_series(...);"

Yeah. I was also thinking that complicated data-generation requirements
could be handled with plpgsql DO blocks, avoiding the need for hard-wired
code inside pgbench.

I do not thing that it is really be needed for what pgbench does, though.
See attached attempt, including a no_foreign_keys option.

The only tricky thing is to have the elapsed/remaining advancement report on
stdout, maybe with some PL/pgSQL.

Timings are very similar compared to "pgbench -i".

The generating data with plpgsql DO blocks means that we do the
data-generation on sever side rather than on client side. I think it's
preferable in a sense because could speed up initialization time by
reducing the network traffic.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#20Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Tom Lane (#17)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Thu, Aug 3, 2017 at 11:31 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Masahiko Sawada <sawada.mshk@gmail.com> writes:

If we want to create other tables and load data to them as we want we
can do that using psql -f. So an alternative ways is having a flexible
style option for example --custom-initialize = { [load, create_pkey,
create_fkey, vacuum], ... }. That would solve this in a better way.

FWIW, I like that significantly better than your original proposal.
It'd allow people to execute parts of pgbench's standard initialization
sequence and then do other things in between (in psql). Realistically,
that's probably about as much win as we need here --- if you're veering
far enough away from the standard scenario that that doesn't do it for
you, you might as well just write an all-custom setup script in psql.

Attached patch introduces --custom-initialize option that allows us to
specify the initialization step and its order. For example, If you
want to skip building primary keys you can specify
--custom-initialize="create_table, load_data, vacuum". Since each
custom initialization commands is invoked in specified order, for
example we also can create primary keys *before* loading data. The
data-generation is doing on client side, so progress information for
initialization is still supported.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization.patchapplication/octet-stream; name=pgbench_custom_initialization.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..8e3d090 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -25,6 +25,12 @@
   </cmdsynopsis>
   <cmdsynopsis>
    <command>pgbench</command>
+   <arg choice="plain"><option>--custom-initailize={<replaceable>custom_init_command</replaceable> [, ...]}</option></arg>
+   <arg rep="repeat"><replaceable>option</replaceable></arg>
+   <arg choice="opt"><replaceable>dbname</replaceable></arg>
+  </cmdsynopsis>
+  <cmdsynopsis>
+   <command>pgbench</command>
    <arg rep="repeat"><replaceable>option</replaceable></arg>
    <arg choice="opt"><replaceable>dbname</replaceable></arg>
   </cmdsynopsis>
@@ -72,10 +78,10 @@ tps = 85.296346 (excluding connections establishing)
   <para>
    The default TPC-B-like transaction test requires specific tables to be
    set up beforehand.  <application>pgbench</> should be invoked with
-   the <option>-i</> (initialize) option to create and populate these
-   tables.  (When you are testing a custom script, you don't need this
-   step, but will instead need to do whatever setup your test needs.)
-   Initialization looks like:
+   the <option>-i</> (initialize) or <option>--custom-initialize</> option
+   to create and populate these tables.  (When you are testing a custom
+   script, you don't need this step, but will instead need to do whatever
+   setup your test needs.) Initialization looks like:
 
 <programlisting>
 pgbench -i <optional> <replaceable>other-options</> </optional> <replaceable>dbname</>
@@ -211,6 +217,64 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [, ...]}</option></term>
+      <listitem>
+       <para>
+        Required to invoke custom initialization mode.
+        <replaceable>custom_init_command</replaceable> specified custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>create_table</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>,
+            and <structname>pgbench_tellers</>, destroying any existing
+            tables of these names.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>load_data</literal></term>
+          <listitem>
+           <para>
+            Load data to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>vacuum</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>create_pkey</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>create_fkey</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--foreign-keys</option></term>
       <listitem>
        <para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 4d364a1..8cb9d3a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -36,6 +36,7 @@
 #include "getopt_long.h"
 #include "libpq-fe.h"
 #include "portability/instr_time.h"
+#include "fe_utils/simple_list.h"
 
 #include <ctype.h>
 #include <float.h>
@@ -445,7 +446,6 @@ static const BuiltinScript builtin_script[] =
 	}
 };
 
-
 /* Function prototypes */
 static void setIntValue(PgBenchValue *pv, int64 ival);
 static void setDoubleValue(PgBenchValue *pv, double dval);
@@ -458,7 +458,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
-
+static bool parseCustomInitialize(char *rawstring, SimpleStringList *list);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 /* callback functions for our flex lexer */
 static const PsqlScanCallbacks pgbench_callbacks = {
@@ -484,6 +489,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  --custom-initilize={[create_table|load_data|vacuum|create_pkey|create_fkey] [, ...]}\n"
+		   "                           invokes custom initialization\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2573,78 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+
+/*
+ * Split rawstring by ',' and parse the custom initialization
+ * command; return a list of custom initialization command.
+ */
+static bool
+parseCustomInitialize(char *rawstring, SimpleStringList *list)
+{
+	char	*nextp = rawstring;
+	bool	done = false;
+
+	while (isspace(*nextp))
+		nextp++;	/* skip leading whitespace */
+
+	do
+	{
+		char	*curname;
+		char	*parsedname;
+		char	*endp;
+		int		len;
+
+		curname = nextp;
+		while (*nextp && *nextp != ',' && !isspace(*nextp))
+			nextp++;
+		endp = nextp;
+		if (curname == nextp)
+			return false;
+
+		len = endp  - curname;
+		parsedname = pg_malloc(sizeof(char) * len + 1);
+		strncpy(parsedname, curname, len);
+
+		while (isspace(*nextp))
+			nextp++;	/* skip trailing whitespace */
+		if (*nextp == ',')
+		{
+			nextp++;
+			while (isspace(*nextp))
+				nextp++;		/* skip leading whitespace for next */
+			/* we expect another name, so done remains false */
+		}
+		else if (*nextp == '\0')
+			done = true;
+		else
+			return false;	/* invalid syntax */
+
+		*endp = '\0';
+
+		if (!(pg_strncasecmp(parsedname, "create_table", 12) == 0 ||
+			  pg_strncasecmp(parsedname, "load_data", 9) == 0 ||
+			  pg_strncasecmp(parsedname, "vacuum", 6) == 0 ||
+			  pg_strncasecmp(parsedname, "create_pkey", 12) == 0 ||
+			  pg_strncasecmp(parsedname, "create_fkey", 12) == 0))
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%s\"\n", parsedname);
+			return false;
+		}
+
+		/* FInished current name, add it to list */
+		simple_string_list_append(list, parsedname);
+
+		/* Loop back if we didn't reach end of string */
+	} while(!done);
+
+	return true;
+}
+
+/*
+ * Create tables, remove old tables if exist.
+ */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2699,8 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
-
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	int i;
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2687,6 +2737,95 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
+	{
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
+
+	fprintf(stderr, "set primary keys...\n");
+	for (i = 0; i < lengthof(DDLINDEXes); i++)
+	{
+		char		buffer[256];
+
+		strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
+
+		if (index_tablespace != NULL)
+		{
+			char	   *escape_tablespace;
+
+			escape_tablespace = PQescapeIdentifier(con, index_tablespace,
+												   strlen(index_tablespace));
+			snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
+					 " using index tablespace %s", escape_tablespace);
+			PQfreemem(escape_tablespace);
+		}
+
+		executeStatement(con, buffer);
+	}
+}
+
+/*
+ * Fill the standard table with some data.
+ */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2792,52 +2931,68 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
+/* invoke each custom initialization commands */
+static void
+custom_init(SimpleStringList custom_initialize_list)
+{
+	SimpleStringListCell *cell;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cell = custom_initialize_list.head; cell; cell = cell->next)
 	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
+		if (pg_strncasecmp(cell->val, "create_table", 12) == 0)
+			initCreateTables(con);
+		else if (pg_strncasecmp(cell->val, "load_data", 9) == 0)
+			initLoadData(con);
+		else if (pg_strncasecmp(cell->val, "vacuum", 6) == 0)
+			initVacuum(con);
+		else if (pg_strncasecmp(cell->val, "create_pkey", 11) == 0)
+			initCreatePKeys(con);
+		else if (pg_strncasecmp(cell->val, "create_fkey", 11) == 0)
+			initCreateFKeys(con);
+		else
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%s\"\n", cell->val);
+			fprintf(stderr, "possible commands are: \"create_table\", \"load_data\", \"vacuum\", \"create_pkeys\", \"create_fkyes\"\n");
+			PQfinish(con);
+			exit(1);
+		}
 	}
 
-	/*
-	 * create indexes
-	 */
-	fprintf(stderr, "set primary keys...\n");
-	for (i = 0; i < lengthof(DDLINDEXes); i++)
-	{
-		char		buffer[256];
+	fprintf(stderr, "done.\n");
+	PQfinish(con);
+}
 
-		strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
+/* create tables and setup data */
+static void
+init(bool is_no_vacuum)
+{
+	PGconn	   *con;
 
-		if (index_tablespace != NULL)
-		{
-			char	   *escape_tablespace;
+	if ((con = doConnect()) == NULL)
+		exit(1);
 
-			escape_tablespace = PQescapeIdentifier(con, index_tablespace,
-												   strlen(index_tablespace));
-			snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
-					 " using index tablespace %s", escape_tablespace);
-			PQfreemem(escape_tablespace);
-		}
+	/* create tables */
+	initCreateTables(con);
 
-		executeStatement(con, buffer);
-	}
+	/* load data */
+	initLoadData(con);
 
-	/*
-	 * create foreign keys
-	 */
+	/* vacuum */
+	if (!is_no_vacuum)
+		initVacuum(con);
+
+	/* create indexes */
+	initCreatePKeys(con);
+
+	/* create foreign keys */
 	if (foreign_keys)
-	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
-		{
-			executeStatement(con, DDLKEYs[i]);
-		}
-	}
+		initCreateFKeys(con);
 
 	fprintf(stderr, "done.\n");
 	PQfinish(con);
@@ -3623,6 +3778,7 @@ main(int argc, char **argv)
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
 		{"no-vacuum", no_argument, NULL, 'n'},
+		{"no-primary-keys", no_argument, NULL, 'I'},
 		{"port", required_argument, NULL, 'p'},
 		{"progress", required_argument, NULL, 'P'},
 		{"protocol", required_argument, NULL, 'M'},
@@ -3645,12 +3801,16 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"custom-initialize", required_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
+	int			is_initialize_suite = 0;
+	int			is_custom_initialize = 0;
 	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	SimpleStringList custom_initialize_list = {NULL, NULL};
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3719,6 +3879,7 @@ main(int argc, char **argv)
 		{
 			case 'i':
 				is_init_mode++;
+				is_initialize_suite++;
 				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
@@ -3991,6 +4152,20 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					char *rawstring = pg_strdup(optarg);
+
+					if (!parseCustomInitialize(rawstring, &custom_initialize_list))
+					{
+						fprintf(stderr, "invalid syntax of custom initialization \"%s\"\n",
+								optarg);
+						exit(1);
+					}
+					is_init_mode++;
+					is_custom_initialize++;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4052,7 +4227,16 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		if (is_initialize_suite && is_custom_initialize)
+		{
+			fprintf(stderr, "cannot initialize specific custom initialize mode and initialize mode at the same time\n");
+			exit(1);
+		}
+
+		if (is_custom_initialize)
+			custom_init(custom_initialize_list);
+		else
+			init(is_no_vacuum);
 		exit(0);
 	}
 	else
#21Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#20)
Re: pgbench: Skipping the creating primary keys after initialization

Attached patch

I'll look into it.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#22Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#20)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Mahahiko-san,

My 0.02€ about the patch & feature, which only reflect my point of view:

Please add a number to patches to avoid ambiguities. This was patch was
the second sent on the thread.

There is no need to have both custom_init & init functions. I'll suggest
to remove function "init", rename "custom_init" as "init", and make the
option default to what is appropriate so that it initialize the schema as
expected, and there is only one initialization mechanism.

I would suggest to make initialization sub options (no-vacuum,
foreign-key...) rely on updating the initialization operations instead of
being maintained separately. Currently "--no-vacuum --custom-init=vacuum"
would skip vacuum, which is quite debatable...

I'm not sure of the "custom-initialize" option name, but why not. I
suggest to remove "is_initialize_suite", and make custom-initialize
require -i as seems logical and upward compatible.

ISTM that there should be short names for the phases. Maybe using only one
letter could simplify the code significantly, dropping commas and list,
eg: "t" for "create_table", "d" for "load_data", "p" for "create_pkey",
"f" for "create_fkey", "v" for "vacuum".

I do not think that allowing a space in the list is a shell-wise idea.

I'm also wondering whether using a list is a good option, because it
implies a large parse function, list management and so on. With the one
letter version, it could be just a string to be scanned char by char for
operations.

Otherwise, at least allow short two letter codes (eg: ct ld pk fk va), in
order to avoid writing a very long thing on the command line, which is
quite a pain:

sh> pgbench --custom-initialize=create_table,load_data,primary_key,foreign_key,vacuum ...

vs

sh> pgbench -i -I tdpfv ...

Maybe there could be short-hands for usual setups, eg "default" for "tdpv"
or maybe "ct,ld,pk,va", "full" for "tdpfv" or maybe "ct,ld,pk,fk,va"...

Typo in doc "initailize" -> "initialize". Option values should be
presented in their logical execution order, i.e. put vacuum at the end.

Typo in help "initilize" -> "initialize". I would suggest to drop the space in the
option value in the presentation so that quotes are not needed.

Remove the "no-primary-keys" from the long option array as it has
disappeared. You might consider make "custom-initialize" take the 'I'
short option.

Ranting unrelated to this patch: the automatic aid type switching based on
scale is a bad idea (tm), because when trying to benchmark it means that
changing the scale also changes the schema, and you really do not need
that. ISTM that it should always use INT8.

--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#23Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#22)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Aug 8, 2017 at 10:50 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Mahahiko-san,

My 0.02€ about the patch & feature, which only reflect my point of view:

Thank you for reviewing the patch!

Please add a number to patches to avoid ambiguities. This was patch was the
second sent on the thread.

There is no need to have both custom_init & init functions. I'll suggest to
remove function "init", rename "custom_init" as "init", and make the option
default to what is appropriate so that it initialize the schema as
expected, and there is only one initialization mechanism.

I would suggest to make initialization sub options (no-vacuum,
foreign-key...) rely on updating the initialization operations instead of
being maintained separately. Currently "--no-vacuum --custom-init=vacuum"
would skip vacuum, which is quite debatable...

I'm not sure of the "custom-initialize" option name, but why not. I suggest
to remove "is_initialize_suite", and make custom-initialize require -i as
seems logical and upward compatible.

ISTM that there should be short names for the phases. Maybe using only one
letter could simplify the code significantly, dropping commas and list, eg:
"t" for "create_table", "d" for "load_data", "p" for "create_pkey", "f" for
"create_fkey", "v" for "vacuum".

I do not think that allowing a space in the list is a shell-wise idea.

I think we can use it like --custom-initialize="create_table, vacuum"
which is similar to what we specify a connection option to psql for
example. But it will be unnecessary if we have the one letter version.

I'm also wondering whether using a list is a good option, because it implies
a large parse function, list management and so on. With the one letter
version, it could be just a string to be scanned char by char for
operations.

I basically agree with the one letter version. But I'm concerned that
it'll confuse users if we have more initialization steps for the
pgbench initialization. If we add more various initialization steps it
makes pgbench command hard to read and the users might have to
remember these options.

Otherwise, at least allow short two letter codes (eg: ct ld pk fk va), in
order to avoid writing a very long thing on the command line, which is quite
a pain:

sh> pgbench
--custom-initialize=create_table,load_data,primary_key,foreign_key,vacuum
...

vs

sh> pgbench -i -I tdpfv ...

Maybe there could be short-hands for usual setups, eg "default" for "tdpv"
or maybe "ct,ld,pk,va", "full" for "tdpfv" or maybe "ct,ld,pk,fk,va"...

If --custom-initialize option requires for i option to be set,
"pgbench -i" means the initialization with full steps and "pgbench -i
--custom-initialize=..." means the initialization with custom
operation steps.

Typo in doc "initailize" -> "initialize". Option values should be presented
in their logical execution order, i.e. put vacuum at the end.

Typo in help "initilize" -> "initialize". I would suggest to drop the space
in the
option value in the presentation so that quotes are not needed.

Remove the "no-primary-keys" from the long option array as it has
disappeared. You might consider make "custom-initialize" take the 'I' short
option.

Ranting unrelated to this patch: the automatic aid type switching based on
scale is a bad idea (tm), because when trying to benchmark it means that
changing the scale also changes the schema, and you really do not need that.
ISTM that it should always use INT8.

Hmm, I think it's a valid point. Should we allow users to specify like
the above thing in the custom initialization feature as well?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#24Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#23)
Re: pgbench: Skipping the creating primary keys after initialization

Hello,

I think we can use it like --custom-initialize="create_table, vacuum"
which is similar to what we specify a connection option to psql for
example.

Even if it is allowed, do not advertise it. Or use space as a separator
and not comma. ISTM that with psql connections space is the higher level
separator, not an optional thing, and comma is used for lower level
splitting: "host=foo port=5432,5433 ..."

But it will be unnecessary if we have the one letter version.

Sure.

I'm also wondering whether using a list is a good option, because it implies
a large parse function, list management and so on. With the one letter
version, it could be just a string to be scanned char by char for
operations.

I basically agree with the one letter version. But I'm concerned that
it'll confuse users if we have more initialization steps for the
pgbench initialization. If we add more various initialization steps it
makes pgbench command hard to read and the users might have to
remember these options.

I think that if we get to the point where so many initialization steps
that people get confused, then adding long names will not be an issue:-)

In the mean time it only needs 5 values.

Maybe there could be short-hands for usual setups, eg "default" for "tdpv"
or maybe "ct,ld,pk,va", "full" for "tdpfv" or maybe "ct,ld,pk,fk,va"...

If --custom-initialize option requires for i option to be set,
"pgbench -i" means the initialization with full steps and "pgbench -i
--custom-initialize=..." means the initialization with custom
operation steps.

Sure. It does not preclude the default to have a name.

Remove the "no-primary-keys" from the long option array as it has
disappeared. You might consider make "custom-initialize" take the 'I' short
option.

Ranting unrelated to this patch: the automatic aid type switching based on
scale is a bad idea (tm), because when trying to benchmark it means that
changing the scale also changes the schema, and you really do not need that.
ISTM that it should always use INT8.

Hmm, I think it's a valid point. Should we allow users to specify like
the above thing in the custom initialization feature as well?

I would be in favor of having an option to do a tpc-b conforming schema
which would include that, but which would also change the default balance
type which is not large enough per spec. Maybe it could be "T".

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#25Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#24)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Aug 15, 2017 at 2:43 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello,

I think we can use it like --custom-initialize="create_table, vacuum"
which is similar to what we specify a connection option to psql for
example.

Even if it is allowed, do not advertise it. Or use space as a separator and
not comma. ISTM that with psql connections space is the higher level
separator, not an optional thing, and comma is used for lower level
splitting: "host=foo port=5432,5433 ..."

But it will be unnecessary if we have the one letter version.

Sure.

I'm also wondering whether using a list is a good option, because it
implies
a large parse function, list management and so on. With the one letter
version, it could be just a string to be scanned char by char for
operations.

I basically agree with the one letter version. But I'm concerned that
it'll confuse users if we have more initialization steps for the
pgbench initialization. If we add more various initialization steps it
makes pgbench command hard to read and the users might have to
remember these options.

I think that if we get to the point where so many initialization steps that
people get confused, then adding long names will not be an issue:-)

In the mean time it only needs 5 values.

Agreed.

Maybe there could be short-hands for usual setups, eg "default" for
"tdpv"
or maybe "ct,ld,pk,va", "full" for "tdpfv" or maybe "ct,ld,pk,fk,va"...

If --custom-initialize option requires for i option to be set,
"pgbench -i" means the initialization with full steps and "pgbench -i
--custom-initialize=..." means the initialization with custom
operation steps.

Sure. It does not preclude the default to have a name.

Remove the "no-primary-keys" from the long option array as it has
disappeared. You might consider make "custom-initialize" take the 'I'
short
option.

Ranting unrelated to this patch: the automatic aid type switching based
on
scale is a bad idea (tm), because when trying to benchmark it means that
changing the scale also changes the schema, and you really do not need
that.
ISTM that it should always use INT8.

Hmm, I think it's a valid point. Should we allow users to specify like
the above thing in the custom initialization feature as well?

I would be in favor of having an option to do a tpc-b conforming schema
which would include that, but which would also change the default balance
type which is not large enough per spec. Maybe it could be "T".

Yeah, once custom initialization patch get committed we can extend it.

Attached updated patch. I've incorporated the all comments from Fabien
to it, and changed it to single letter version.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v3.patchapplication/octet-stream; name=pgbench_custom_initialization_v3.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..add6b88 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -25,6 +25,12 @@
   </cmdsynopsis>
   <cmdsynopsis>
    <command>pgbench</command>
+   <arg choice="plain"><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></arg>
+   <arg rep="repeat"><replaceable>option</replaceable></arg>
+   <arg choice="opt"><replaceable>dbname</replaceable></arg>
+  </cmdsynopsis>
+  <cmdsynopsis>
+   <command>pgbench</command>
    <arg rep="repeat"><replaceable>option</replaceable></arg>
    <arg choice="opt"><replaceable>dbname</replaceable></arg>
   </cmdsynopsis>
@@ -177,6 +183,8 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
       <listitem>
        <para>
         Perform no vacuuming after initialization.
+        This option cannot be set with <option>--custom-initialize</option>
+        at the same time.
        </para>
       </listitem>
      </varlistentry>
@@ -211,10 +219,70 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Required to invoke initialization with custom steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>t</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>,
+            and <structname>pgbench_tellers</>, destroying any existing
+            tables of these names.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>d</literal></term>
+          <listitem>
+           <para>
+            Load data to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--foreign-keys</option></term>
       <listitem>
        <para>
         Create foreign key constraints between the standard tables.
+        This option cannot be set with <option>--custom-initialize</option>
+        at the same time.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..22d5ad9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -117,6 +117,11 @@ int			fillfactor = 100;
 int			foreign_keys = 0;
 
 /*
+ * no vacuum at all before testing?
+ */
+int			is_no_vacuum = 0;
+
+/*
  * use unlogged tables?
  */
 int			unlogged_tables = 0;
@@ -458,6 +463,11 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +494,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  --custom-initialize={t|d|p|f|v] [...]}\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2578,39 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static bool
+checkCustomCmds(char *initialize_cmds)
+{
+	char	*cmd;
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
+		{
+			case 't':
+			case 'd':
+			case 'p':
+			case 'f':
+			case 'v':
+				/* valid commands */
+				break;
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				fprintf(stderr, "possible commands are: \"t\", \"d\", \"p\", \"f\", \"v\"\n");
+				return false;
+			}
+		}
+	}
+	return true;
+}
+
+/*
+ * Create tables, remove old tables if exist.
+ */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2665,8 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
-
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
-
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
 
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	int i;
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2687,6 +2703,95 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
+	{
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
+
+	fprintf(stderr, "set primary keys...\n");
+	for (i = 0; i < lengthof(DDLINDEXes); i++)
+	{
+		char		buffer[256];
+
+		strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
+
+		if (index_tablespace != NULL)
+		{
+			char	   *escape_tablespace;
+
+			escape_tablespace = PQescapeIdentifier(con, index_tablespace,
+												   strlen(index_tablespace));
+			snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
+					 " using index tablespace %s", escape_tablespace);
+			PQfreemem(escape_tablespace);
+		}
+
+		executeStatement(con, buffer);
+	}
+}
+
+/*
+ * Fill the standard table with some data.
+ */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2792,50 +2897,45 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
-
-	/*
-	 * create indexes
-	 */
-	fprintf(stderr, "set primary keys...\n");
-	for (i = 0; i < lengthof(DDLINDEXes); i++)
-	{
-		char		buffer[256];
-
-		strlcpy(buffer, DDLINDEXes[i], sizeof(buffer));
-
-		if (index_tablespace != NULL)
-		{
-			char	   *escape_tablespace;
-
-			escape_tablespace = PQescapeIdentifier(con, index_tablespace,
-												   strlen(index_tablespace));
-			snprintf(buffer + strlen(buffer), sizeof(buffer) - strlen(buffer),
-					 " using index tablespace %s", escape_tablespace);
-			PQfreemem(escape_tablespace);
-		}
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds, bool is_custom)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
 
-		executeStatement(con, buffer);
-	}
+	if ((con = doConnect()) == NULL)
+		exit(1);
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'd':
+				initLoadData(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				if (is_custom || foreign_keys)
+					initCreateFKeys(con);
+				break;
+			case 'v':
+				if (is_custom || !is_no_vacuum)
+					initVacuum(con);
+				break;
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3641,12 +3741,14 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"custom-initialize", required_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	int			is_custom_init = 0;
+	char		*initialize_cmds = "tdpv";
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3944,8 +4046,11 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables || foreign_keys)
 					initialization_option_set = true;
+
+				if (foreign_keys)
+					initialize_cmds = "tdpvf";
 				break;
 			case 2:				/* tablespace */
 				initialization_option_set = true;
@@ -3982,6 +4087,18 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				initialize_cmds = pg_strdup(optarg);
+
+				if (!checkCustomCmds(initialize_cmds))
+				{
+					fprintf(stderr, "invalid syntax of custom initialization \"%s\"\n",
+							initialize_cmds);
+					exit(1);
+				}
+				is_custom_init++;
+				initialization_option_set = true;
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4178,19 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		if (is_custom_init && is_no_vacuum)
+		{
+			fprintf(stderr, "cannot specify no vacuum (-n) and custom initialization mode at the same time\n");
+			exit(1);
+		}
+
+		if (is_custom_init && foreign_keys)
+		{
+			fprintf(stderr, "cannot specify to create foreign keys (--foreign-keys) and custom initialization mode at the same time\n");
+			exit(1);
+		}
+
+		init(initialize_cmds, is_custom_init);
 		exit(0);
 	}
 	else
#26Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#25)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Yeah, once custom initialization patch get committed we can extend it.

Attached updated patch. I've incorporated the all comments from Fabien
to it, and changed it to single letter version.

Patch applies and works.

A few comments and questions about the code and documentation:

Why not allow -I as a short option for --custom-initialize?

I do not think that the --custom-initialize option needs to appear as a
specific synopsis in the documentation, as it is now a sub-option of -i.

checkCustomCmds: I would suggest to simplify test code with strchr
and to merge the two fprintf into one, something like:

if (strchr("tdpfv", *cmd) == NULL) {
fprintf(stderr,
"....\n"
"....\n",
...);
...

Moreover there is already an error message later if checkCustomCmds fails, I think
it could be expanded and the two-line one in the function dropped
entirely? It seems strange to have two level error messages for that.

Help message looks strange. I suggest something regexp-like:

--custom-initialize=[tdvpf]+

I would suggest to put the various init* functions in a more logical order:
first create table, then load data, etc.

In case 0: do not exchange unlogged_tables & foreign_keys gratuiously.

After checking the initial code, I understand that the previous default
was "tdvp", but you put "tdpv". I'm unsure whether vaccuum would do
something to the indexes and that would make more sense. In doubt, I
suggest to keep the previous default.

Maybe --foreign-keys should really do "tdvpf"?

I may be okay with disallowing --foreign-keys and --no-vacuum if --custom-init is used,
but then there is no need to test it again in init... I think that in any case 'f' and 'v'
should always trigger the corresponding initializations.

On the other hand, I think that it could be more pragmatic with these
options, i.e. --foreign-keys could just append 'f' to the current command
if not already there, and '--no-vacuum' could remove 'v' if there, on the
fly, so that nothing would be banned. This would require to always have a
malloced custom_init string. It would allow to remove the "foreign_keys"
global variable. "is_no_vacuum" is probably still needed for benchmarking.
This way there would be no constraints and "is_custom_init" could be
dropped as well.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#27Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#26)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 16, 2017 at 4:55 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Yeah, once custom initialization patch get committed we can extend it.

Attached updated patch. I've incorporated the all comments from Fabien
to it, and changed it to single letter version.

Patch applies and works.

A few comments and questions about the code and documentation:

Thank you for the comments!

Why not allow -I as a short option for --custom-initialize?

Other options for similar purpose such as --foreign-keys also have
only a long option. Since I think --custom-initialize option is in the
same context as other options I didn't add short option to it for now.
Because the options name is found by a prefix searching we can use a
short name --cu for now.

I do not think that the --custom-initialize option needs to appear as a
specific synopsis in the documentation, as it is now a sub-option of -i.

checkCustomCmds: I would suggest to simplify test code with strchr
and to merge the two fprintf into one, something like:

if (strchr("tdpfv", *cmd) == NULL) {
fprintf(stderr,
"....\n"
"....\n",
...);
...

Moreover there is already an error message later if checkCustomCmds fails, I
think
it could be expanded and the two-line one in the function dropped entirely?
It seems strange to have two level error messages for that.

Help message looks strange. I suggest something regexp-like:

--custom-initialize=[tdvpf]+

I would suggest to put the various init* functions in a more logical order:
first create table, then load data, etc.

In case 0: do not exchange unlogged_tables & foreign_keys gratuiously.

After checking the initial code, I understand that the previous default was
"tdvp", but you put "tdpv". I'm unsure whether vaccuum would do something to
the indexes and that would make more sense. In doubt, I suggest to keep the
previous default.

Maybe --foreign-keys should really do "tdvpf"?

I may be okay with disallowing --foreign-keys and --no-vacuum if
--custom-init is used,
but then there is no need to test it again in init... I think that in any
case 'f' and 'v'
should always trigger the corresponding initializations.

On the other hand, I think that it could be more pragmatic with these
options, i.e. --foreign-keys could just append 'f' to the current command if
not already there, and '--no-vacuum' could remove 'v' if there, on the fly,
so that nothing would be banned. This would require to always have a
malloced custom_init string. It would allow to remove the "foreign_keys"
global variable. "is_no_vacuum" is probably still needed for benchmarking.
This way there would be no constraints and "is_custom_init" could be dropped
as well.

I'm inclined to remove the restriction so that we can specify
--foreign-keys, --no-vacuum and --custom-initialize at the same time.
I think a list of char would be better here rather than a single
malloced string to remove particular initialization step easily.
Thought?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#28Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#27)
Re: pgbench: Skipping the creating primary keys after initialization

Why not allow -I as a short option for --custom-initialize?

Other options for similar purpose such as --foreign-keys also have
only a long option. Since I think --custom-initialize option is in the
same context as other options I didn't add short option to it for now.
Because the options name is found by a prefix searching we can use a
short name --cu for now.

Hmmm. I like short options:-)

I'm inclined to remove the restriction so that we can specify
--foreign-keys, --no-vacuum and --custom-initialize at the same time.

Ok. I favor that as well.

I think a list of char would be better here rather than a single
malloced string to remove particular initialization step easily.
Thought?

My thought is not to bother with a list of char.

To remove a char from a string, I suggest to allow ' ' to stand for
nothing and be skipped, so that substituting a letter by space would
simply remove an initialization phase.

For adding, realloc & append.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#29Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#28)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Aug 22, 2017 at 5:15 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Why not allow -I as a short option for --custom-initialize?

Other options for similar purpose such as --foreign-keys also have
only a long option. Since I think --custom-initialize option is in the
same context as other options I didn't add short option to it for now.
Because the options name is found by a prefix searching we can use a
short name --cu for now.

Hmmm. I like short options:-)

Okay, I added -I option for custom initialization :)

I'm inclined to remove the restriction so that we can specify
--foreign-keys, --no-vacuum and --custom-initialize at the same time.

Ok. I favor that as well.

If building foreign keys command is not specified in -I but
--foreign-keys options is specified (e.g. pgbench -i -I tpd
--foreign-keys) I think we can add 'f' to the end of the
initialization commands.

I think a list of char would be better here rather than a single
malloced string to remove particular initialization step easily.
Thought?

My thought is not to bother with a list of char.

To remove a char from a string, I suggest to allow ' ' to stand for nothing
and be skipped, so that substituting a letter by space would simply remove
an initialization phase.

I think we should not add/remove a command of initialization commands
during parsing pgbench options in order to not depend on its order.
Therefore, if -I, --foreign-keys and --no-vacuum are specified at the
same time, what we do is removing some 'v' commands if novacuum and
adding a 'f' command if foreignkey. Also we expect that the length of
initialization steps would not long. Using malloced string would less
the work. Ok, I changed the patch so.

Attached latest v4 patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v4.patchapplication/octet-stream; name=pgbench_custom_initialization_v4.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..6736d61 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -211,6 +211,65 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Required to invoke initialization with custom steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>t</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>,
+            and <structname>pgbench_tellers</>, destroying any existing
+            tables of these names.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>d</literal></term>
+          <listitem>
+           <para>
+            Load data to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--foreign-keys</option></term>
       <listitem>
        <para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..71aae6b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -117,6 +117,11 @@ int			fillfactor = 100;
 int			foreign_keys = 0;
 
 /*
+ * no vacuum at all before testing?
+ */
+int			is_no_vacuum = 0;
+
+/*
  * use unlogged tables?
  */
 int			unlogged_tables = 0;
@@ -458,6 +463,11 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +494,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  --custom-initialize=[tdvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2578,37 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static bool
+checkCustomCmds(char *commands)
+{
+	char	*cmd;
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
+		{
+			case 't':
+			case 'd':
+			case 'p':
+			case 'f':
+			case 'v':
+				/* valid commands */
+				break;
+			default:
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				fprintf(stderr, "possible commands are: \"t\", \"d\", \"p\", \"f\", \"v\"\n");
+				return false;
+		}
+	}
+	return true;
+}
+
+/*
+ * Create tables, remove old tables if exist.
+ */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2663,8 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
-
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	int i;
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2687,6 +2701,25 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/*
+ * Fill the standard table with some data.
+ */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2792,20 +2825,56 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
+		executeStatement(con, DDLKEYs[i]);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2895,45 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'd':
+				initLoadData(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3713,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3646,7 +3745,8 @@ main(int argc, char **argv)
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	int			is_custom_init = 0;
+	char		*initialize_cmds = "tdvp";
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3807,7 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,6 +3816,15 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				initialize_cmds = pg_strdup(optarg);
+
+				if (!checkCustomCmds(initialize_cmds))
+					exit(1);
+
+				is_custom_init++;
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
@@ -4061,7 +4170,47 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		if (initialize_cmds[0] == '\0')
+		{
+			fprintf(stderr, "initialize command is empty\n");
+			exit(1);
+		}
+
+		/*
+		 * If custom initialization option and other options related to initialization
+		 * are specified at the same time, we process them. Even if vacuum initialization
+		 * command is specified in custom initialization commands we remove it if --no-vacuum
+		 * is specified. Also, if building foreign keys command is not specified in custom
+		 * initialization commands but --foreign-key option is specified, building foreign
+		 * keys command is added to at end of the commands.
+		 */
+		if (is_custom_init && (is_no_vacuum || foreign_keys))
+		{
+			char *cmd;
+			int	n_cmds = 0;
+			bool fk_given = false;
+
+			for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+			{
+				if (*cmd == 'v' && is_no_vacuum)
+					*cmd = ' ';
+				if (*cmd== 'f')
+					fk_given = true;
+				n_cmds++;
+			}
+
+			if (!fk_given && foreign_keys)
+			{
+				/* Reallocate space for 'f' command */
+				n_cmds++;
+				initialize_cmds = (char *) pg_realloc(initialize_cmds,
+													  sizeof(char) * n_cmds);
+				initialize_cmds[n_cmds -1] = 'f';
+				initialize_cmds[n_cmds] = '\0';
+			}
+		}
+
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#30Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#29)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Attached latest v4 patch. Please review it.

Patch applies, compiles.

The messages/options do not seem to work properly:

sh> ./pgbench -i -I t
done.

Does not seem to have initialized the tables although it was requested...

sh> ./pgbench -i -I d
creating tables...
100000 of 100000 tuples (100%) done (elapsed 0.08 s, remaining 0.00 s)
done.

It seems that "d" triggered table creation... In fact it seems that the
work is done correctly, but the messages are not in the right place.

Also another issue:

sh> ./pgbench -i --foreign-keys
creating tables...
100000 of 100000 tuples (100%) done (elapsed 0.09 s, remaining 0.00 s)
vacuum...
set primary keys...
done.

Foreign keys do not seem to have been set... Please check that all really
work as expected.

About the documentation:

If a native English speaker could review the text, it would be great.

At least: "Required to invoke" -> "Require to invoke".

About the code:

is_no_vacuum should be a bool?

I'm really hesitating about the out of order processing of options. If the user writes

sh> pgbench -i --no-vacuum -I v
done.

Then does it make sense to ignore the last thing the user asked for? ISTM
that processing options in order and keeping the last resulting spec is
more natural. Appending contradictory options can happen easily when
scripting, and usual what is meant is the last one.

Again, as pointed out in the previous review, I do not like much
checkCustomCmds implementation: switch/case, fprintf and return on error
which will trigger another fprintf and error above... ISTM that you should
either take into account previous comments or explain why you disagree
with them, but not send the same code without addressing them in any way.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#31Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Fabien COELHO (#30)
Re: pgbench: Skipping the creating primary keys after initialization

Quick precision to my previous review.

sh> ./pgbench -i -I t
done.

There should be "creating tables..."

Does not seem to have initialized the tables although it was requested...

sh> ./pgbench -i -I d
creating tables...

Probably "filling tables..." would be more appropriate.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#32Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#30)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Sun, Aug 27, 2017 at 5:12 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Attached latest v4 patch. Please review it.

Thank you for reviewing this patch!

Patch applies, compiles.

The messages/options do not seem to work properly:

sh> ./pgbench -i -I t
done.

Fixed this so that it ouptut "creating tables..." as you pointed out.

Does not seem to have initialized the tables although it was requested...

sh> ./pgbench -i -I d
creating tables...
100000 of 100000 tuples (100%) done (elapsed 0.08 s, remaining 0.00 s)
done.

It seems that "d" triggered table creation... In fact it seems that the
work is done correctly, but the messages are not in the right place.

Fixed, but I just removed "creating tables..." from -I d command. I
think it's not good if we change the output messages by this patch.

Also another issue:

sh> ./pgbench -i --foreign-keys
creating tables...
100000 of 100000 tuples (100%) done (elapsed 0.09 s, remaining 0.00 s)
vacuum...
set primary keys...
done.

Foreign keys do not seem to have been set... Please check that all really
work as expected.

Fixed.

About the documentation:

If a native English speaker could review the text, it would be great.

At least: "Required to invoke" -> "Require to invoke".

Fixed.

About the code:

is_no_vacuum should be a bool?

We can change it but I think there is no difference actually. So
keeping it would be better.

I'm really hesitating about the out of order processing of options. If the
user writes

sh> pgbench -i --no-vacuum -I v
done.

Then does it make sense to ignore the last thing the user asked for? ISTM
that processing options in order and keeping the last resulting spec is more
natural. Appending contradictory options can happen easily when scripting,
and usual what is meant is the last one.

Agreed. I changed it so that it processes options in order and keeps
the last resulting spec.

Again, as pointed out in the previous review, I do not like much
checkCustomCmds implementation: switch/case, fprintf and return on error
which will trigger another fprintf and error above... ISTM that you should
either take into account previous comments or explain why you disagree with
them, but not send the same code without addressing them in any way.

Sorry, I didn't mean to ignore, I'd just missed the comment. Fixed it.

Attached latest patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v5.patchapplication/octet-stream; name=pgbench_custom_initialization_v5.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..b5dd954 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -153,7 +153,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
       <term><option>--initialize</option></term>
       <listitem>
        <para>
-        Required to invoke initialization mode.
+        Require to invoke initialization mode.
        </para>
       </listitem>
      </varlistentry>
@@ -211,6 +211,65 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Required to invoke initialization with custom steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>t</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>,
+            and <structname>pgbench_tellers</>, destroying any existing
+            tables of these names.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>d</literal></term>
+          <listitem>
+           <para>
+            Load data to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--foreign-keys</option></term>
       <listitem>
        <para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..7687378 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "tdvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+int			is_no_vacuum = 0;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,11 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +491,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  --custom-initialize=[tdvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2575,35 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static bool
+checkCustomCmds(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		return false;
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("tdvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"t\", \"d\", \"p\", \"f\", \"v\"\n");
+			return false;
+		}
+	}
+	return true;
+}
+
+/*
+ * Create tables, remove old tables if exist.
+ */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2658,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2687,6 +2698,25 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/*
+ * Fill the standard table with some data.
+ */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2713,8 +2743,6 @@ init(bool is_no_vacuum)
 	/*
 	 * fill the pgbench_accounts table with some data
 	 */
-	fprintf(stderr, "creating tables...\n");
-
 	executeStatement(con, "begin");
 	executeStatement(con, "truncate pgbench_accounts");
 
@@ -2792,20 +2820,56 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
+		executeStatement(con, DDLKEYs[i]);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2890,45 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'd':
+				initLoadData(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3708,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3633,7 +3727,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3641,12 +3734,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3801,10 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = (char *) pg_malloc(sizeof(char) * strlen(DEFAULT_INIT_COMMANDS) + 1);
+	strcpy(initialize_cmds, DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,12 +3813,29 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				initialize_cmds = pg_strdup(optarg);
+
+				if (!checkCustomCmds(initialize_cmds))
+					exit(1);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum++;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3944,7 +4058,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -3982,6 +4096,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4193,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#33Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#32)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Patch applies cleanly, compiles, works for me.

At least: "Required to invoke" -> "Require to invoke".

Fixed.

I meant the added one about -I, not the pre-existing one about -i.

About the code:

is_no_vacuum should be a bool?

We can change it but I think there is no difference actually. So
keeping it would be better.

I would like to insist a little bit: the name was declared as an int but
passed to init and used as a bool there before the patch. Conceptually
what is meant is really a bool, and I see no added value bar saving a few
strokes to have an int. ISTM that recent pgbench changes have started
turning old int-for-bool habits into using bool when bool is meant.

initialize_cmds initialization: rather use pg_strdup instead of
pg_malloc/strcpy?

-I: pg_free before pg_strdup to avoid a small memory leak?

I'm not sure I would have bothered with sizeof(char), but why not!

I'm still a little bit annoyed by checkCustomCmds: when unhappy, it shows
an error message and return false, which immediatly results in exit(1).
However the pattern elsewhere in pgbench is to show the error and exit
immediatly. I would suggest to simplify by void-ing the function and
exiting instead of returning false.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#34Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#33)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Mon, Aug 28, 2017 at 4:59 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Patch applies cleanly, compiles, works for me.

Thank you for reviewing!

At least: "Required to invoke" -> "Require to invoke".

Fixed.

I meant the added one about -I, not the pre-existing one about -i.

Fixed.

About the code:

is_no_vacuum should be a bool?

We can change it but I think there is no difference actually. So
keeping it would be better.

I would like to insist a little bit: the name was declared as an int but
passed to init and used as a bool there before the patch. Conceptually what
is meant is really a bool, and I see no added value bar saving a few strokes
to have an int. ISTM that recent pgbench changes have started turning old
int-for-bool habits into using bool when bool is meant.

Since is_no_vacuum is a existing one, if we follow the habit we should
change other similar variables as well: is_init_mode,
do_vacuum_accounts and debug. And I think we should change them in a
separated patch.

initialize_cmds initialization: rather use pg_strdup instead of
pg_malloc/strcpy?

Fixed.

-I: pg_free before pg_strdup to avoid a small memory leak?

Fixed.

I'm not sure I would have bothered with sizeof(char), but why not!

I'm still a little bit annoyed by checkCustomCmds: when unhappy, it shows an
error message and return false, which immediatly results in exit(1). However
the pattern elsewhere in pgbench is to show the error and exit immediatly. I
would suggest to simplify by void-ing the function and exiting instead of
returning false.

Agreed, fixed.

After more thought, I'm bit inclined to not have a short option for
--custom-initialize because this option will be used for not primary
cases. It would be better to save short options for future
enhancements of pgbench. Thought?

Attached latest patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v6.patchapplication/octet-stream; name=pgbench_custom_initialization_v6.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..40f7717 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -211,6 +211,65 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Require to invoke initialization with custom steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>t</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>,
+            and <structname>pgbench_tellers</>, destroying any existing
+            tables of these names.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>d</literal></term>
+          <listitem>
+           <para>
+            Load data to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>--foreign-keys</option></term>
       <listitem>
        <para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..f4228a9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "tdvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+int			is_no_vacuum = 0;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,11 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +491,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  --custom-initialize=[tdvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2575,34 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("tdvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"t\", \"d\", \"p\", \"f\", \"v\"\n");
+			exit(1);
+		}
+	}
+}
+
+/*
+ * Create tables, remove old tables if exist.
+ */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2657,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
-
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
+	int i;
 
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2687,6 +2697,25 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/*
+ * Fill the standard table with some data.
+ */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2713,8 +2742,6 @@ init(bool is_no_vacuum)
 	/*
 	 * fill the pgbench_accounts table with some data
 	 */
-	fprintf(stderr, "creating tables...\n");
-
 	executeStatement(con, "begin");
 	executeStatement(con, "truncate pgbench_accounts");
 
@@ -2792,20 +2819,56 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
+		executeStatement(con, DDLKEYs[i]);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2889,45 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'd':
+				initLoadData(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3707,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3633,7 +3726,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3641,12 +3733,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3800,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,12 +3811,31 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				if (initialize_cmds)
+					pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum++;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3944,7 +4058,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -3982,6 +4096,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4193,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#35Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#34)
Re: pgbench: Skipping the creating primary keys after initialization

Hello,

Patch applies and works.

I would like to insist a little bit: the name was declared as an int but
passed to init and used as a bool there before the patch. Conceptually what
is meant is really a bool, and I see no added value bar saving a few strokes
to have an int. ISTM that recent pgbench changes have started turning old
int-for-bool habits into using bool when bool is meant.

Since is_no_vacuum is a existing one, if we follow the habit we should
change other similar variables as well: is_init_mode, do_vacuum_accounts
and debug. And I think we should change them in a separated patch.

Hmmm. The existing "is_no_vacuum" variable is typed *both* as int (in
"main") and as bool (in "init"), called by main (yuk!). I see no reason to
choose the bad one for the global:-)

After more thought, I'm bit inclined to not have a short option for
--custom-initialize because this option will be used for not primary
cases. It would be better to save short options for future
enhancements of pgbench. Thought?

I like it as is, especially as now the associated value is a simple and
short string, I think that it makes sense to have a simple and short
option to trigger it. Moreover -I stands cleanly for "initialization", and
the capital stands for something a little special which it is. Its to good
to miss.

I think that the "-I" it should be added to the "--help" line, as it is
done with other short & long options.

Repeating "-I f" results in multiple foreign key constraints:

Foreign-key constraints:
"pgbench_tellers_bid_fkey" FOREIGN KEY (bid) REFERENCES pgbench_branches(bid)
"pgbench_tellers_bid_fkey1" FOREIGN KEY (bid) REFERENCES pgbench_branches(bid)
"pgbench_tellers_bid_fkey2" FOREIGN KEY (bid) REFERENCES pgbench_branches(bid)
"pgbench_tellers_bid_fkey3" FOREIGN KEY (bid) REFERENCES pgbench_branches(bid)

I wonder if this could be avoided easily? Maybe by setting the constraint
name explicitely so that the second one fails on the existing one, which
is fine, like for primary keys? Or adding a DROP CONSTRAINT IF EXISTS
before the CREATE CONSTRAINT, like for tables? Or doing nothing about it?
I would prefer the first option.

Maybe the initial cleanup (DROP TABLE) could be made an option added to
the default, so that cleaning up the database could be achieved with some
"pgbench -i -I c", instead of connecting and droping the tables one by one
which I have done quite a few times... What do you think?

Before it is definitely engraved, I'm thinking about the letters:

c - cleanup
t - create table
d - data
p - primary key
f - foreign key
v - vacuum

I think it is mostly okay, but it is the last time to think about it.
Using "d" for cleanup (drop) would mean finding another letter for filling
in data... maybe "g" for data generation? "c" may have been chosen for
"create table", but then would not be available for "cleanup". Thoughts?

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#36Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#35)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Aug 29, 2017 at 4:47 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello,

Patch applies and works.

I would like to insist a little bit: the name was declared as an int but
passed to init and used as a bool there before the patch. Conceptually
what
is meant is really a bool, and I see no added value bar saving a few
strokes
to have an int. ISTM that recent pgbench changes have started turning old
int-for-bool habits into using bool when bool is meant.

Since is_no_vacuum is a existing one, if we follow the habit we should
change other similar variables as well: is_init_mode, do_vacuum_accounts and
debug. And I think we should change them in a separated patch.

Hmmm. The existing "is_no_vacuum" variable is typed *both* as int (in
"main") and as bool (in "init"), called by main (yuk!). I see no reason to
choose the bad one for the global:-)

Yeah, I think this might be a good timing to re-consider int-for-bool
habits in pgbench. If we decided to change is_no_vacuum to bool I want
to change other similar variables as well.

After more thought, I'm bit inclined to not have a short option for
--custom-initialize because this option will be used for not primary
cases. It would be better to save short options for future
enhancements of pgbench. Thought?

I like it as is, especially as now the associated value is a simple and
short string, I think that it makes sense to have a simple and short option
to trigger it. Moreover -I stands cleanly for "initialization", and the
capital stands for something a little special which it is. Its to good to
miss.

I think that the "-I" it should be added to the "--help" line, as it is done
with other short & long options.

Okay, I'll leave it as of now. Maybe we can discuss later.

Repeating "-I f" results in multiple foreign key constraints:

Foreign-key constraints:
"pgbench_tellers_bid_fkey" FOREIGN KEY (bid) REFERENCES
pgbench_branches(bid)
"pgbench_tellers_bid_fkey1" FOREIGN KEY (bid) REFERENCES
pgbench_branches(bid)
"pgbench_tellers_bid_fkey2" FOREIGN KEY (bid) REFERENCES
pgbench_branches(bid)
"pgbench_tellers_bid_fkey3" FOREIGN KEY (bid) REFERENCES
pgbench_branches(bid)

I wonder if this could be avoided easily? Maybe by setting the constraint
name explicitely so that the second one fails on the existing one, which is
fine, like for primary keys? Or adding a DROP CONSTRAINT IF EXISTS before
the CREATE CONSTRAINT, like for tables? Or doing nothing about it? I would
prefer the first option.

Good point, I agree with first option.

Maybe the initial cleanup (DROP TABLE) could be made an option added to the
default, so that cleaning up the database could be achieved with some
"pgbench -i -I c", instead of connecting and droping the tables one by one
which I have done quite a few times... What do you think?

Yeah, I sometimes wanted that. Having the cleaning up tables option
would be good idea.

Before it is definitely engraved, I'm thinking about the letters:

c - cleanup
t - create table
d - data
p - primary key
f - foreign key
v - vacuum

I think it is mostly okay, but it is the last time to think about it. Using
"d" for cleanup (drop) would mean finding another letter for filling in
data... maybe "g" for data generation? "c" may have been chosen for "create
table", but then would not be available for "cleanup". Thoughts?

I'd say "g" for data generation would be better. Also, I'm inclined to
add a command for the unlogged tables. How about this?

c - cleanup
t - create table
u - create unlogged table
g - data generation
p - primary key
f - foreign key
v - vacuum

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#37Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#36)
Re: pgbench: Skipping the creating primary keys after initialization

Hello,

Hmmm. The existing "is_no_vacuum" variable is typed *both* as int (in
"main") and as bool (in "init"), called by main (yuk!). I see no reason to
choose the bad one for the global:-)

Yeah, I think this might be a good timing to re-consider int-for-bool
habits in pgbench. If we decided to change is_no_vacuum to bool I want
to change other similar variables as well.

Franckly I would be fine with that, but committers might get touchy about
"unrelated changes" in the patch... The "is_no_vacuum" is related to the
patch and is already a bool -- if you chose the "init" definition as a
reference -- so it is okay to bool it.

I think that the "-I" it should be added to the "--help" line, as it is done
with other short & long options.

Okay, I'll leave it as of now. Maybe we can discuss later.

Maybe we did not understand one another. I'm just suggesting to insert
-I in the help line, that is change:

" --custom-initialize=[...]+\n"

to

" -I, --custom-initialize=[...]+\n"

I'm not sure it deserves to be discussed in depth later:-)

I wonder if this could be avoided easily? Maybe by setting the constraint
name explicitely so that the second one fails on the existing one, which is
fine, like for primary keys? [...]

Good point, I agree with first option.

Ok.

Maybe the initial cleanup (DROP TABLE) could be made an option added to the
default, so that cleaning up the database could be achieved with some
"pgbench -i -I c", instead of connecting and droping the tables one by one
which I have done quite a few times... What do you think?

Yeah, I sometimes wanted that. Having the cleaning up tables option
would be good idea.

Ok.

I'd say "g" for data generation would be better. Also, I'm inclined to
add a command for the unlogged tables. How about this?

c - [c]leanup / or [d]rop tables
t - create table / [t]able creation or [c]reate table
u - create unlogged table
g - data generation / [g]enerate data
p - [p]rimary key
f - [f]oreign key
v - [v]acuum

I'm okay with that. I also put an alternative with d/c above, without
any preference from my part.

I'm not sure about "u", though. Unlogged, like tablespace, is an
orthogonal option: other table creation options (I intend to submit one
which conforms to the TPC-B standard, that is use an INT8 balance as INT4
is not wide enough per spec, and always use an INT8 aid) may be also
unlogged or tablespaced. So that would mean having two ways to trigger
them... thus I would avoid it and keep only --unlogged.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#38Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#37)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Aug 30, 2017 at 3:39 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello,

Hmmm. The existing "is_no_vacuum" variable is typed *both* as int (in
"main") and as bool (in "init"), called by main (yuk!). I see no reason
to
choose the bad one for the global:-)

Yeah, I think this might be a good timing to re-consider int-for-bool
habits in pgbench. If we decided to change is_no_vacuum to bool I want
to change other similar variables as well.

Franckly I would be fine with that, but committers might get touchy about
"unrelated changes" in the patch... The "is_no_vacuum" is related to the
patch and is already a bool -- if you chose the "init" definition as a
reference -- so it is okay to bool it.

Okay, I changed only is_no_vacuum in this patch and other similar
variables would be changed in another patch.

I think that the "-I" it should be added to the "--help" line, as it is
done
with other short & long options.

Okay, I'll leave it as of now. Maybe we can discuss later.

Maybe we did not understand one another. I'm just suggesting to insert
-I in the help line, that is change:

" --custom-initialize=[...]+\n"

to

" -I, --custom-initialize=[...]+\n"

Fixed.

I'm not sure it deserves to be discussed in depth later:-)

Sorry, I meant about having short option --custom-initialize.

I wonder if this could be avoided easily? Maybe by setting the constraint
name explicitely so that the second one fails on the existing one, which
is
fine, like for primary keys? [...]

Good point, I agree with first option.

Ok.

Maybe the initial cleanup (DROP TABLE) could be made an option added to
the
default, so that cleaning up the database could be achieved with some
"pgbench -i -I c", instead of connecting and droping the tables one by
one
which I have done quite a few times... What do you think?

Yeah, I sometimes wanted that. Having the cleaning up tables option
would be good idea.

Ok.

I'd say "g" for data generation would be better. Also, I'm inclined to
add a command for the unlogged tables. How about this?

c - [c]leanup / or [d]rop tables
t - create table / [t]able creation or [c]reate table
u - create unlogged table
g - data generation / [g]enerate data
p - [p]rimary key
f - [f]oreign key
v - [v]acuum

I'm okay with that. I also put an alternative with d/c above, without any
preference from my part.

Personally I prefer "t" for table creation because "c" for create is a
generic word. We might want to have another initialization command
that creates something.

I'm not sure about "u", though. Unlogged, like tablespace, is an orthogonal
option: other table creation options (I intend to submit one which conforms
to the TPC-B standard, that is use an INT8 balance as INT4 is not wide
enough per spec, and always use an INT8 aid) may be also unlogged or
tablespaced. So that would mean having two ways to trigger them... thus I
would avoid it and keep only --unlogged.

Yeah, I think I had misunderstood it. -I option is for specifying some
particular initialization steps. So we don't need to have a command as
a option for other initializatoin commands.

Attached latest patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v7.patchapplication/octet-stream; name=pgbench_custom_initialization_v7.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..c742562 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,74 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Require to invoke initialization with custom steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization commands for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal></term>
+          <listitem>
+           <para>
+            Destroying any existing tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal></term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..378aa07 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +492,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  -I, --custom-initialize=[ctgvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2576,51 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"t\", \"d\", \"p\", \"f\", \"v\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2675,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
-
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
+	int i;
 
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2659,10 +2687,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2687,6 +2711,23 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2713,8 +2754,6 @@ init(bool is_no_vacuum)
 	/*
 	 * fill the pgbench_accounts table with some data
 	 */
-	fprintf(stderr, "creating tables...\n");
-
 	executeStatement(con, "begin");
 	executeStatement(con, "truncate pgbench_accounts");
 
@@ -2792,20 +2831,56 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
+		executeStatement(con, DDLKEYs[i]);
 	}
+}
+
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2901,48 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3722,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3633,7 +3741,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3641,12 +3748,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3815,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,12 +3826,31 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				if (initialize_cmds)
+					pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3944,7 +4073,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -3982,6 +4111,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4208,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#39Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#38)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

[...] Personally I prefer "t" for table creation because "c" for create
is a generic word. We might want to have another initialization command
that creates something.

Ok, good point.

About the patch: applies, compiles, works for me. A few minor comments.

While re-reading the documentation, I think that it should be "Set custom
initialization steps". It could be "Require ..." when -I implied -i, but
since -i is still required the sentence does not seem to apply as such.

"Destroying any existing tables: ..." -> "Destroy existing pgbench tables:
...".

I would suggest to add short expanded explanations in the term definition,
next to the triggering letter, to underline the mnemonic. Something like:

c (cleanup)
t (table creation)
g (generate data)
v (vacuum)
p (primary key)
f (foreign key)

Also update the error message in checkCustomCommands to "ctgvpf".

Cleanup should have a message when it is executed. I suggest "cleaning
up...".

Maybe add a comment in front of the array tables to say that the order is
important, something like "tables in reverse foreign key dependencies
order"?

case 'I': ISTM that initialize_cmds is necessarily already allocated, thus
I would not bother to test before pg_free.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#40Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#39)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Thu, Aug 31, 2017 at 4:35 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

[...] Personally I prefer "t" for table creation because "c" for create is
a generic word. We might want to have another initialization command that
creates something.

Ok, good point.

About the patch: applies, compiles, works for me. A few minor comments.

Thank you for dedicated reviewing this patch!

While re-reading the documentation, I think that it should be "Set custom
initialization steps". It could be "Require ..." when -I implied -i, but
since -i is still required the sentence does not seem to apply as such.

"Destroying any existing tables: ..." -> "Destroy existing pgbench tables:
...".

Fixed.

I would suggest to add short expanded explanations in the term definition,
next to the triggering letter, to underline the mnemonic. Something like:

c (cleanup)
t (table creation)
g (generate data)
v (vacuum)
p (primary key)
f (foreign key)

Nice idea, agreed.

Also update the error message in checkCustomCommands to "ctgvpf".

Fixed.

Cleanup should have a message when it is executed. I suggest "cleaning
up...".

Fixed.

Maybe add a comment in front of the array tables to say that the order is
important, something like "tables in reverse foreign key dependencies
order"?

Fixed.

case 'I': ISTM that initialize_cmds is necessarily already allocated, thus I
would not bother to test before pg_free.

Agreed, fixed.

Attached latest patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v8.patchapplication/octet-stream; name=pgbench_custom_initialization_v8.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..fbef61c 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,74 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c (cleanup)</literal></term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t (table creation)</literal></term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g (generate data)</literal></term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v (vacuum)</literal></term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p (primary key)</literal></term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f (foreign key)</literal></term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..a213bd9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +492,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  -I, --custom-initialize=[ctgvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2576,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
 static void
-init(bool is_no_vacuum)
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
+static void
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2679,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
-
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
+	int i;
 
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2659,10 +2691,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2687,6 +2715,23 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
@@ -2713,8 +2758,6 @@ init(bool is_no_vacuum)
 	/*
 	 * fill the pgbench_accounts table with some data
 	 */
-	fprintf(stderr, "creating tables...\n");
-
 	executeStatement(con, "begin");
 	executeStatement(con, "truncate pgbench_accounts");
 
@@ -2792,20 +2835,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2882,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
+	{
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3726,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3633,7 +3745,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3641,12 +3752,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3819,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,12 +3830,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3944,7 +4076,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -3982,6 +4114,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4211,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#41Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#40)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Patch applies and compiles.

One bug found, and some minor points again. Sorry for this hopefully last
iteration... I'm kind of an iterative person...

I've generated the doc to look a it.

Short option "-I" does not use a "=", it should be "-I
custom_init_commands".

Also maybe it would look nicer and clearer if the short mnemonic was
outside the literal, that is with:

<term><literal>c</> (cleanup)</>

instead of:

<term><literal>c (cleanup)</></>

But this is debatable. Do it the way you think is best.

Command "g" does not work after "f", something I had not tested before:

./pgbench -i -I ctvpfg
cleaning up...
creating tables...
vacuum...
set primary keys...
set foreign keys...
ERROR: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "pgbench_history" references "pgbench_accounts".
HINT: Truncate table "pgbench_history" at the same time, or use TRUNCATE ... CASCADE.

I think it should work. It probably just mean to TRUNCATE all tables as
one command, or add the suggested CASCADE. I would favor the first option.

I'm wondering whether this truncation should be yet another available
command? Hmmm... maybe not.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#42Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#41)
Re: pgbench: Skipping the creating primary keys after initialization

On Fri, Sep 1, 2017 at 4:42 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Patch applies and compiles.

One bug found, and some minor points again. Sorry for this hopefully last
iteration... I'm kind of an iterative person...

I've generated the doc to look a it.

Short option "-I" does not use a "=", it should be "-I
custom_init_commands".

Also maybe it would look nicer and clearer if the short mnemonic was outside
the literal, that is with:

<term><literal>c</> (cleanup)</>

instead of:

<term><literal>c (cleanup)</></>

But this is debatable. Do it the way you think is best.

Command "g" does not work after "f", something I had not tested before:

./pgbench -i -I ctvpfg
cleaning up...
creating tables...
vacuum...
set primary keys...
set foreign keys...
ERROR: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "pgbench_history" references "pgbench_accounts".
HINT: Truncate table "pgbench_history" at the same time, or use TRUNCATE
... CASCADE.

I think it should work. It probably just mean to TRUNCATE all tables as one
command, or add the suggested CASCADE. I would favor the first option.

I'm wondering whether this truncation should be yet another available
command? Hmmm... maybe not.

Currently TRUNCATE pgbench_accounts command is executed within a
transaction started immediately before it. If we move it out of the
transaction, the table data will be truncated even if the copying data
failed. Maybe we can do TRUNCATE pgbench_accounts, pgbench_history
instead. Thought?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#43Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#42)
Re: pgbench: Skipping the creating primary keys after initialization

I'm wondering whether this truncation should be yet another available
command? Hmmm... maybe not.

Currently TRUNCATE pgbench_accounts command is executed within a
transaction started immediately before it. If we move it out of the
transaction, the table data will be truncated even if the copying data
failed. Maybe we can do TRUNCATE pgbench_accounts, pgbench_history
instead. Thought?

Keep the truncate in the transaction, and truncate both (or all?) tables
together.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#44Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#43)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Fri, Sep 1, 2017 at 11:29 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

I'm wondering whether this truncation should be yet another available
command? Hmmm... maybe not.

Currently TRUNCATE pgbench_accounts command is executed within a
transaction started immediately before it. If we move it out of the
transaction, the table data will be truncated even if the copying data
failed. Maybe we can do TRUNCATE pgbench_accounts, pgbench_history
instead. Thought?

Keep the truncate in the transaction, and truncate both (or all?) tables
together.

Attached latest patch incorporated the comments I got so far. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v9.patchapplication/octet-stream; name=pgbench_custom_initialization_v9.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..02ebbc5 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,74 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. Each command is invoked
+        in the specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..a60a9c4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +492,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  -I, --custom-initialize=[ctgvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2566,9 +2576,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2623,34 +2679,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
-
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
+	int i;
 
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2659,10 +2691,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2687,9 +2715,28 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
+
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
 
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_branches, pgbench_tellers");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2713,10 +2760,10 @@ init(bool is_no_vacuum)
 	/*
 	 * fill the pgbench_accounts table with some data
 	 */
-	fprintf(stderr, "creating tables...\n");
-
 	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
+
+	/* to avoid foreign key constraint violation */
+	executeStatement(con, "truncate pgbench_accounts, pgbench_history");
 
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
@@ -2792,20 +2839,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2826,16 +2886,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
+	{
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3615,6 +3730,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3633,7 +3749,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3641,12 +3756,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3707,7 +3823,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3716,12 +3834,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3944,7 +4080,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -3982,6 +4118,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4061,7 +4215,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#45Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#44)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Currently TRUNCATE pgbench_accounts command is executed within a
transaction started immediately before it. If we move it out of the
transaction, the table data will be truncated even if the copying data
failed. Maybe we can do TRUNCATE pgbench_accounts, pgbench_history
instead. Thought?

Keep the truncate in the transaction, and truncate both (or all?) tables
together.

Attached latest patch incorporated the comments I got so far. Please review it.

"g" does not work for me yet when after "f", only the message is slightly
different, it chokes on another dependency, branches instead of accounts.

sh> ./pgbench -i -I ctpfg
cleaning up...
creating tables...
set primary keys...
set foreign keys...
ERROR: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "pgbench_history" references "pgbench_branches".
HINT: Truncate table "pgbench_history" at the same time, or use TRUNCATE ... CASCADE.

I think that the whole data generation should be in *one* transaction
which starts with "truncate pgbench_history, pgbench_branches,
pgbench_tellers, pgbench_accounts;"

In passing, I think that the documentation should tell explicitely what
the default value is (aka "ctgvp").

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#46Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#45)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Sep 5, 2017 at 2:33 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Currently TRUNCATE pgbench_accounts command is executed within a
transaction started immediately before it. If we move it out of the
transaction, the table data will be truncated even if the copying data
failed. Maybe we can do TRUNCATE pgbench_accounts, pgbench_history
instead. Thought?

Keep the truncate in the transaction, and truncate both (or all?) tables
together.

Attached latest patch incorporated the comments I got so far. Please
review it.

"g" does not work for me yet when after "f", only the message is slightly
different, it chokes on another dependency, branches instead of accounts.

sh> ./pgbench -i -I ctpfg
cleaning up...
creating tables...
set primary keys...
set foreign keys...
ERROR: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "pgbench_history" references "pgbench_branches".
HINT: Truncate table "pgbench_history" at the same time, or use TRUNCATE
... CASCADE.

Sorry, I'd missed something.

I think that the whole data generation should be in *one* transaction which
starts with "truncate pgbench_history, pgbench_branches, pgbench_tellers,
pgbench_accounts;"

Agreed, and fixed.

In passing, I think that the documentation should tell explicitely what the
default value is (aka "ctgvp").

Agreed.

Attached the latest patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v10.patchapplication/octet-stream; name=pgbench_custom_initialization_v10.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..558e25c 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>.Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..74e12a4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +492,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  -I, --custom-initialize=[ctgvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#47Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#46)
Re: pgbench: Skipping the creating primary keys after initialization

Attached the latest patch. Please review it.

Patch applies and compiles cleanly.

Three very minor points:

Typo added in the documentation: ".Each" -> ". Each".

In "case 8:" there is a very long line which lacks a newline before
pg_realloc second argument.

I think that the check should silently accept spaces as they are ignored
by init later, i.e.:

sh> ./pgbench -i -I "ctg vpf"
invalid custom initialization script command " "
possible commands are: "c", "t", "g", "v", "p", "f"

Could just work...

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#48Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#47)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Sep 5, 2017 at 4:06 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Attached the latest patch. Please review it.

Patch applies and compiles cleanly.

Three very minor points:

Typo added in the documentation: ".Each" -> ". Each".

In "case 8:" there is a very long line which lacks a newline before
pg_realloc second argument.

Sorry, I don't follow that. You meant I should add a newline before
pg_realloc()? That is,

+                        initialize_cmds =
+                            (char *) pg_realloc(initialize_cmds,
+                                                sizeof(char) * n_cmds + 1);

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#49Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#48)
Re: pgbench: Skipping the creating primary keys after initialization

Sorry, I don't follow that. You meant I should add a newline before
pg_realloc()? That is,

+                        initialize_cmds =
+                            (char *) pg_realloc(initialize_cmds,
+                                                sizeof(char) * n_cmds + 1);

Yes. Or maybe my terminal was doing tricks, because I had the impression
that both argument where on the same line with many tabs in between, but
maybe I just misinterpreted the diff file. My apology if it is the case.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#50Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#49)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Sep 6, 2017 at 12:11 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Sorry, I don't follow that. You meant I should add a newline before
pg_realloc()? That is,

+                        initialize_cmds =
+                            (char *) pg_realloc(initialize_cmds,
+                                                sizeof(char) * n_cmds +
1);

Yes. Or maybe my terminal was doing tricks, because I had the impression
that both argument where on the same line with many tabs in between, but
maybe I just misinterpreted the diff file. My apology if it is the case.

I understood. It looks ugly in the patch but can be applied properly
by git apply command. Attached latest patch incorporated the comments
I got so far.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v11.patchtext/x-patch; charset=US-ASCII; name=pgbench_custom_initialization_v11.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..48e3581 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..f6b0452 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -484,6 +492,8 @@ usage(void)
 		   "                           create indexes in the specified tablespace\n"
 		   "  --tablespace=TABLESPACE  create tables in the specified tablespace\n"
 		   "  --unlogged-tables        create tables as unlogged tables\n"
+		   "  -I, --custom-initialize=[ctgvpf]+\n"
+		   "                           initialize with custom initialization commands\n"
 		   "\nOptions to select what to run:\n"
 		   "  -b, --builtin=NAME[@W]   add builtin script NAME weighted at W (default: 1)\n"
 		   "                           (use \"-b list\" to list available scripts)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#51Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#50)
Re: pgbench: Skipping the creating primary keys after initialization

Applies, compiles, works for me.

Very very minor comments that I should have noticed before, sorry for this
additional round trip.

In the help line, move -I just after -i, to put short options in
alphabetical and decreasing importance order. On this line, also add the
information about the default, something like:

-i, --ini... ....
-I, --custom....=[...]+ (default "ctgvp")
...

When/if the pgbench tap test patch get through, the feature should be
tested there as well. No action needed now.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#52Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#51)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Sep 6, 2017 at 4:01 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Applies, compiles, works for me.

Very very minor comments that I should have noticed before, sorry for this
additional round trip.

Thank you for the dedicated review!

In the help line, move -I just after -i, to put short options in
alphabetical and decreasing importance order. On this line, also add the
information about the default, something like:

-i, --ini... ....
-I, --custom....=[...]+ (default "ctgvp")
...

Agreed. Attached latest patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v12.patchapplication/octet-stream; name=pgbench_custom_initialization_v12.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..48e3581 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..f98968f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -475,6 +483,8 @@ usage(void)
 		   "  %s [OPTION]... [DBNAME]\n"
 		   "\nInitialization options:\n"
 		   "  -i, --initialize         invokes initialization mode\n"
+		   "  -I, --custom-initialize=[ctgvpf]+ (default \"ctgvp\")\n"
+		   "                           initialize with custom initialization commands\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
#53Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#52)
Re: pgbench: Skipping the creating primary keys after initialization

Very very minor comments that I should have noticed before, sorry for this
additional round trip.

Thank you for the dedicated review!

I'm someone at times pigheaded, I think in the good sense if it is
possible, and I like to finish what I start:-)

Patch applies, compiles, works, everything is fine from my point of view.

I switched it to "Ready for Committer".

Again, if the pgbench tap test patch get through, it should be tap tested.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#54Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#53)
Re: pgbench: Skipping the creating primary keys after initialization

On Thu, Sep 7, 2017 at 4:15 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Very very minor comments that I should have noticed before, sorry for
this
additional round trip.

Thank you for the dedicated review!

I'm someone at times pigheaded, I think in the good sense if it is possible,
and I like to finish what I start:-)

Patch applies, compiles, works, everything is fine from my point of view.

I switched it to "Ready for Committer".

Thanks.

Again, if the pgbench tap test patch get through, it should be tap tested.

Thank you for the remainder, I'll add tap tests once the patch got committed.

--
Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#55Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Masahiko Sawada (#54)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Fri, Sep 8, 2017 at 9:52 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Thu, Sep 7, 2017 at 4:15 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Very very minor comments that I should have noticed before, sorry for
this
additional round trip.

Thank you for the dedicated review!

I'm someone at times pigheaded, I think in the good sense if it is possible,
and I like to finish what I start:-)

Patch applies, compiles, works, everything is fine from my point of view.

I switched it to "Ready for Committer".

Thanks.

Again, if the pgbench tap test patch get through, it should be tap tested.

Thank you for the remainder, I'll add tap tests once the patch got committed.

Attached the latest version patch incorporated the tap tests.
Please review it.

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v13.patchapplication/octet-stream; name=pgbench_custom_initialization_v13.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..48e3581 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..f98968f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -475,6 +483,8 @@ usage(void)
 		   "  %s [OPTION]... [DBNAME]\n"
 		   "\nInitialization options:\n"
 		   "  -i, --initialize         invokes initialization mode\n"
+		   "  -I, --custom-initialize=[ctgvpf]+ (default \"ctgvp\")\n"
+		   "                           initialize with custom initialization commands\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3609b9b..46bdcd6 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -84,13 +84,36 @@ pgbench(
 '--initialize --scale=1 --unlogged-tables --fillfactor=98 --foreign-keys --quiet --tablespace=pg_default --index-tablespace=pg_default',
 	0,
 	[qr{^$}i],
-	[   qr{creating tables},
+	[   qr{cleaning up},
+		qr{creating tables},
 		qr{vacuum},
 		qr{set primary keys},
 		qr{set foreign keys},
 		qr{done\.} ],
 	'pgbench scale 1 initialization');
 
+# Custom initialization option, including a space
+pgbench(
+	'--initialize --custom-initialize=ctpvgvf', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{set primary keys},
+		qr{vacuum},
+		qr{vacuum},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization script');
+
+# Custom initialization with other initialization options
+pgbench(
+	'--initialize --custom-initialize=ctpvgvv --no-vacuum --foreign-keys', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{set primary keys},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization with other initialization options');
+
 # Run all builtin scripts, for a few transactions each
 pgbench(
 	'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index d6b3d4f..b0c7f63 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -73,6 +73,9 @@ my @options = (
 	[ 'ambiguous builtin', '-b s', [qr{ambiguous}] ],
 	[   '--progress-timestamp => --progress', '--progress-timestamp',
 		[qr{allowed only under}] ],
+	[ 'custom init without init option', '-I ctg', [qr{cannot be used in benchmarking mode}] ],
+	[ 'invalid custom init command', '-i -I cta',
+		[qr{invalid custom initialization script command}, qr{possible commands are} ] ],
 
 	# loging sub-options
 	[   'sampling => log', '--sampling-rate=0.01',
#56Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#55)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Attached the latest version patch incorporated the tap tests.
Please review it.

Patch applies, compilation & make check ok.

Tests are simple and provide good coverage of new functionalities.

I would suggest to add '--unlogged-tables' so speedup the tests a little.

Comment: "# Custom initialization option, including a space"... ISTM that
there is no space. Space is tested in the next test because of the v's and
the --no-vacuum which turned them into space, which is enough.

Regex are just check for the whole output, so putting twice "qr{vacuum}"
does not check that vacuum appears twice, it checks twice that vacuum
appears once. I do not think that it is worth trying to check for the v
repetition, so I suggest to remove one from the first test. Repetition of
' ' is checked with the second test.

Maybe you could check that the data generation message is there.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#57Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#56)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Tue, Sep 19, 2017 at 12:41 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Attached the latest version patch incorporated the tap tests.
Please review it.

Patch applies, compilation & make check ok.

Tests are simple and provide good coverage of new functionalities.

I would suggest to add '--unlogged-tables' so speedup the tests a little.

Good idea, added.

Comment: "# Custom initialization option, including a space"... ISTM that
there is no space. Space is tested in the next test because of the v's and
the --no-vacuum which turned them into space, which is enough.

You're right, I removed it.

Regex are just check for the whole output, so putting twice "qr{vacuum}"
does not check that vacuum appears twice, it checks twice that vacuum
appears once. I do not think that it is worth trying to check for the v
repetition, so I suggest to remove one from the first test. Repetition of '
' is checked with the second test.

Agreed.

Maybe you could check that the data generation message is there.

Added the check.

Attached the latest patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v14.patchapplication/octet-stream; name=pgbench_custom_initialization_v14.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..48e3581 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..f98968f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -475,6 +483,8 @@ usage(void)
 		   "  %s [OPTION]... [DBNAME]\n"
 		   "\nInitialization options:\n"
 		   "  -i, --initialize         invokes initialization mode\n"
+		   "  -I, --custom-initialize=[ctgvpf]+ (default \"ctgvp\")\n"
+		   "                           initialize with custom initialization commands\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3609b9b..07f8882 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -84,13 +84,37 @@ pgbench(
 '--initialize --scale=1 --unlogged-tables --fillfactor=98 --foreign-keys --quiet --tablespace=pg_default --index-tablespace=pg_default',
 	0,
 	[qr{^$}i],
-	[   qr{creating tables},
+	[   qr{cleaning up},
+		qr{creating tables},
 		qr{vacuum},
 		qr{set primary keys},
 		qr{set foreign keys},
 		qr{done\.} ],
 	'pgbench scale 1 initialization');
 
+# Custom initialization option
+pgbench(
+	'--initialize --custom-initialize=ctpvgf --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{.* of .* tuples (.*) done},
+		qr{set primary keys},
+		qr{vacuum},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization script');
+
+# Custom initialization with other initialization options
+pgbench(
+	'--initialize --custom-initialize=ctpvgvv --no-vacuum --foreign-keys --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{set primary keys},
+		qr{.* of .* tuples (.*) done},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization with other initialization options');
+
 # Run all builtin scripts, for a few transactions each
 pgbench(
 	'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index d6b3d4f..b0c7f63 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -73,6 +73,9 @@ my @options = (
 	[ 'ambiguous builtin', '-b s', [qr{ambiguous}] ],
 	[   '--progress-timestamp => --progress', '--progress-timestamp',
 		[qr{allowed only under}] ],
+	[ 'custom init without init option', '-I ctg', [qr{cannot be used in benchmarking mode}] ],
+	[ 'invalid custom init command', '-i -I cta',
+		[qr{invalid custom initialization script command}, qr{possible commands are} ] ],
 
 	# loging sub-options
 	[   'sampling => log', '--sampling-rate=0.01',
#58Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#57)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

v14 applies, compiles and works. TAP tests provide good coverage.

ISTM that you probably intended "\(.*\)" (actual parenthesis) instead of
"(.*)" (memorization) in the data generation message check.

Otherwise all is well for me.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#59Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#58)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Sep 20, 2017 at 3:26 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

v14 applies, compiles and works. TAP tests provide good coverage.

ISTM that you probably intended "\(.*\)" (actual parenthesis) instead of
"(.*)" (memorization) in the data generation message check.

Thank you, fixed it.

Otherwise all is well for me.

Attached the updated version patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v15.patchapplication/octet-stream; name=pgbench_custom_initialization_v15.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..48e3581 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</>,
+            <structname>pgbench_branches</>, <structname>pgbench_history</>
+            and <structname>pgbench_tellers</>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..f98968f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -475,6 +483,8 @@ usage(void)
 		   "  %s [OPTION]... [DBNAME]\n"
 		   "\nInitialization options:\n"
 		   "  -i, --initialize         invokes initialization mode\n"
+		   "  -I, --custom-initialize=[ctgvpf]+ (default \"ctgvp\")\n"
+		   "                           initialize with custom initialization commands\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
@@ -2605,9 +2615,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2662,34 +2718,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2698,10 +2730,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2726,9 +2754,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2747,16 +2797,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2831,20 +2871,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2865,16 +2918,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3648,6 +3756,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3666,7 +3775,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3674,12 +3782,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3740,7 +3849,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3749,12 +3860,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3977,7 +4106,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4015,6 +4144,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4094,7 +4241,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3609b9b..6da8b2a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -84,13 +84,37 @@ pgbench(
 '--initialize --scale=1 --unlogged-tables --fillfactor=98 --foreign-keys --quiet --tablespace=pg_default --index-tablespace=pg_default',
 	0,
 	[qr{^$}i],
-	[   qr{creating tables},
+	[   qr{cleaning up},
+		qr{creating tables},
 		qr{vacuum},
 		qr{set primary keys},
 		qr{set foreign keys},
 		qr{done\.} ],
 	'pgbench scale 1 initialization');
 
+# Custom initialization option
+pgbench(
+	'--initialize --custom-initialize=ctpvgf --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{.* of .* tuples \(.*\) done},
+		qr{set primary keys},
+		qr{vacuum},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization script');
+
+# Custom initialization with other initialization options
+pgbench(
+	'--initialize --custom-initialize=ctpvgvv --no-vacuum --foreign-keys --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{set primary keys},
+		qr{.* of .* tuples \(.*\) done},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization with other initialization options');
+
 # Run all builtin scripts, for a few transactions each
 pgbench(
 	'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index d6b3d4f..b0c7f63 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -73,6 +73,9 @@ my @options = (
 	[ 'ambiguous builtin', '-b s', [qr{ambiguous}] ],
 	[   '--progress-timestamp => --progress', '--progress-timestamp',
 		[qr{allowed only under}] ],
+	[ 'custom init without init option', '-I ctg', [qr{cannot be used in benchmarking mode}] ],
+	[ 'invalid custom init command', '-i -I cta',
+		[qr{invalid custom initialization script command}, qr{possible commands are} ] ],
 
 	# loging sub-options
 	[   'sampling => log', '--sampling-rate=0.01',
#60Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#59)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

ISTM that you probably intended "\(.*\)" (actual parenthesis) instead of
"(.*)" (memorization) in the data generation message check.

Thank you, fixed it.

Otherwise all is well for me.

Attached the updated version patch.

Applies, compiles, make check & tap test ok, doc is fine.

All is well for me: the feature is useful, code is simple and clean, it is
easy to invoke, easy to extend as well, which I'm planning to do once it
gets in.

I switched the patch to "Ready for Committers". No doubt they will have
their own opinions about it. Let's wait and see.

Thanks,

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#61Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#60)
Re: pgbench: Skipping the creating primary keys after initialization

On Thu, Sep 21, 2017 at 5:23 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

ISTM that you probably intended "\(.*\)" (actual parenthesis) instead of
"(.*)" (memorization) in the data generation message check.

Thank you, fixed it.

Otherwise all is well for me.

Attached the updated version patch.

Applies, compiles, make check & tap test ok, doc is fine.

All is well for me: the feature is useful, code is simple and clean, it is
easy to invoke, easy to extend as well, which I'm planning to do once it
gets in.

I switched the patch to "Ready for Committers". No doubt they will have
their own opinions about it. Let's wait and see.

Thank you for the reviewing this patch!!

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#62Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Fabien COELHO (#60)
Re: pgbench: Skipping the creating primary keys after initialization

Hello Masahiko-san,

Attached the updated version patch.

Applies, compiles, make check & tap test ok, doc is fine.

All is well for me: the feature is useful, code is simple and clean, it is
easy to invoke, easy to extend as well, which I'm planning to do once it gets
in.

I switched the patch to "Ready for Committers". No doubt they will have their
own opinions about it. Let's wait and see.

The patch needs a rebase in the documentation because of the xml-ilization
of the sgml doc.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#63Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fabien COELHO (#62)
1 attachment(s)
Re: pgbench: Skipping the creating primary keys after initialization

On Wed, Oct 18, 2017 at 5:32 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Masahiko-san,

Attached the updated version patch.

Applies, compiles, make check & tap test ok, doc is fine.

All is well for me: the feature is useful, code is simple and clean, it is
easy to invoke, easy to extend as well, which I'm planning to do once it
gets in.

I switched the patch to "Ready for Committers". No doubt they will have
their own opinions about it. Let's wait and see.

The patch needs a rebase in the documentation because of the xml-ilization
of the sgml doc.

Thank you for the notification! Attached rebased patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachments:

pgbench_custom_initialization_v16.patchapplication/octet-stream; name=pgbench_custom_initialization_v16.patchDownload
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e509e6c..5a16b81 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -159,6 +159,75 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
      </varlistentry>
 
      <varlistentry>
+      <term><option>-I {<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <term><option>--custom-initialize={<replaceable>custom_init_command</replaceable> [...]}</option></term>
+      <listitem>
+       <para>
+        Specify custom initialization steps.
+        <replaceable>custom_init_command</replaceable> specifies custom
+        initialization steps for the initialization. The default is
+        <literal>ctgvp</literal>. Each command is invoked in the
+        specified order. The supported commands are:
+
+        <variablelist>
+         <varlistentry>
+          <term><literal>c</literal> (cleanup)</term>
+          <listitem>
+           <para>
+            Destroying existing pgbench tables: <structname>pgbench_accounts</structname>,
+            <structname>pgbench_branches</structname>, <structname>pgbench_history</structname>
+            and <structname>pgbench_tellers</structname>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>t</literal> (table creation)</term>
+          <listitem>
+           <para>
+            Create four tables <structname>pgbench_accounts</structname>,
+            <structname>pgbench_branches</structname>, <structname>pgbench_history</structname>
+            and <structname>pgbench_tellers</structname>.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>g</literal> (generate data)</term>
+          <listitem>
+           <para>
+            Generate data and load it to standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>v</literal> (vacuum)</term>
+          <listitem>
+           <para>
+            Invoke vacuum on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+          <term><literal>p</literal> (primary key)</term>
+          <listitem>
+           <para>
+            Create primary keys on standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+         <varlistentry>
+         <term><literal>f</literal> (foreign key)</term>
+          <listitem>
+           <para>
+            Create foreign keys constraints between the standard tables.
+           </para>
+          </listitem>
+         </varlistentry>
+        </variablelist>
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-F</option> <replaceable>fillfactor</replaceable></term>
       <term><option>--fillfactor=</option><replaceable>fillfactor</replaceable></term>
       <listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 5d8a01c..6063cc6 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -95,6 +95,8 @@ static int	pthread_join(pthread_t th, void **thread_return);
 
 #define MIN_GAUSSIAN_PARAM		2.0 /* minimum parameter for gauss */
 
+#define DEFAULT_INIT_COMMANDS "ctgvp"
+
 int			nxacts = 0;			/* number of transactions per client */
 int			duration = 0;		/* duration in seconds */
 int64		end_time = 0;		/* when to stop in micro seconds, under -T */
@@ -112,9 +114,9 @@ int			scale = 1;
 int			fillfactor = 100;
 
 /*
- * create foreign key constraints on the tables?
+ * no vacuum at all before testing?
  */
-int			foreign_keys = 0;
+bool			is_no_vacuum = false;
 
 /*
  * use unlogged tables?
@@ -458,6 +460,12 @@ static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
 static void addScript(ParsedScript script);
 static void *threadRun(void *arg);
 static void setalarm(int seconds);
+static void initCleanupTables(PGconn *con);
+static void initCreateTables(PGconn *con);
+static void initLoadData(PGconn *con);
+static void initVacuum(PGconn *con);
+static void initCreatePKeys(PGconn *con);
+static void initCreateFKeys(PGconn *con);
 
 
 /* callback functions for our flex lexer */
@@ -475,6 +483,8 @@ usage(void)
 		   "  %s [OPTION]... [DBNAME]\n"
 		   "\nInitialization options:\n"
 		   "  -i, --initialize         invokes initialization mode\n"
+		   "  -I, --custom-initialize=[ctgvpf]+ (default \"ctgvp\")\n"
+		   "                           initialize with custom initialization commands\n"
 		   "  -F, --fillfactor=NUM     set fill factor\n"
 		   "  -n, --no-vacuum          do not run VACUUM after initialization\n"
 		   "  -q, --quiet              quiet logging (one message each 5 seconds)\n"
@@ -2601,9 +2611,55 @@ disconnect_all(CState *state, int length)
 	}
 }
 
-/* create tables and setup data */
+/* Check custom initialization commands */
+static void
+checkCustomCommands(char *commands)
+{
+	char	*cmd;
+
+	if (commands[0] == '\0')
+	{
+		fprintf(stderr, "initialize command is empty\n");
+		exit(1);
+	}
+
+	for (cmd = commands; *cmd != '\0'; cmd++)
+	{
+		if (strchr("ctgvpf ", *cmd) == NULL)
+		{
+			fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+			fprintf(stderr, "possible commands are: \"c\", \"t\", \"g\", \"v\", \"p\", \"f\"\n");
+			exit(1);
+		}
+	}
+}
+
+/* Remove old tables, if it exists */
+static void
+initCleanupTables(PGconn *con)
+{
+	int i;
+
+	/* tables in reverse foreign key dependencies order */
+	static const char *Tables[] = {
+		"pgbench_history", "pgbench_tellers", "pgbench_accounts", "pgbench_branches"
+	};
+
+	fprintf(stderr, "cleaning up...\n");
+
+	for (i = 0; i < lengthof(Tables); i++)
+	{
+		char	buffer[256];
+
+		/* Remove old table, if it exists. */
+		snprintf(buffer, sizeof(buffer), "drop table if exists %s", Tables[i]);
+		executeStatement(con, buffer);
+	}
+}
+
+/* Create tables */
 static void
-init(bool is_no_vacuum)
+initCreateTables(PGconn *con)
 {
 /*
  * The scale factor at/beyond which 32-bit integers are insufficient for
@@ -2658,34 +2714,10 @@ init(bool is_no_vacuum)
 			1
 		}
 	};
-	static const char *const DDLINDEXes[] = {
-		"alter table pgbench_branches add primary key (bid)",
-		"alter table pgbench_tellers add primary key (tid)",
-		"alter table pgbench_accounts add primary key (aid)"
-	};
-	static const char *const DDLKEYs[] = {
-		"alter table pgbench_tellers add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_accounts add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (bid) references pgbench_branches",
-		"alter table pgbench_history add foreign key (tid) references pgbench_tellers",
-		"alter table pgbench_history add foreign key (aid) references pgbench_accounts"
-	};
 
-	PGconn	   *con;
-	PGresult   *res;
-	char		sql[256];
-	int			i;
-	int64		k;
+	int i;
 
-	/* used to track elapsed time and estimate of the remaining time */
-	instr_time	start,
-				diff;
-	double		elapsed_sec,
-				remaining_sec;
-	int			log_interval = 1;
-
-	if ((con = doConnect()) == NULL)
-		exit(1);
+	fprintf(stderr, "creating tables...\n");
 
 	for (i = 0; i < lengthof(DDLs); i++)
 	{
@@ -2694,10 +2726,6 @@ init(bool is_no_vacuum)
 		const struct ddlinfo *ddl = &DDLs[i];
 		const char *cols;
 
-		/* Remove old table, if it exists. */
-		snprintf(buffer, sizeof(buffer), "drop table if exists %s", ddl->table);
-		executeStatement(con, buffer);
-
 		/* Construct new create table statement. */
 		opts[0] = '\0';
 		if (ddl->declare_fillfactor)
@@ -2722,9 +2750,31 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
+/* Fill the standard table with some data */
+static void
+initLoadData(PGconn *con)
+{
+	char		sql[256];
+	PGresult   *res;
+	int			i;
+	int64		k;
+
+	/* used to track elapsed time and estimate of the remaining time */
+	instr_time	start,
+				diff;
+	double		elapsed_sec,
+				remaining_sec;
+	int			log_interval = 1;
+
+	/*
+	 * fill the pgbench_accounts table with some data
+	 */
 	executeStatement(con, "begin");
 
+	executeStatement(con, "truncate pgbench_history, pgbench_branches, pgbench_tellers, pgbench_accounts");
+
 	for (i = 0; i < nbranches * scale; i++)
 	{
 		/* "filler" column defaults to NULL */
@@ -2743,16 +2793,6 @@ init(bool is_no_vacuum)
 		executeStatement(con, sql);
 	}
 
-	executeStatement(con, "commit");
-
-	/*
-	 * fill the pgbench_accounts table with some data
-	 */
-	fprintf(stderr, "creating tables...\n");
-
-	executeStatement(con, "begin");
-	executeStatement(con, "truncate pgbench_accounts");
-
 	res = PQexec(con, "copy pgbench_accounts from stdin");
 	if (PQresultStatus(res) != PGRES_COPY_IN)
 	{
@@ -2827,20 +2867,33 @@ init(bool is_no_vacuum)
 		exit(1);
 	}
 	executeStatement(con, "commit");
+}
 
-	/* vacuum */
-	if (!is_no_vacuum)
-	{
-		fprintf(stderr, "vacuum...\n");
-		executeStatement(con, "vacuum analyze pgbench_branches");
-		executeStatement(con, "vacuum analyze pgbench_tellers");
-		executeStatement(con, "vacuum analyze pgbench_accounts");
-		executeStatement(con, "vacuum analyze pgbench_history");
-	}
+/* Invoke vacuum on all tables */
+static void
+initVacuum(PGconn *con)
+{
+	fprintf(stderr, "vacuum...\n");
+	executeStatement(con, "vacuum analyze pgbench_branches");
+	executeStatement(con, "vacuum analyze pgbench_tellers");
+	executeStatement(con, "vacuum analyze pgbench_accounts");
+	executeStatement(con, "vacuum analyze pgbench_history");
+}
+
+/*
+ * Create primary keys on three tables; pgbench_accounts,
+ * pgbench_branches and pgbench_tellers.
+ */
+static void
+initCreatePKeys(PGconn *con)
+{
+	static const char *const DDLINDEXes[] = {
+		"alter table pgbench_branches add primary key (bid)",
+		"alter table pgbench_tellers add primary key (tid)",
+		"alter table pgbench_accounts add primary key (aid)"
+	};
+	int i;
 
-	/*
-	 * create indexes
-	 */
 	fprintf(stderr, "set primary keys...\n");
 	for (i = 0; i < lengthof(DDLINDEXes); i++)
 	{
@@ -2861,16 +2914,71 @@ init(bool is_no_vacuum)
 
 		executeStatement(con, buffer);
 	}
+}
 
-	/*
-	 * create foreign keys
-	 */
-	if (foreign_keys)
+/*
+ * Create foreign key constraints between the standard tables
+ */
+static void
+initCreateFKeys(PGconn *con)
+{
+	static const char *const DDLKEYs[] = {
+		"alter table pgbench_tellers add constraint pgbench_tellers_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_accounts add constraint pgbench_accounts_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_bid_fkey foreign key (bid) references pgbench_branches",
+		"alter table pgbench_history add constraint pgbench_history_tid_fkey foreign key (tid) references pgbench_tellers",
+		"alter table pgbench_history add constraint pgbench_history_aid_fkey foreign key (aid) references pgbench_accounts"
+	};
+
+	int i;
+
+	fprintf(stderr, "set foreign keys...\n");
+	for (i = 0; i < lengthof(DDLKEYs); i++)
 	{
-		fprintf(stderr, "set foreign keys...\n");
-		for (i = 0; i < lengthof(DDLKEYs); i++)
+		executeStatement(con, DDLKEYs[i]);
+	}
+}
+
+/* invoke each initialization commands */
+static void
+init(char *initialize_cmds)
+{
+	char *cmd = initialize_cmds;
+	PGconn	   *con;
+
+	if ((con = doConnect()) == NULL)
+		exit(1);
+
+	for (cmd = initialize_cmds; *cmd != '\0'; cmd++)
+	{
+		switch (*cmd)
 		{
-			executeStatement(con, DDLKEYs[i]);
+			case 'c':
+				initCleanupTables(con);
+				break;
+			case 't':
+				initCreateTables(con);
+				break;
+			case 'g':
+				initLoadData(con);
+				break;
+			case 'v':
+				initVacuum(con);
+				break;
+			case 'p':
+				initCreatePKeys(con);
+				break;
+			case 'f':
+				initCreateFKeys(con);
+				break;
+			case ' ':
+				break;	/* skip */
+			default:
+			{
+				fprintf(stderr, "invalid custom initialization script command \"%c\"\n", *cmd);
+				PQfinish(con);
+				exit(1);
+			}
 		}
 	}
 
@@ -3644,6 +3752,7 @@ main(int argc, char **argv)
 		{"fillfactor", required_argument, NULL, 'F'},
 		{"host", required_argument, NULL, 'h'},
 		{"initialize", no_argument, NULL, 'i'},
+		{"custom-initialize", required_argument, NULL, 'I'},
 		{"jobs", required_argument, NULL, 'j'},
 		{"log", no_argument, NULL, 'l'},
 		{"latency-limit", required_argument, NULL, 'L'},
@@ -3662,7 +3771,6 @@ main(int argc, char **argv)
 		{"username", required_argument, NULL, 'U'},
 		{"vacuum-all", no_argument, NULL, 'v'},
 		/* long-named only options */
-		{"foreign-keys", no_argument, &foreign_keys, 1},
 		{"index-tablespace", required_argument, NULL, 3},
 		{"tablespace", required_argument, NULL, 2},
 		{"unlogged-tables", no_argument, &unlogged_tables, 1},
@@ -3670,12 +3778,13 @@ main(int argc, char **argv)
 		{"aggregate-interval", required_argument, NULL, 5},
 		{"progress-timestamp", no_argument, NULL, 6},
 		{"log-prefix", required_argument, NULL, 7},
+		{"foreign-keys", no_argument, NULL, 8},
 		{NULL, 0, NULL, 0}
 	};
 
 	int			c;
 	int			is_init_mode = 0;	/* initialize mode? */
-	int			is_no_vacuum = 0;	/* no vacuum at all before testing? */
+	char		*initialize_cmds;
 	int			do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
 	int			optindex;
 	bool		scale_given = false;
@@ -3736,7 +3845,9 @@ main(int argc, char **argv)
 	state = (CState *) pg_malloc(sizeof(CState));
 	memset(state, 0, sizeof(CState));
 
-	while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+	initialize_cmds = pg_strdup(DEFAULT_INIT_COMMANDS);
+
+	while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
 	{
 		char	   *script;
 
@@ -3745,12 +3856,30 @@ main(int argc, char **argv)
 			case 'i':
 				is_init_mode++;
 				break;
+			case 'I':
+				pg_free(initialize_cmds);
+				initialize_cmds = pg_strdup(optarg);
+
+				/* Check input custom initialization command string */
+				checkCustomCommands(initialize_cmds);
+
+				initialization_option_set = true;
+				break;
 			case 'h':
 				pghost = pg_strdup(optarg);
 				break;
 			case 'n':
-				is_no_vacuum++;
-				break;
+				{
+					char *p;
+
+					is_no_vacuum = true;
+
+					/* Get rid of vacuum commands from initialization commands */
+					while ((p = strchr(initialize_cmds, 'v')) != NULL)
+						*p = ' ';
+
+					break;
+				}
 			case 'v':
 				do_vacuum_accounts++;
 				break;
@@ -3973,7 +4102,7 @@ main(int argc, char **argv)
 				break;
 			case 0:
 				/* This covers long options which take no argument. */
-				if (foreign_keys || unlogged_tables)
+				if (unlogged_tables)
 					initialization_option_set = true;
 				break;
 			case 2:				/* tablespace */
@@ -4011,6 +4140,24 @@ main(int argc, char **argv)
 				benchmarking_option_set = true;
 				logfile_prefix = pg_strdup(optarg);
 				break;
+			case 8:
+				{
+					/*
+					 * If a building foreign key command is not specified in
+					 * initialization commands, the command is added to the end.
+					 */
+					if (strchr(initialize_cmds, 'f') == NULL)
+					{
+						int n_cmds = strlen(initialize_cmds) + 1;
+
+						initialize_cmds = (char *) pg_realloc(initialize_cmds,
+															  sizeof(char) * n_cmds + 1);
+						initialize_cmds[n_cmds - 1] = 'f';
+						initialize_cmds[n_cmds] = '\0';
+					}
+					initialization_option_set = true;
+				}
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -4090,7 +4237,7 @@ main(int argc, char **argv)
 			exit(1);
 		}
 
-		init(is_no_vacuum);
+		init(initialize_cmds);
 		exit(0);
 	}
 	else
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 11bc0fe..5545202 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -84,13 +84,37 @@ pgbench(
 '--initialize --scale=1 --unlogged-tables --fillfactor=98 --foreign-keys --quiet --tablespace=pg_default --index-tablespace=pg_default',
 	0,
 	[qr{^$}i],
-	[   qr{creating tables},
+	[   qr{cleaning up},
+		qr{creating tables},
 		qr{vacuum},
 		qr{set primary keys},
 		qr{set foreign keys},
 		qr{done\.} ],
 	'pgbench scale 1 initialization');
 
+# Custom initialization option
+pgbench(
+	'--initialize --custom-initialize=ctpvgf --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{.* of .* tuples \(.*\) done},
+		qr{set primary keys},
+		qr{vacuum},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization script');
+
+# Custom initialization with other initialization options
+pgbench(
+	'--initialize --custom-initialize=ctpvgvv --no-vacuum --foreign-keys --unlogged-tables', 0, [qr{^$}],
+	[   qr{cleaning up},
+		qr{creating tables},
+		qr{set primary keys},
+		qr{.* of .* tuples \(.*\) done},
+		qr{set foreign keys},
+		qr{done\.} ],
+	'pgbench custom initialization with other initialization options');
+
 # Run all builtin scripts, for a few transactions each
 pgbench(
 	'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index d6b3d4f..b0c7f63 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -73,6 +73,9 @@ my @options = (
 	[ 'ambiguous builtin', '-b s', [qr{ambiguous}] ],
 	[   '--progress-timestamp => --progress', '--progress-timestamp',
 		[qr{allowed only under}] ],
+	[ 'custom init without init option', '-I ctg', [qr{cannot be used in benchmarking mode}] ],
+	[ 'invalid custom init command', '-i -I cta',
+		[qr{invalid custom initialization script command}, qr{possible commands are} ] ],
 
 	# loging sub-options
 	[   'sampling => log', '--sampling-rate=0.01',
#64Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Masahiko Sawada (#63)
Re: pgbench: Skipping the creating primary keys after initialization

The patch needs a rebase in the documentation because of the xml-ilization
of the sgml doc.

Thank you for the notification! Attached rebased patch.

Ok. The new version works for me.

--
Fabien.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#65Tom Lane
tgl@sss.pgh.pa.us
In reply to: Masahiko Sawada (#63)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

Masahiko Sawada <sawada.mshk@gmail.com> writes:

[ pgbench_custom_initialization_v16.patch ]

I'm starting to review this patch, and I wonder how it is that you
ended up with "c" as the command letter for dropping existing tables.
Seems like "d" for DROP would be much less confusing. I see that at
one point "d" meant the data load step, but since you've gone with
"g" for "generate data" that conflict is gone.

regards, tom lane

#66Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Tom Lane (#65)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

Hello Tom,

Masahiko Sawada <sawada.mshk@gmail.com> writes:

[ pgbench_custom_initialization_v16.patch ]

I'm starting to review this patch, and I wonder how it is that you
ended up with "c" as the command letter for dropping existing tables.
Seems like "d" for DROP would be much less confusing. I see that at
one point "d" meant the data load step, but since you've gone with
"g" for "generate data" that conflict is gone.

Indeed, you are right. As a reviewer, I can recall that there were some
hesitations, not sure we ended up with the best possible choice.

Note that if "c" is freed by "d" (drop), then it may be worth considering
that "t" (table) could be replaced by "c" (create).

I'm fine with anything consistent and easy to memorize, really.

--
Fabien.

#67Tom Lane
tgl@sss.pgh.pa.us
In reply to: Fabien COELHO (#66)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

Fabien COELHO <coelho@cri.ensmp.fr> writes:

Masahiko Sawada <sawada.mshk@gmail.com> writes:

[ pgbench_custom_initialization_v16.patch ]

I'm starting to review this patch, and I wonder how it is that you
ended up with "c" as the command letter for dropping existing tables.
Seems like "d" for DROP would be much less confusing. I see that at
one point "d" meant the data load step, but since you've gone with
"g" for "generate data" that conflict is gone.

Indeed, you are right. As a reviewer, I can recall that there were some
hesitations, not sure we ended up with the best possible choice.

OK, will make the appropriate changes.

Note that if "c" is freed by "d" (drop), then it may be worth considering
that "t" (table) could be replaced by "c" (create).

I thought about that, but the argument that 'c' might mean different
sorts of create steps (e.g. create index) seemed reasonable. I think
we're best off leaving it as 't' in case of future expansion.

regards, tom lane

#68Fabien COELHO
coelho@cri.ensmp.fr
In reply to: Tom Lane (#67)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

Note that if "c" is freed by "d" (drop), then it may be worth considering
that "t" (table) could be replaced by "c" (create).

I thought about that, but the argument that 'c' might mean different
sorts of create steps (e.g. create index) seemed reasonable. I think
we're best off leaving it as 't' in case of future expansion.

Ok. Fine with me.

--
Fabien.

#69Tom Lane
tgl@sss.pgh.pa.us
In reply to: Fabien COELHO (#68)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

Fabien COELHO <coelho@cri.ensmp.fr> writes:

Note that if "c" is freed by "d" (drop), then it may be worth considering
that "t" (table) could be replaced by "c" (create).

I thought about that, but the argument that 'c' might mean different
sorts of create steps (e.g. create index) seemed reasonable. I think
we're best off leaving it as 't' in case of future expansion.

Ok. Fine with me.

Committed that way, with some additional mostly-cosmetic changes:

* I did not much like the "--custom-initialize" option name, as that
conveys no information; arguably any option makes the initialization
"custom" in some way. Moreover it invites confusion with the use of
"custom" elsewhere in pgbench's docs for custom benchmarking scripts.
After some thought I chose "--init-steps", though if anyone has a better
idea I'm happy to listen. I would have gone with just "--steps", except
that the -s and -S short forms are both taken already. (Hm, maybe we
should use --steps and not have a short form?)

* I also tried to make the code use the "step" terminology consistently
internally, since it's already using "command" to refer to commands in
benchmarking scripts.

* I fixed it so that conflicts between the step-selecting options
are resolved after we've scanned all the options. The submitted
coding would do the wrong thing with, eg, --foreign-keys before -I.

* I changed the "drop" step to just drop all four tables in one
command; that way avoids having to make any assumption about what
foreign keys exist. (I suppose that constraints leading in different
directions aren't all that likely, but if we're trying to cater to
not-quite-default configurations, we might as well do this.)

* Minor other cosmetic cleanup.

regards, tom lane

#70Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Tom Lane (#69)
Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

On Tue, Nov 14, 2017 at 6:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Fabien COELHO <coelho@cri.ensmp.fr> writes:

Note that if "c" is freed by "d" (drop), then it may be worth considering
that "t" (table) could be replaced by "c" (create).

I thought about that, but the argument that 'c' might mean different
sorts of create steps (e.g. create index) seemed reasonable. I think
we're best off leaving it as 't' in case of future expansion.

Ok. Fine with me.

Committed that way, with some additional mostly-cosmetic changes:

* I did not much like the "--custom-initialize" option name, as that
conveys no information; arguably any option makes the initialization
"custom" in some way. Moreover it invites confusion with the use of
"custom" elsewhere in pgbench's docs for custom benchmarking scripts.
After some thought I chose "--init-steps", though if anyone has a better
idea I'm happy to listen. I would have gone with just "--steps", except
that the -s and -S short forms are both taken already. (Hm, maybe we
should use --steps and not have a short form?)

* I also tried to make the code use the "step" terminology consistently
internally, since it's already using "command" to refer to commands in
benchmarking scripts.

* I fixed it so that conflicts between the step-selecting options
are resolved after we've scanned all the options. The submitted
coding would do the wrong thing with, eg, --foreign-keys before -I.

* I changed the "drop" step to just drop all four tables in one
command; that way avoids having to make any assumption about what
foreign keys exist. (I suppose that constraints leading in different
directions aren't all that likely, but if we're trying to cater to
not-quite-default configurations, we might as well do this.)

* Minor other cosmetic cleanup.

Thank you for reviewing and committing the patch. I couldn't respond
promptly because slept but I agree with the changes.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center