Adding a pgbench run to buildfarm

"Bort, Paul" <pbort@tmwsystems.com> writes:

Andrew said I should solicit opinions as to what parameters to use. A
cursory search through the archives led me to pick a scaling factor of
10, 5 users, and 100 transactions.

100 transactions seems barely enough to get through startup transients.
Maybe 1000 would be good.

I think the hard part of this is the reporting process. How do we
track how performance varies over time? It doesn't seem very useful
to compare different buildfarm members, but a longitudinal display of
performance on a single buildfarm machine over time would be cool.
(I'm still missing Mark Wong's daily OSDL performance reports :-()

Actually the $64 question here is whether we trust pgbench as the
standard performance test ...

regards, tom lane

Tom Lane wrote:

"Bort, Paul" <pbort@tmwsystems.com> writes:

Andrew said I should solicit opinions as to what parameters to use. A
cursory search through the archives led me to pick a scaling factor of
10, 5 users, and 100 transactions.

100 transactions seems barely enough to get through startup transients.
Maybe 1000 would be good.

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

Cheers

Mark

100 transactions seems barely enough to get through startup
transients.
Maybe 1000 would be good.

OK.

I think the hard part of this is the reporting process. How
do we track how performance varies over time? It doesn't
seem very useful to compare different buildfarm members, but
a longitudinal display of performance on a single buildfarm
machine over time would be cool.
(I'm still missing Mark Wong's daily OSDL performance reports :-()

I was thinking that the output from pgbench would be sent back to the
server and stored somewhere for later analysis.

Actually the $64 question here is whether we trust pgbench as
the standard performance test ...

I think that it's what we've got today, and if tomorrow it gets better,
then the data we get from the buildfarm will improve similarly.

Regards,
Paul Bort

Mark Kirkwood <markir@paradise.net.nz> writes:

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

That gets into a different set of questions, which is what we want the
buildfarm turnaround time to be like. The faster members today produce
a result within 10-15 minutes of pulling their CVS snaps, and I'd be
seriously unhappy if that changed to an hour or three. Maybe we need to
divorce compile/regression tests from performance tests?

regards, tom lane

Tom Lane wrote:

Mark Kirkwood <markir@paradise.net.nz> writes:

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

That gets into a different set of questions, which is what we want the
buildfarm turnaround time to be like. The faster members today produce
a result within 10-15 minutes of pulling their CVS snaps, and I'd be
seriously unhappy if that changed to an hour or three. Maybe we need to
divorce compile/regression tests from performance tests?

Right - this leads to further questions like, what the performance
testing on the buildfarms is actually for. If it is mainly to catch
regressions introduced by any new code, then scale factor 10 (i.e
essentially in memory testing) may in fact be the best way to show this up.

Cheers

Mark

On Mon, 24 Jul 2006, Mark Kirkwood wrote:

Tom Lane wrote:

Mark Kirkwood <markir@paradise.net.nz> writes:

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

That gets into a different set of questions, which is what we want the
buildfarm turnaround time to be like. The faster members today produce
a result within 10-15 minutes of pulling their CVS snaps, and I'd be
seriously unhappy if that changed to an hour or three. Maybe we need to
divorce compile/regression tests from performance tests?

Right - this leads to further questions like, what the performance
testing on the buildfarms is actually for. If it is mainly to catch
regressions introduced by any new code, then scale factor 10 (i.e
essentially in memory testing) may in fact be the best way to show this up.

It introduces a problem though. Not all machines stay the same over time.
A machine may by upgraded, a machine may be getting backed up or may in
some other way be utilised during a performance test. This would skew the
stats for that machine. It may confuse people more than help them...

At the very least, the performance figures would need to be accompanied by
details of what other processes were running and what resources they were
chewing during the test.

This is what was nice about the OSDL approach. Each test was preceeded by
an automatic reinstall of the OS and the machines were specifically for
testing. The tester had complete control.

We could perhaps mimic some of that using virtualisation tools which
control access to system resources but it wont work on all platforms. The
problem is that it probably introduces a new variable, in that I'm not
sure that virtualisation software can absolutely limit CPU resources a
particular container has. That is, you might not be able to get
reproducible runs with the same code. :(

Just some thoughts.

Thanks,

Gavin

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Bort, Paul
Sent: 24 July 2006 04:52
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Adding a pgbench run to buildfarm

-hackers,

With help from Andrew Dunstan, I'm adding the ability to do a pgbench
run after all of the other tests during a buildfarm run.

Andrew said I should solicit opinions as to what parameters to use. A
cursory search through the archives led me to pick a scaling factor of
10, 5 users, and 100 transactions. All of these will be
adjustable using
the build-farm.conf mechanism already in place.

Comments? Suggestions?

Please ensure the run is optional. The machine hosting Snake and
Bandicoot is currently running 16 builds a day, and I'd prefer not to
significantly add to it's load.

Regards, Dave.

Mark Kirkwood wrote:

Tom Lane wrote:

"Bort, Paul" <pbort@tmwsystems.com> writes:

Andrew said I should solicit opinions as to what parameters to use. A
cursory search through the archives led me to pick a scaling factor of
10, 5 users, and 100 transactions.

100 transactions seems barely enough to get through startup transients.
Maybe 1000 would be good.

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

hmm - that "1GB" is a rather optimistic estimate for most of the
buildfarm boxes(mine at least).
Out of the 6 ones I have - only one that actually has much RAM
(allocated) and lionfish for example is rather resource starved at only
48(!) MB of RAM and very limited diskspace - which has been plenty
enough until now doing the builds (with enough swap of course).
I supposed that anything that would result in additional diskspace usage
in excess of maybe 150MB or so would run it out of resources :-(

I'm also not too keen on running excessivly long pgbench runs on some of
the buildfarm members so I would prefer to make that one optional.

Stefan

Dave Page wrote:

With help from Andrew Dunstan, I'm adding the ability to do a pgbench
run after all of the other tests during a buildfarm run.

Please ensure the run is optional. The machine hosting Snake and
Bandicoot is currently running 16 builds a day, and I'd prefer not to
significantly add to it's load.

Rest easy. It will be optional, of course.

cheers

andrew

Gavin Sherry wrote:

Not all machines stay the same over time.
A machine may by upgraded, a machine may be getting backed up or may in
some other way be utilised during a performance test. This would skew the
stats for that machine. It may confuse people more than help them...

At the very least, the performance figures would need to be accompanied by
details of what other processes were running and what resources they were
chewing during the test.

This is what was nice about the OSDL approach. Each test was preceeded by
an automatic reinstall of the OS and the machines were specifically for
testing. The tester had complete control.

We could perhaps mimic some of that using virtualisation tools which
control access to system resources but it wont work on all platforms. The
problem is that it probably introduces a new variable, in that I'm not
sure that virtualisation software can absolutely limit CPU resources a
particular container has. That is, you might not be able to get
reproducible runs with the same code. :(

We are really not going to go in this direction. If you want ideal
performance tests then a heterogenous distributed collection of
autonomous systems like buildfarm is not what you want.

You are going to have to live with the fatc that there will be
occasional, possibly even frequent, blips in the data due to other
activity on the machine.

If you want tightly controlled or very heavy load testing this is the
wrong vehicle.

You might think that what that leaves us is not worth having - the
consensus in Toronto seemed to be that it is worth having, which is why
it is being pursued.

cheers

andrew

Tom Lane wrote:

Mark Kirkwood <markir@paradise.net.nz> writes:

Scale factor 10 produces an accounts table of about 130 Mb. Given that
most HW these days has at least 1G of ram, this probably means not much
retrieval IO is tested (only checkpoint and wal fsync). Do we want to
try 100 or even 200? (or recommend scale factor such that size > ram)?

That gets into a different set of questions, which is what we want the
buildfarm turnaround time to be like. The faster members today produce
a result within 10-15 minutes of pulling their CVS snaps, and I'd be
seriously unhappy if that changed to an hour or three. Maybe we need to
divorce compile/regression tests from performance tests?

We could have the system report build/regression results before going on
to do performance testing. I don't want to divorce them altogether if I
can help it, as it will make cleanup a lot messier.

cheers

andrew

Andrew Dunstan wrote:

We are really not going to go in this direction. If you want ideal
performance tests then a heterogenous distributed collection of
autonomous systems like buildfarm is not what you want.

You are going to have to live with the fatc that there will be
occasional, possibly even frequent, blips in the data due to other
activity on the machine.

If you want tightly controlled or very heavy load testing this is the
wrong vehicle.

You might think that what that leaves us is not worth having - the
consensus in Toronto seemed to be that it is worth having,
which is why
it is being pursued.

I wasn't at the conference, but the impression I'm under is that the
point of this isn't to catch a change that causes a 1% slowdown; the
point is to catch much larger problems, probably 20% slowdown or more.

Given the concerns about running this on machines that don't have a lot
of CPU and disk to spare, should it ship disabled?

Andrew, what do you think of pgbench reports shipping separately? I have
no idea how the server end is set up, so I don't know how much of a pain
that would be.

Regards,
Paul Bort

P.S. My current thought for settings is scaling factor 10, users 5,
transactions 1000.

Bort, Paul wrote:

Given the concerns about running this on machines that don't have a lot
of CPU and disk to spare, should it ship disabled?

Yes, certainly.

Andrew, what do you think of pgbench reports shipping separately? I have
no idea how the server end is set up, so I don't know how much of a pain
that would be.

Well, we'll need to put in some changes to collect the data, certainly.
I don't see why we shouldn't ship the pgbench result separately, but ...

P.S. My current thought for settings is scaling factor 10, users 5,
transactions 1000.

... at this size it's hardly worth it. A quick test on my laptop showed
this taking about a minute for the setup and another minute for the run,
Unless we scale way beyond this I don't see any point in separate reporting.

cheers

andrew

On Sun, Jul 23, 2006 at 11:52:14PM -0400, Bort, Paul wrote:

-hackers,

With help from Andrew Dunstan, I'm adding the ability to do a pgbench
run after all of the other tests during a buildfarm run.

Andrew said I should solicit opinions as to what parameters to use. A
cursory search through the archives led me to pick a scaling factor of
10, 5 users, and 100 transactions. All of these will be adjustable using
the build-farm.conf mechanism already in place.

Why is it being hard-coded? I think it makes a lot more sense to allow
pg_bench options to be specified in the buildfarm config. Even better
yet would be specifying them on the command line, which would allow
members to run a more rigorous test once a day/week (I'm thinking one
that might take 30 minutes, which could well ferret out some issues that
a simple 5 minute test won't).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim Nasby wrote:

Why is it being hard-coded? I think it makes a lot more sense to allow
pg_bench options to be specified in the buildfarm config. Even better
yet would be specifying them on the command line, which would allow
members to run a more rigorous test once a day/week (I'm thinking one
that might take 30 minutes, which could well ferret out some
issues that
a simple 5 minute test won't).

They absolutely won't be hard-coded. I'm asking for values to use as
defaults in the config file.

Also allowing command-line parameters is interesting, but I think we
should wait on it until the initial version is in place.