Should we use make -k on the buildfarm?

Andrew Dunstan <andrew@dunslane.net> writes:

Peter Eisentraut has suggested that we should run "make -k" instead of
plain "make" for most or all of the buildfarm steps. This flag
essentially instructs make to keep going rather than fail at the first
error. We haven't done that for the last five or six years that the
buildfarm has been running, and nobody up to now has complained (that I
recall). I don't have any great objection, but before I make this change
I thought it might be as well to canvas a wider range of opinion.

So, does anyone else have thoughts about it?

I don't really care about make -k as such. What I *have* occasionally
wished for is that the buildfarm script would act more like make -k with
respect to the various test stages. That is, not abandon the whole test
after one stage fails, but allow stages that don't logically depend on
the failed one to proceed. But I'm not sure how hard that would be ---
quite aside from coding complexity, it would mean that you could have
more than one failing stage, and I don't know how you'd show that in the
dashboard.

Anyway, no objection to Peter's request; but there's another TODO item
for your buildfarm list, if you want to accept it.

regards, tom lane

On 11/06/2010 01:07 PM, Tom Lane wrote:

What I *have* occasionally
wished for is that the buildfarm script would act more like make -k with
respect to the various test stages. That is, not abandon the whole test
after one stage fails, but allow stages that don't logically depend on
the failed one to proceed. But I'm not sure how hard that would be ---
quite aside from coding complexity, it would mean that you could have
more than one failing stage, and I don't know how you'd show that in the
dashboard.

That would be a significant architectural change. I'm not sure how many
steps we could run this way.

Here's the list of tests from a recent run, leaving out stopping and
starting the installed postmaster, and locale specifiers:

SCM-checkout
configure
make
check
make-contrib
make-install
install-contrib
initdb
install-check
pl-install-check
contrib-install-check
ecpg-check

Currently, the implied dependency list is in this order. We could have
"make-contrib" depend only on "make" rather than "check",
"pl-install-check" and "contrib-install-check" depend on "initdb", and
"ecpg-check" depend on "make" rather than anything that comes after. I
think that's about the limit of what we could sensibly relax

I'm not sure that would be a great advance. Certainly, right now I'm
going to be putting effort into the FTS stuff which I think should be
much higher up your list of wants.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/06/2010 01:07 PM, Tom Lane wrote:

What I *have* occasionally
wished for is that the buildfarm script would act more like make -k with
respect to the various test stages.

I'm not sure that would be a great advance. Certainly, right now I'm
going to be putting effort into the FTS stuff which I think should be
much higher up your list of wants.

Agreed, that would be far more useful.

regards, tom lane

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/06/2010 01:07 PM, Tom Lane wrote:

What I *have* occasionally
wished for is that the buildfarm script would act more like make -k with
respect to the various test stages.

I'm not sure that would be a great advance. Certainly, right now I'm
going to be putting effort into the FTS stuff which I think should be
much higher up your list of wants.

Agreed, that would be far more useful.

FTS stuff? Would that also include additions to the current SOAP
interface of the buildfarm?

lg

Stefan

On 11/08/2010 10:27 AM, Stefan Kaltenbrunner wrote:

Tom Lane wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/06/2010 01:07 PM, Tom Lane wrote:

What I *have* occasionally
wished for is that the buildfarm script would act more like make -k
with
respect to the various test stages.

I'm not sure that would be a great advance. Certainly, right now I'm
going to be putting effort into the FTS stuff which I think should
be much higher up your list of wants.

Agreed, that would be far more useful.

FTS stuff? Would that also include additions to the current SOAP
interface of the buildfarm?

Possibly. I haven't looked at how we'll use it yet.

cheers

andrew

On lör, 2010-11-06 at 14:45 -0400, Andrew Dunstan wrote:

Here's the list of tests from a recent run, leaving out stopping and
starting the installed postmaster, and locale specifiers:

SCM-checkout
configure
make
check
make-contrib
make-install
install-contrib
initdb
install-check
pl-install-check
contrib-install-check
ecpg-check

Currently, the implied dependency list is in this order. We could have
"make-contrib" depend only on "make" rather than "check",
"pl-install-check" and "contrib-install-check" depend on "initdb",
and "ecpg-check" depend on "make" rather than anything that comes
after. I think that's about the limit of what we could sensibly relax

In principle you could get this down to

SCM-checkout
configure
make -k world
make -k check-world # target doesn't exist yet
make -k install-world
initdb
make -k installcheck-world

That way you don't have to update the buildfarm code every time a new
test suite is added.

On 11/08/2010 02:49 PM, Peter Eisentraut wrote:

On lör, 2010-11-06 at 14:45 -0400, Andrew Dunstan wrote:

Here's the list of tests from a recent run, leaving out stopping and
starting the installed postmaster, and locale specifiers:

SCM-checkout
configure
make
check
make-contrib
make-install
install-contrib
initdb
install-check
pl-install-check
contrib-install-check
ecpg-check

Currently, the implied dependency list is in this order. We could have
"make-contrib" depend only on "make" rather than "check",
"pl-install-check" and "contrib-install-check" depend on "initdb",
and "ecpg-check" depend on "make" rather than anything that comes
after. I think that's about the limit of what we could sensibly relax

In principle you could get this down to

SCM-checkout
configure
make -k world
make -k check-world # target doesn't exist yet
make -k install-world
initdb
make -k installcheck-world

That way you don't have to update the buildfarm code every time a new
test suite is added.

No, we can't do that for several reasons. Here are a couple.

First, the buildfarm doesn't build the docs. That's a deliberate
decision, based on the fact that not every member has the required
software installed. And second these targets only exist for 9.0 and/or
later.

cheers

andrew

On mån, 2010-11-08 at 15:01 -0500, Andrew Dunstan wrote:

First, the buildfarm doesn't build the docs. That's a deliberate
decision, based on the fact that not every member has the required
software installed. And second these targets only exist for 9.0 and/or
later.

I'm aware of those issues. I'm just saying you could consolidate things
along those lines in the long term.

Peter Eisentraut <peter_e@gmx.net> writes:

On mån, 2010-11-08 at 15:01 -0500, Andrew Dunstan wrote:

First, the buildfarm doesn't build the docs. That's a deliberate
decision, based on the fact that not every member has the required
software installed. And second these targets only exist for 9.0 and/or
later.

I'm aware of those issues. I'm just saying you could consolidate things
along those lines in the long term.

Probably not: your proposal depends on having "make" available, which it
won't be for Windows builds. I doubt Andrew wishes to implement
make-equivalent logic in the buildfarm script.

regards, tom lane

On 11/08/2010 03:38 PM, Tom Lane wrote:

Peter Eisentraut<peter_e@gmx.net> writes:

On mån, 2010-11-08 at 15:01 -0500, Andrew Dunstan wrote:

First, the buildfarm doesn't build the docs. That's a deliberate
decision, based on the fact that not every member has the required
software installed. And second these targets only exist for 9.0 and/or
later.

I'm aware of those issues. I'm just saying you could consolidate things
along those lines in the long term.

Probably not: your proposal depends on having "make" available, which it
won't be for Windows builds. I doubt Andrew wishes to implement
make-equivalent logic in the buildfarm script.

No, and what is more, it runs counter to the philosophy of the
buildfarm, which is basically to do by automation what a human would do
by hand in building and testing Postgres.

cheers

andrew

On mån, 2010-11-08 at 15:48 -0500, Andrew Dunstan wrote:

No, and what is more, it runs counter to the philosophy of the
buildfarm, which is basically to do by automation what a human would
do by hand in building and testing Postgres.

Yes, but a human would rarely actually do by hand all the 12 steps you
list, which is kind of part of the problem I'm trying to address. It's
much simpler to run the 7 steps I list, which are easy-to-remember
top-level targets, than to remember to run 5 or 6 obscure test suites by
hand.

Anyway, doesn't matter, the buildfarm works fine, but your argument
above is backwards.