Going for "all green" buildfarm results

Tom Lane wrote:

I've been making another pass over getting rid of buildfarm failures.
The remaining ones I see at the moment are:

firefly HEAD: intermittent failures in the stats test. We seem to have
fixed every other platform back in January, but not this one.

kudu HEAD: one-time failure 6/1/06 in statement_timeout test, never seen
before. Is it possible system was under enough load that the 1-second
timeout fired before control reached the exception block?

[...]

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&dt=2006-05-12%2005:30:14

[...]

cobra, stoat, sponge 7.4: pilot error. Either install Tk or configure
--without-tk.

sorry for that but the issue with sponge on 7.4 was fixed nearly a week
ago though there have been no changes until today to trigger a new build ;-)

Stefan

Tom Lane wrote:

I've been making another pass over getting rid of buildfarm failures.
The remaining ones I see at the moment are:

firefly HEAD: intermittent failures in the stats test. We seem to
have fixed every other platform back in January, but not this one.

firefly 7.4: dblink test fails, with what looks like an rpath problem.
Another one that we fixed awhile ago, and the fix worked on every
platform but this one.

firefly 7.3: trivial regression diffs; we could install variant
comparison files if anyone cared.

Firefly is obviously the outlier here. I dunno if anyone cares
enough about SCO to spend time investigating it (I don't). Most of
the others just need a little bit of attention from the machine
owner.

If I generate fixes for firefly (I'm the owner), would they have a prayer
Of being applied?

LER

regards, tom lane

---------------------------(end of
broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 512-248-2683 E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3683 US

Larry Rosenman said:

Tom Lane wrote:

I've been making another pass over getting rid of buildfarm failures.
The remaining ones I see at the moment are:

firefly HEAD: intermittent failures in the stats test. We seem to
have fixed every other platform back in January, but not this one.

firefly 7.4: dblink test fails, with what looks like an rpath problem.
Another one that we fixed awhile ago, and the fix worked on every
platform but this one.

firefly 7.3: trivial regression diffs; we could install variant
comparison files if anyone cared.

Firefly is obviously the outlier here. I dunno if anyone cares
enough about SCO to spend time investigating it (I don't). Most of
the others just need a little bit of attention from the machine
owner.

If I generate fixes for firefly (I'm the owner), would they have a
prayer Of being applied?

Sure, although I wouldn't bother with 7.3 - just take 7.3 out of firefly's
build schedule. That's not carte blanche on fixes, of course - we'd have to
see them.

cheers

andrew

Tom Lane wrote:

Or is
it worth improving buildfarm to be able to skip specific tests?

There is a session on buildfarm improvements scheduled for the Toronto
conference. This is certainly one possibility.

cheers

andrew

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-05-12%2005:30:14

Weird.

SELECT ''::text AS eleven, unique1, unique2, stringu1
FROM onek WHERE unique1 < 50
ORDER BY unique1 DESC LIMIT 20 OFFSET 39;
! ERROR: could not open relation with OID 27035

AFAICS, the only way to get that error in HEAD is if ScanPgRelation
can't find a pg_class row with the mentioned OID. Presumably 27035
belongs to "onek" or one of its indexes. The very next command also
refers to "onek", and doesn't fail, so what we seem to have here is
a transient lookup failure. We've found a btree bug like that once
before ... wonder if there's still one left?

regards, tom lane

"Andrew Dunstan" <andrew@dunslane.net> writes:

Larry Rosenman said:

If I generate fixes for firefly (I'm the owner), would they have a
prayer Of being applied?

Sure, although I wouldn't bother with 7.3 - just take 7.3 out of firefly's
build schedule. That's not carte blanche on fixes, of course - we'd have to
see them.

What he said ... it'd depend entirely on how ugly the fixes are ;-)

regards, tom lane

Tom Lane wrote:

"Andrew Dunstan" <andrew@dunslane.net> writes:

Larry Rosenman said:

If I generate fixes for firefly (I'm the owner), would they have a
prayer Of being applied?

Sure, although I wouldn't bother with 7.3 - just take 7.3 out of
firefly's build schedule. That's not carte blanche on fixes, of
course - we'd have to see them.

What he said ... it'd depend entirely on how ugly the fixes are ;-)

Ok, 7.3 is out of firefly's crontab.

I'll look into 7.4.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 512-248-2683 E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-05-12%2005:30:14

Weird.

SELECT ''::text AS eleven, unique1, unique2, stringu1
FROM onek WHERE unique1 < 50
ORDER BY unique1 DESC LIMIT 20 OFFSET 39;
! ERROR: could not open relation with OID 27035

AFAICS, the only way to get that error in HEAD is if ScanPgRelation
can't find a pg_class row with the mentioned OID. Presumably 27035
belongs to "onek" or one of its indexes. The very next command also
refers to "onek", and doesn't fail, so what we seem to have here is
a transient lookup failure. We've found a btree bug like that once
before ... wonder if there's still one left?

If there is still one left it must be quite hard to trigger (using the
regression tests). Like i said before - I tried quite hard to reproduce
the issue back then - without any success.

Stefan

Larry Rosenman wrote:

Tom Lane wrote:

"Andrew Dunstan" <andrew@dunslane.net> writes:

Larry Rosenman said:

If I generate fixes for firefly (I'm the owner), would they have a
prayer Of being applied?

Sure, although I wouldn't bother with 7.3 - just take 7.3 out of
firefly's build schedule. That's not carte blanche on fixes, of
course - we'd have to see them.

What he said ... it'd depend entirely on how ugly the fixes are ;-)

Ok, 7.3 is out of firefly's crontab.

I'll look into 7.4.

LER

I've taken the cheaters way out for 7.4, and turned off the perl stuff for
now.

as to HEAD, I've played with the system send/recv space parms, and let's see
if
that helps the stats stuff.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 512-248-2683 E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893

Larry Rosenman wrote:

Larry Rosenman wrote:

Tom Lane wrote:

"Andrew Dunstan" <andrew@dunslane.net> writes:

Larry Rosenman said:

If I generate fixes for firefly (I'm the owner), would they have a
prayer Of being applied?

Sure, although I wouldn't bother with 7.3 - just take 7.3 out of
firefly's build schedule. That's not carte blanche on fixes, of
course - we'd have to see them.

What he said ... it'd depend entirely on how ugly the fixes are ;-)

Ok, 7.3 is out of firefly's crontab.

I'll look into 7.4.

LER

I've taken the cheaters way out for 7.4, and turned off the perl
stuff for now.

as to HEAD, I've played with the system send/recv space parms, and
let's see if
that helps the stats stuff.

LER

well, the changes didn't help.

I've pulled ALL the cronjobs from firefly.

consider it dead.

Since it is an outlier, it's not useful.

LER

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 512-248-2683 E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893

-------- Original Message --------
From: Tom Lane <tgl@sss.pgh.pa.us>

kudu HEAD: one-time failure 6/1/06 in statement_timeout test, never seen
before. Is it possible system was under enough load that the 1-second
timeout fired before control reached the exception block?

The load here was no different than any other day. As to whether it's a
real issue or not I have no idea. It is a virtual machine that is subject
to the load on other VMs, but none of them were scheduled to do
anything at the time.

Kris Jurka

Larry Rosenman wrote:

well, the changes didn't help.

I've pulled ALL the cronjobs from firefly.

consider it dead.

Since it is an outlier, it's not useful.

OK, I am marking firefly as retired. That means we have no coverage for
Unixware.

cheers

andrew

I can take other if that helps.

Larry, could you help me in the setup?

Regards,
On Thu, 8 Jun 2006, Andrew Dunstan wrote:

Date: Thu, 08 Jun 2006 10:54:09 -0400
From: Andrew Dunstan <andrew@dunslane.net>
Newsgroups: pgsql.hackers
Subject: Re: Going for 'all green' buildfarm results

Larry Rosenman wrote:

well, the changes didn't help.

I've pulled ALL the cronjobs from firefly.

consider it dead.

Since it is an outlier, it's not useful.

OK, I am marking firefly as retired. That means we have no coverage for
Unixware.

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
15, Chemin des Monges +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)

On Fri, 9 Jun 2006 ohp@pyrenet.fr wrote:

Date: Fri, 9 Jun 2006 11:12:07 +0200
From: ohp@pyrenet.fr
To: Andrew Dunstan <andrew@dunslane.net>, Larry Rosenman <ler@lerctr.org>
Newsgroups: pgsql.hackers
Subject: Re: Going for 'all green' buildfarm results

I can take other if that helps.

Ooops... takeover :)

Larry, could you help me in the setup?

Regards,
On Thu, 8 Jun 2006, Andrew Dunstan wrote:

Date: Thu, 08 Jun 2006 10:54:09 -0400
From: Andrew Dunstan <andrew@dunslane.net>
Newsgroups: pgsql.hackers
Subject: Re: Going for 'all green' buildfarm results

Larry Rosenman wrote:

well, the changes didn't help.

I've pulled ALL the cronjobs from firefly.

consider it dead.

Since it is an outlier, it's not useful.

OK, I am marking firefly as retired. That means we have no coverage for
Unixware.

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
15, Chemin des Monges +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-05-12%2005:30:14

Weird.

SELECT ''::text AS eleven, unique1, unique2, stringu1
FROM onek WHERE unique1 < 50
ORDER BY unique1 DESC LIMIT 20 OFFSET 39;
! ERROR: could not open relation with OID 27035

AFAICS, the only way to get that error in HEAD is if ScanPgRelation
can't find a pg_class row with the mentioned OID. Presumably 27035
belongs to "onek" or one of its indexes. The very next command also
refers to "onek", and doesn't fail, so what we seem to have here is
a transient lookup failure. We've found a btree bug like that once
before ... wonder if there's still one left?

FYI: lionfish just managed to hit that problem again:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-07-29%2023:30:06

Stefan

Stefan Kaltenbrunner wrote:

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-05-12%2005:30:14

Weird.

SELECT ''::text AS eleven, unique1, unique2, stringu1
FROM onek WHERE unique1 < 50
ORDER BY unique1 DESC LIMIT 20 OFFSET 39;
! ERROR: could not open relation with OID 27035

AFAICS, the only way to get that error in HEAD is if ScanPgRelation
can't find a pg_class row with the mentioned OID. Presumably 27035
belongs to "onek" or one of its indexes. The very next command also
refers to "onek", and doesn't fail, so what we seem to have here is
a transient lookup failure. We've found a btree bug like that once
before ... wonder if there's still one left?

FYI: lionfish just managed to hit that problem again:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-07-29%2023:30:06

The error message this time is

! ERROR: could not open relation with OID 27006

It's worth mentioning that the portals_p2 test, which happens in the
parallel group previous to where this test is run, also accesses the
onek table successfully. It may be interesting to see exactly what
relation is 27006.

The test alter_table, which is on the same parallel group as limit (the
failing test), contains these lines:

ALTER INDEX onek_unique1 RENAME TO tmp_onek_unique1;
ALTER INDEX tmp_onek_unique1 RENAME TO onek_unique1;

Maybe this is related.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:

Stefan Kaltenbrunner wrote:

Tom Lane wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

FWIW: lionfish had a weird make check error 3 weeks ago which I
(unsuccessfully) tried to reproduce multiple times after that:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-05-12%2005:30:14

Weird.

SELECT ''::text AS eleven, unique1, unique2, stringu1
FROM onek WHERE unique1 < 50
ORDER BY unique1 DESC LIMIT 20 OFFSET 39;
! ERROR: could not open relation with OID 27035

AFAICS, the only way to get that error in HEAD is if ScanPgRelation
can't find a pg_class row with the mentioned OID. Presumably 27035
belongs to "onek" or one of its indexes. The very next command also
refers to "onek", and doesn't fail, so what we seem to have here is
a transient lookup failure. We've found a btree bug like that once
before ... wonder if there's still one left?

FYI: lionfish just managed to hit that problem again:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-07-29%2023:30:06

The error message this time is

! ERROR: could not open relation with OID 27006

yeah and before it was:
! ERROR: could not open relation with OID 27035

which looks quite related :-)

It's worth mentioning that the portals_p2 test, which happens in the
parallel group previous to where this test is run, also accesses the
onek table successfully. It may be interesting to see exactly what
relation is 27006.

sorry but i don't have access to the cluster in question any more
(lionfish is quite resource starved and I only enabled to keep failed
builds on -HEAD after the last incident ...)

The test alter_table, which is on the same parallel group as limit (the
failing test), contains these lines:

ALTER INDEX onek_unique1 RENAME TO tmp_onek_unique1;
ALTER INDEX tmp_onek_unique1 RENAME TO onek_unique1;

hmm interesting - lionfish is a slow box(250Mhz MIPS) and particulary
low on memory(48MB+140MB swap) so it is quite likely that the parallel
regress tests are driving it into swap - maybe some sort of subtile
timing issue ?

Stefan

Alvaro Herrera <alvherre@commandprompt.com> writes:

Stefan Kaltenbrunner wrote:

FYI: lionfish just managed to hit that problem again:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-07-29%2023:30:06

The test alter_table, which is on the same parallel group as limit (the
failing test), contains these lines:
ALTER INDEX onek_unique1 RENAME TO tmp_onek_unique1;
ALTER INDEX tmp_onek_unique1 RENAME TO onek_unique1;

I bet Alvaro's spotted the problem. ALTER INDEX RENAME doesn't seem to
take any lock on the index's parent table, only on the index itself.
That means that a query on "onek" could be trying to read the pg_class
entries for onek's indexes concurrently with someone trying to commit
a pg_class update to rename an index. If the query manages to visit
the new and old versions of the row in that order, and the commit
happens between, *neither* of the versions would look valid. MVCC
doesn't save us because this is all SnapshotNow.

Not sure what to do about this. Trying to lock the parent table could
easily be a cure-worse-than-the-disease, because it would create
deadlock risks (we've already locked the index before we could look up
and lock the parent). Thoughts?

The path of least resistance might just be to not run these tests in
parallel. The chance of this issue causing problems in the real world
seems small.

regards, tom lane

On Sun, Jul 30, 2006 at 11:44:44AM -0400, Tom Lane wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

Stefan Kaltenbrunner wrote:

FYI: lionfish just managed to hit that problem again:
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=lionfish&amp;dt=2006-07-29%2023:30:06

The test alter_table, which is on the same parallel group as limit (the
failing test), contains these lines:
ALTER INDEX onek_unique1 RENAME TO tmp_onek_unique1;
ALTER INDEX tmp_onek_unique1 RENAME TO onek_unique1;

I bet Alvaro's spotted the problem. ALTER INDEX RENAME doesn't seem to
take any lock on the index's parent table, only on the index itself.
That means that a query on "onek" could be trying to read the pg_class
entries for onek's indexes concurrently with someone trying to commit
a pg_class update to rename an index. If the query manages to visit
the new and old versions of the row in that order, and the commit
happens between, *neither* of the versions would look valid. MVCC
doesn't save us because this is all SnapshotNow.

Not sure what to do about this. Trying to lock the parent table could
easily be a cure-worse-than-the-disease, because it would create
deadlock risks (we've already locked the index before we could look up
and lock the parent). Thoughts?

The path of least resistance might just be to not run these tests in
parallel. The chance of this issue causing problems in the real world
seems small.

It doesn't seem that unusual to want to rename an index on a running
system, and it certainly doesn't seem like the kind of operation that
should pose a problem. So at the very least, we'd need a big fat warning
in the docs about how renaming an index could cause other queries in the
system to fail, and the error message needs to be improved.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461