CVS corruption/mistagging?

Kris Jurka <books@ejurka.com> writes:

It looks like parts of the CVS repository have been mistagged as belonging
to REL7_4_STABLE or have been corrupted somehow:
http://developer.postgresql.org/cvsweb.cgi/pgsql/contrib/btree_gist/btree_bit.c?sortby=date;only_with_tag=REL7_4_STABLE

Hmm ... btree_bit.c shouldn't be in 7.4 at all.

I can't help thinking this has something to do with the recent CVS
server move. Magnus, any thoughts?

regards, tom lane

I wrote:

Kris Jurka <books@ejurka.com> writes:

It looks like parts of the CVS repository have been mistagged as belonging
to REL7_4_STABLE or have been corrupted somehow:
http://developer.postgresql.org/cvsweb.cgi/pgsql/contrib/btree_gist/btree_bit.c?sortby=date;only_with_tag=REL7_4_STABLE

Hmm ... btree_bit.c shouldn't be in 7.4 at all.

I did a fresh checkout of the 7.4 branch and diff'd against my local
copy, and it seems clear that every file that was not in 7.4 at all has
had its HEAD version tagged as REL7_4_STABLE. The files that did exist
then are all right. That's throughout the whole tree, not just in
contrib/btree_gist. I have no idea how you make CVS do that, but I'm
sure there is some magic one-liner for it.

Checking the files in contrib/btree_gist on the CVS server gives a
pretty good fix on who changed it and when:

ls -la

total 466
drwxrwxr-x 6 scrappy dev 1024 Aug 14 22:30 .
drwxrwxr-x 90 scrappy dev 2048 Aug 14 22:27 ..
drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
-r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
-r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
-r--r--r-- 1 meskes dev 12052 Aug 12 09:50 btree_bit.c,v
-r--r--r-- 1 meskes dev 9921 Aug 12 09:50 btree_bytea.c,v
-r--r--r-- 1 meskes dev 10246 Aug 12 09:50 btree_cash.c,v
-r--r--r-- 1 meskes dev 11433 Aug 12 09:50 btree_date.c,v
-r--r--r-- 1 meskes dev 10230 Aug 12 09:50 btree_float4.c,v
-r--r--r-- 1 meskes dev 10093 Aug 12 09:50 btree_float8.c,v
-r--r--r-- 1 meskes dev 33935 Aug 12 09:50 btree_gist.c,v
-r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
-r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
-r--r--r-- 1 meskes dev 14736 Aug 12 09:50 btree_inet.c,v
-r--r--r-- 1 meskes dev 10253 Aug 12 09:50 btree_int2.c,v
-r--r--r-- 1 meskes dev 10240 Aug 12 09:50 btree_int4.c,v
-r--r--r-- 1 meskes dev 10254 Aug 12 09:50 btree_int8.c,v
-r--r--r-- 1 meskes dev 18111 Aug 12 09:50 btree_interval.c,v
-r--r--r-- 1 meskes dev 12025 Aug 12 09:50 btree_macaddr.c,v
-r--r--r-- 1 meskes dev 14671 Aug 12 09:50 btree_numeric.c,v
-r--r--r-- 1 meskes dev 9796 Aug 12 09:50 btree_oid.c,v
-r--r--r-- 1 meskes dev 18247 Aug 12 09:50 btree_text.c,v
-r--r--r-- 1 meskes dev 26180 Aug 12 09:50 btree_time.c,v
-r--r--r-- 1 258 dev 29712 Apr 20 15:16 btree_ts.c,v
-r--r--r-- 1 meskes dev 17588 Aug 12 09:50 btree_utils_num.c,v
-r--r--r-- 1 meskes dev 8959 Aug 12 09:50 btree_utils_num.h,v
-r--r--r-- 1 meskes dev 48824 Aug 12 09:50 btree_utils_var.c,v
-r--r--r-- 1 meskes dev 7099 Aug 12 09:50 btree_utils_var.h,v
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 data
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 expected
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 sql
-r--r--r-- 1 meskes dev 10021 Aug 12 09:50 uninstall_btree_gist.sql,v

Michael, want to fess up to what you did?

In the meantime, though, it's not quite clear why this would lead to
a buildfarm failure --- it should just mean a lot of extraneous files
appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
like btree_gist has some files that used to be autogenerated and are
now in CVS, so the bogusly new versions from CVS are suppressing the
desired generation from the old btree_num.c file.)

regards, tom lane

I wrote:

I did a fresh checkout of the 7.4 branch and diff'd against my local
copy, and it seems clear that every file that was not in 7.4 at all has
had its HEAD version tagged as REL7_4_STABLE. The files that did exist
then are all right. That's throughout the whole tree, not just in
contrib/btree_gist.

As for getting out of it: it looks like the misapplied tags are revision
tags not branch tags, eg

RCS file: /cvsroot/pgsql/contrib/btree_gist/btree_bit.c,v
Working file: btree_bit.c
head: 1.7
branch:
locks: strict
access list:
symbolic names:
REL7_4_STABLE: 1.7 <--- wrong
REL8_0_13: 1.2
REL8_1_9: 1.6
REL8_2_4: 1.6
REL8_0_12: 1.2
REL8_1_8: 1.6
REL8_2_3: 1.6
REL8_0_11: 1.2
REL8_1_7: 1.6
REL8_2_2: 1.6
REL8_0_10: 1.2
REL8_1_6: 1.6
REL8_2_1: 1.6
REL8_2_STABLE: 1.6.0.4

which I think means that there's a one-liner way out of it. The CVS
manual quoth:

To delete a tag, specify the `-d' option to either `cvs tag' or `cvs
rtag'. For example:

cvs rtag -d rel-0-4 tc

deletes the non-branch tag `rel-0-4' from the module `tc'. In the
event that branch tags are encountered within the repository with the
given name, a warning message will be issued and the branch tag will
not be deleted. If you are absolutely certain you know what you are
doing, the `-B' option may be specified to allow deletion of branch
tags. In that case, any non-branch tags encountered will trigger
warnings and will not be deleted.

So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
I'd like someone to double-check that though. Also maybe we should back
up the repository first?

regards, tom lane

Tom Lane wrote:

In the meantime, though, it's not quite clear why this would lead to
a buildfarm failure --- it should just mean a lot of extraneous files
appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
like btree_gist has some files that used to be autogenerated and are
now in CVS, so the bogusly new versions from CVS are suppressing the
desired generation from the old btree_num.c file.)

Looking at the Committers mail, it looks like there have only been two
very small commits since Michael's series of commits around 12 to 13.5
hours ago, and before that nothing since around 28 hours ago. Do we have
a backup snapshot of the repo taken during that time that we can roll
back to? That might be simpler than doing surgery on the repo.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

Looking at the Committers mail, it looks like there have only been two
very small commits since Michael's series of commits around 12 to 13.5
hours ago, and before that nothing since around 28 hours ago. Do we have
a backup snapshot of the repo taken during that time that we can roll
back to? That might be simpler than doing surgery on the repo.

It looks to me like the mistake was unrelated to any commit, and was
made at Aug 12 09:50 UTC. (Aside from the file-timestamp evidence,
caracara's build failure at 2007-08-12 213845 shows that the problem
is more than 2 days old.) I count seven commits of my own and several
of Michael's since then. If I'm right that a simple "cvs rtag" will fix
it, I'd rather do that than try to reapply patches.

regards, tom lane

Tom Lane wrote:

So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
I'd like someone to double-check that though.

I will test on a copy of my mirror.

Also maybe we should back
up the repository first?

Amen.

cheers

andrew

Andrew Dunstan wrote:

Tom Lane wrote:

So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
I'd like someone to double-check that though.

I will test on a copy of my mirror.

I copied the mirror, did a checkout from it, ran the command above in
the checked out version, then remove my checkout and checked out
REL7_4_STABLE and did a diff against my copy from the postgres repo that
hasn't been updated for some time. There were no significant
differences, and no extra files.

So I'd say your strategy looks good - backup and remove the phony tag.
I'd also say we should probably be logging tag commands in taginfo.
Presumably we mere mortal committers should not be doing any tagging
whatsoever, and tags should only be applied in preparation for releases.
Anyway, if we actually logged it we'd catch anything untoward early
rather than possibly days later.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

So I'd say your strategy looks good - backup and remove the phony tag.
I'd also say we should probably be logging tag commands in taginfo.
Presumably we mere mortal committers should not be doing any tagging
whatsoever, and tags should only be applied in preparation for releases.

+1 ... we should at least log such commands, and maybe disallow to anyone
except Marc's "pgsql" account. Particularly since they don't get
reported in pgsql-committers messages (or is that something we
can/should change?)

Meanwhile, is there anyone around who can either (1) tar up the
repository directory tree as root, or (2) confirm that a tarball
made by a non-root committer is sufficient?

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 22:20:16 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

+1 ... we should at least log such commands, and maybe disallow to anyone
except Marc's "pgsql" account. Particularly since they don't get
reported in pgsql-committers messages (or is that something we
can/should change?)

Meanwhile, is there anyone around who can either (1) tar up the
repository directory tree as root, or (2) confirm that a tarball
made by a non-root committer is sufficient?

I've just finished tar'ng up the cvs repository as root ... I believe one as a
non-root would be sufficient also, since it *should* retain file ownership /
permissions with the -p option ...

cvs# ls -lt cvsroot.aug14_2007.tar.bz2
- -rw-r--r-- 1 root wheel 62555651 Aug 15 02:52 cvsroot.aug14_2007.tar.bz2

cvs# du -sk cvsroot
414406 cvsroot

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwmq14QvfyHIvDvMRAsyFAJ9AND3JTDmOMR48Hl3Fdse2lVCjSwCglsu1
Pxj1LPPPx311rpY2XFj+Kvo=
=Gz6E
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy@hub.org> writes:

On Tuesday, August 14, 2007 22:20:16 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

Meanwhile, is there anyone around who can either (1) tar up the
repository directory tree as root, or (2) confirm that a tarball
made by a non-root committer is sufficient?

I've just finished tar'ng up the cvs repository as root ...

Great --- launching cvs rtag command now.

I believe one as a
non-root would be sufficient also, since it *should* retain file ownership /
permissions with the -p option ...

That's what I thought too, but a little extra paranoia doesn't hurt.

regards, tom lane

I wrote:

Great --- launching cvs rtag command now.

Done, and I got a plausible-looking mix of messages like
cvs rtag: Not removing branch tag `REL7_4_STABLE' from `/cvsroot/pgsql/src/tutorial/funcs_new.c,v'.
and a fresh checkout of REL7_4_STABLE now matches what I had locally.
So I think we walked away from that one.

TODO items:

* restrict, or at least log, cvs tag/rtag commands. Maybe report them
to pgsql-committers.

* I notice that the repository contains some files owned by
no-longer-existent accounts, eg 258 in contrib/btree_gist:

ls -la

total 460
drwxrwxr-x 6 scrappy dev 1024 Aug 15 03:00 .
drwxrwxr-x 90 scrappy dev 2048 Aug 15 03:00 ..
drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
-r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
-r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
-r--r--r-- 1 tgl dev 12033 Aug 15 03:00 btree_bit.c,v
-r--r--r-- 1 tgl dev 9902 Aug 15 03:00 btree_bytea.c,v
-r--r--r-- 1 tgl dev 10227 Aug 15 03:00 btree_cash.c,v
-r--r--r-- 1 tgl dev 11414 Aug 15 03:00 btree_date.c,v
-r--r--r-- 1 tgl dev 10211 Aug 15 03:00 btree_float4.c,v
-r--r--r-- 1 tgl dev 10074 Aug 15 03:00 btree_float8.c,v
-r--r--r-- 1 tgl dev 33915 Aug 15 03:00 btree_gist.c,v
-r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
-r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
-r--r--r-- 1 tgl dev 14717 Aug 15 03:00 btree_inet.c,v
-r--r--r-- 1 tgl dev 10234 Aug 15 03:00 btree_int2.c,v
-r--r--r-- 1 tgl dev 10221 Aug 15 03:00 btree_int4.c,v
-r--r--r-- 1 tgl dev 10235 Aug 15 03:00 btree_int8.c,v

I suppose this is due to Magnus' cleanup of unused committers accounts.
That was a good thing, but for security's sake these files ought to be
chown'd to some existing committer's account.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 23:26:03 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

I wrote:

Great --- launching cvs rtag command now.

Done, and I got a plausible-looking mix of messages like
cvs rtag: Not removing branch tag `REL7_4_STABLE' from
`/cvsroot/pgsql/src/tutorial/funcs_new.c,v'. and a fresh checkout of
REL7_4_STABLE now matches what I had locally. So I think we walked away from
that one.

TODO items:

* restrict, or at least log, cvs tag/rtag commands. Maybe report them
to pgsql-committers.

working on it ... have code for it, I've just spent so much time with CVSROOT =
/cvsroot, I have to relearn remote CVS :)

ls -la

total 460
drwxrwxr-x 6 scrappy dev 1024 Aug 15 03:00 .
drwxrwxr-x 90 scrappy dev 2048 Aug 15 03:00 ..
drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
-r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
-r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
-r--r--r-- 1 tgl dev 12033 Aug 15 03:00 btree_bit.c,v
-r--r--r-- 1 tgl dev 9902 Aug 15 03:00 btree_bytea.c,v
-r--r--r-- 1 tgl dev 10227 Aug 15 03:00 btree_cash.c,v
-r--r--r-- 1 tgl dev 11414 Aug 15 03:00 btree_date.c,v
-r--r--r-- 1 tgl dev 10211 Aug 15 03:00 btree_float4.c,v
-r--r--r-- 1 tgl dev 10074 Aug 15 03:00 btree_float8.c,v
-r--r--r-- 1 tgl dev 33915 Aug 15 03:00 btree_gist.c,v
-r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
-r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
-r--r--r-- 1 tgl dev 14717 Aug 15 03:00 btree_inet.c,v
-r--r--r-- 1 tgl dev 10234 Aug 15 03:00 btree_int2.c,v
-r--r--r-- 1 tgl dev 10221 Aug 15 03:00 btree_int4.c,v
-r--r--r-- 1 tgl dev 10235 Aug 15 03:00 btree_int8.c,v

I suppose this is due to Magnus' cleanup of unused committers accounts.
That was a good thing, but for security's sake these files ought to be
chown'd to some existing committer's account.

I can do a quick chown -R scrappy on the whole repository ... ok?

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwnf34QvfyHIvDvMRAnGTAJ43gELVZ5Q5SpV/jppR9rODB/ILaACeKCEN
DE16fnY/PmKlrvdHFaJeIKI=
=VCFu
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 23:26:03 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

* restrict, or at least log, cvs tag/rtag commands. Maybe report them
to pgsql-committers.

It should be done ... if you try and create a tag, it should generate an error
message ... not sure of the best way to test that it actually works, since, if
it doesn't work, of course, you'll add on another tag, but do you want to try
and create a 'fake tag' and then delete it afterwards?

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwnoO4QvfyHIvDvMRArHSAJ4qYuexUGn6Ff3/PkHU/DjJBlCQnwCgxyy9
Qc852wMVRRMYdwoD3+p7X3Y=
=FCD9
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy@hub.org> writes:

That was a good thing, but for security's sake these files ought to be
chown'd to some existing committer's account.

I can do a quick chown -R scrappy on the whole repository ... ok?

Seems close enough, but please keep that tarball around for awhile just
in case we need to look at the prior state.

regards, tom lane

"Marc G. Fournier" <scrappy@hub.org> writes:

It should be done ... if you try and create a tag, it should generate
an error message ...

Uh, nope:

$ cvs tag fooey README
T README
$ cvs log README | more

RCS file: /cvsroot/pgsql/src/backend/optimizer/README,v
Working file: README
head: 1.39
branch:
locks: strict
access list:
symbolic names:
fooey: 1.39
REL7_3_19: 1.25
REL7_4_17: 1.29
REL8_0_13: 1.30
REL8_1_9: 1.33
REL8_2_4: 1.35.2.1
REL8_0_12: 1.30
REL8_1_8: 1.33
REL8_2_3: 1.35
REL7_3_18: 1.25
REL7_4_16: 1.29
REL8_0_11: 1.30
REL8_1_7: 1.33
REL8_2_2: 1.35
REL8_0_10: 1.30
cvs [log aborted]: received broken pipe signal
$ cvs tag -d fooey README
D README
$

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

'k, let me play with it ...

- --On Wednesday, August 15, 2007 00:13:11 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

It should be done ... if you try and create a tag, it should generate
an error message ...

Uh, nope:

$ cvs tag fooey README
T README
$ cvs log README | more

RCS file: /cvsroot/pgsql/src/backend/optimizer/README,v
Working file: README
head: 1.39
branch:
locks: strict
access list:
symbolic names:
fooey: 1.39
REL7_3_19: 1.25
REL7_4_17: 1.29
REL8_0_13: 1.30
REL8_1_9: 1.33
REL8_2_4: 1.35.2.1
REL8_0_12: 1.30
REL8_1_8: 1.33
REL8_2_3: 1.35
REL7_3_18: 1.25
REL7_4_16: 1.29
REL8_0_11: 1.30
REL8_1_7: 1.33
REL8_2_2: 1.35
REL8_0_10: 1.30
cvs [log aborted]: received broken pipe signal
$ cvs tag -d fooey README
D README
$

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwn9q4QvfyHIvDvMRAuqOAJwPcIcsfQ74wBwdjBR478C72nr+bwCfd8PY
EeH2sh+biTfoFbTF2KxqYVU=
=iMwZ
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

One better (and done now):

find . -nouser -exec chown scrappy {} \;

so, all files that had valid users owning them stay with those ...

- --On Wednesday, August 15, 2007 00:08:07 -0400 Tom Lane <tgl@sss.pgh.pa.us>
wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

That was a good thing, but for security's sake these files ought to be
chown'd to some existing committer's account.

I can do a quick chown -R scrappy on the whole repository ... ok?

Seems close enough, but please keep that tarball around for awhile just
in case we need to look at the prior state.

regards, tom lane

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwonj4QvfyHIvDvMRAhNBAJ9b6Y1o4eICLrtuWncRIqN8g07XggCgnnc8
ttJoOScGbFEY/y3qP3MZYWY=
=RUO5
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy@hub.org> writes:

One better (and done now):
find . -nouser -exec chown scrappy {} \;

[ eyeballs repository ... ] Check, that looks great from here.

regards, tom lane

Tom Lane wrote:

In the meantime, though, it's not quite clear why this would lead to
a buildfarm failure --- it should just mean a lot of extraneous files
appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
like btree_gist has some files that used to be autogenerated and are
now in CVS, so the bogusly new versions from CVS are suppressing the
desired generation from the old btree_num.c file.)

Funny enoughy, that's exactly the problem I saw on win32 when I rsynced
from a windows box and tried a local CVS checkout :-) Once I rsynced to
a local linux instead and checked out from there, things worked again :)

//Magnus