Re: PostgreSQL Developer meeting minutes up

Andrew Dunstan wrote:

[moving this onto -hackers, where I think it belongs]

Tom Lane wrote:

Huh? The buildfarm will only prove that HEAD of the active branches
builds. What the concern was was whether we could correctly extract
past states (particularly, but not solely, the tags corresponding to
releases) from a converted git repository. The testing I had in mind
was to check out various tags and diff that tree against actual release
tarballs.

It appears that our git repo is only picking up the branch tags (e.g.
REL8_0_STABLE) , not all the release tags (e.g. REL8_0_5) . That needs
to be fixed (if possible).

Hmm. I looked through the source of the import script. It appears to
mention tags here and there, but doesn't seem to do it. There is a
comment that reads:

# Previous CVS versions just added the tag to the current HEAD
# revision and didn't insert a dead revision on the branch with
# the same date, like it is happening now.
# This means history is unclear as we can't reliably determine
# if the tagging happened at the same time as the addition to
# the branch. For now, just assume it did.
#
# XXX can't reproduce for now, disabling, as it breaks some
# things
#

Basically, it comes down to cvs tags not being actual first class
happening, but just metadata on files.

I'm sure we could script the creation of these tags fairly reliably on
*our* repository since we know which files are always updated when a tag
is added. I'm thinking we could just parse the log for configure.in and
grab the tags from there. Thoughts?

--
Magnus Hagander
Self: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Again,

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken" (i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

Scripts to "fix" (actually, remove) the broken tags have also been
posted, along with requests that if somebody is "mucking" with the
actual repository, to make sure it's known about, and access is "denied"
during the mucking period (access being any rsync/anoncvs/mirroring of
the cvs root).

As long as the tags are broken, you aren't going to get the tags
imported.

If you're going to fix the tags, warn everybody (because most people
doing automatic conversions must know - they may need to be very
careful to avoid a full re-import), do it, and let us know when it's
done.

a.

* Andrew Dunstan <andrew@dunslane.net> [090526 10:41]:

[moving this onto -hackers, where I think it belongs]

Tom Lane wrote:

Huh? The buildfarm will only prove that HEAD of the active branches
builds. What the concern was was whether we could correctly extract
past states (particularly, but not solely, the tags corresponding to
releases) from a converted git repository. The testing I had in mind
was to check out various tags and diff that tree against actual release
tarballs.

It appears that our git repo is only picking up the branch tags (e.g.
REL8_0_STABLE) , not all the release tags (e.g. REL8_0_5) . That needs
to be fixed (if possible).

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Aidan Van Dyk <aidan@highrise.ca> writes:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken" (i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

Scripts to "fix" (actually, remove) the broken tags have also been
posted, along with requests that if somebody is "mucking" with the
actual repository, to make sure it's known about, and access is "denied"
during the mucking period (access being any rsync/anoncvs/mirroring of
the cvs root).

Up to now I've always been of the opinion that fixing those tags wasn't
worth taking any risk for. But if we are thinking of moving away from
CVS, then this clearly becomes one of the hurdles we have to jump on the
way. Can you refresh our memory about which tags are problematic and
exactly what needs to be done about 'em?

regards, tom lane

* Tom Lane <tgl@sss.pgh.pa.us> [090526 11:20]:

Aidan Van Dyk <aidan@highrise.ca> writes:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken" (i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

Scripts to "fix" (actually, remove) the broken tags have also been
posted, along with requests that if somebody is "mucking" with the
actual repository, to make sure it's known about, and access is "denied"
during the mucking period (access being any rsync/anoncvs/mirroring of
the cvs root).

Up to now I've always been of the opinion that fixing those tags wasn't
worth taking any risk for. But if we are thinking of moving away from
CVS, then this clearly becomes one of the hurdles we have to jump on the
way. Can you refresh our memory about which tags are problematic and
exactly what needs to be done about 'em?

Specifically, it's 2 tags, and I just remove them:
REL7_1_BETA2
REL7_1_BETA3

Previous threads:
http://news.gmane.org/find-root.php?message_id=20080220225300.GE16099@yugib.highrise.ca
http://news.gmane.org/find-root.php?message_id=20081229155140.GP12094@yugib.highrise.ca

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Tom Lane wrote:

Aidan Van Dyk <aidan@highrise.ca> writes:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken" (i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

Scripts to "fix" (actually, remove) the broken tags have also been
posted, along with requests that if somebody is "mucking" with the
actual repository, to make sure it's known about, and access is "denied"
during the mucking period (access being any rsync/anoncvs/mirroring of
the cvs root).

Up to now I've always been of the opinion that fixing those tags wasn't
worth taking any risk for. But if we are thinking of moving away from
CVS, then this clearly becomes one of the hurdles we have to jump on the
way. Can you refresh our memory about which tags are problematic and
exactly what needs to be done about 'em?

I think we need just to remove the two tags in question (they have long
been irrelevant). Prudence suggests that we should do that some time
(weeks, I think) after the 8.4 release, when reverting ,if we find any
breakage, won't be too painful.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

Tom Lane wrote:

Up to now I've always been of the opinion that fixing those tags wasn't
worth taking any risk for. But if we are thinking of moving away from
CVS, then this clearly becomes one of the hurdles we have to jump on the
way.

I think we need just to remove the two tags in question (they have long
been irrelevant). Prudence suggests that we should do that some time
(weeks, I think) after the 8.4 release, when reverting ,if we find any
breakage, won't be too painful.

I don't see a lot of point in waiting till after 8.4.0. There is no
time, ever, where we are sure there will be no release for weeks ---
a security or data-loss bug could crop up at any time. And not messing
up back branch update releases is even more important than not messing
up 8.4.0, because the back branches are much more likely to get dropped
straight into production.

Obviously we want a solid backup of the pre-modification CVS repository,
and we have to follow Aidan's advice about synchronizing the change with
mirror repositories, but I don't see a strong argument for waiting weeks
to do this. I think we should get it over with, so people can get on
with the work that it's blocking.

regards, tom lane

Aidan Van Dyk <aidan@highrise.ca> writes:

Specifically, it's 2 tags, and I just remove them:
REL7_1_BETA2
REL7_1_BETA3

Previous threads:
http://news.gmane.org/find-root.php?message_id=20080220225300.GE16099@yugib.highrise.ca
http://news.gmane.org/find-root.php?message_id=20081229155140.GP12094@yugib.highrise.ca

It looks like the ill-considered commit message mentioned in that first
thread hasn't been dealt with, either.

regards, tom lane

On Tue, 26 May 2009, Aidan Van Dyk wrote:

* Tom Lane <tgl@sss.pgh.pa.us> [090526 11:20]:

Aidan Van Dyk <aidan@highrise.ca> writes:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken" (i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

Scripts to "fix" (actually, remove) the broken tags have also been
posted, along with requests that if somebody is "mucking" with the
actual repository, to make sure it's known about, and access is "denied"
during the mucking period (access being any rsync/anoncvs/mirroring of
the cvs root).

Up to now I've always been of the opinion that fixing those tags wasn't
worth taking any risk for. But if we are thinking of moving away from
CVS, then this clearly becomes one of the hurdles we have to jump on the
way. Can you refresh our memory about which tags are problematic and
exactly what needs to be done about 'em?

Specifically, it's 2 tags, and I just remove them:
REL7_1_BETA2
REL7_1_BETA3

So, you are suggesting:

cvs -q tag -d REL7_1_BETA2 .
cvs -q tag -d REL7_1_BETA3 .

correct?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

On Tuesday 26 May 2009 17:44:59 Magnus Hagander wrote:

Hmm. I looked through the source of the import script. It appears to
mention tags here and there, but doesn't seem to do it.

Which is part of the reason we use this script and not one of the other ones,
because some of our tags are broken. Any conversion will either have to drop
or repair some tags; see my previous posts to hackers about this.

Hi,

Quoting "Aidan Van Dyk" <aidan@highrise.ca>:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken"

Please keep in mind that the amount of "brokenness" here depends a lot
on the tool used for the conversion to git. AFAIK 'git cvsimport' is
used for the conversion of the Postgres repository to git.
git-cvsimport uses cvsps, which is known for its deficiencies.

(i.e they
are such that it's impossible to actually create all the tags), so the
git "cvsimport" tools that try to tags all croak on the PG CVS repository.

The tool which doesn't croak doesn't try and import all the tags, just
the sticky "branch tags"...

I cannot confirm that assertion. I've just tried with cvs2svn, which
converts the Postgres repository just fine, including 170 tags, among
them REL7_1_BETA2 and REL7_1_BETA3. A quick glance at the resulting
checkouts' diff looks pretty good as well.

I consider cvsps to be lacking rather than blaming the Postgres CVS
repository. (Of course that doesn't mean the Postgres CVS repository
is perfectly self-consistent - CVS repositories aren't by definition.
I'm just pointing out that there are tools with better heuristics than
those of cvsps.)

Has anybody ever tried using cvs2git? Being based on cvs2svn, it
should yield better results than cvsps. It's even recommended from the
issues section of the git-cvsimport man page [1]. And git-cvsimport
seems to be able to continue from an initial import with cvs2git.

Regards

Markus Wanner

Hi,

Quoting "Peter Eisentraut" <peter_e@gmx.net>:

.. some of our tags are broken. Any conversion will either have to drop
or repair some tags; see my previous posts to hackers about this.

Can you please point me to that post? I didn't find it.

In what way do you consider the tags "broken"? (As CVS does not
guarantee any inter-file consistency, I don't think one can speak of
brokenness at all. IMO it's rather just a matter of how to convert to
another VCS's representation. Certainly, it's not always possible to
convert without any kind of information loss. However, as just pointed
out, there are certainly differences between converters).

Regards

Markus Wanner

Markus Wanner wrote:

Hi,

Quoting "Aidan Van Dyk" <aidan@highrise.ca>:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken"

Please keep in mind that the amount of "brokenness" here depends a lot
on the tool used for the conversion to git. AFAIK 'git cvsimport' is
used for the conversion of the Postgres repository to git. git-cvsimport
uses cvsps, which is known for its deficiencies.

No, we use fromcvs, not "git cvsimport".

IIRC that was the only one people could make working with incremental
updates.

--
Magnus Hagander
Self: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:

Markus Wanner wrote:

Quoting "Aidan Van Dyk" <aidan@highrise.ca>:

This has been raised and ignored many times before on -hackers... The
reason is because the tags in the CVS repository are "broken"

Please keep in mind that the amount of "brokenness" here depends a lot
on the tool used for the conversion to git. AFAIK 'git cvsimport' is
used for the conversion of the Postgres repository to git. git-cvsimport
uses cvsps, which is known for its deficiencies.

No, we use fromcvs, not "git cvsimport".

IIRC that was the only one people could make working with incremental
updates.

Right. When I looked at the converters last time, there was others that
produce a better conversion, but they didn't work incrementally. If
we're going to switch over the main repository, we should look at the
alternatives.

OTOH, there's some value in staying with current GIT repository. In
EnterpriseDB, we maintain all the Oracle-compatibility stuff in a GIT
repository that's based on the PostgreSQL mirror. If PostgreSQL switches
to a new GIT repository/mirror, I'll have to rebase all that, and I'm
not sure how well that works with all the merges and stuff. I'm probably
the one with most complex situation, but others who have
work-in-progress patches in local repositories will face the same issue
at a smaller scale.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Hi,

Quoting "Magnus Hagander" <magnus@hagander.net>:

No, we use fromcvs, not "git cvsimport".

Oh, thanks for the correction.

I haven't heard of fromcvs before, but solely judging by lines of
code, it's hardly as elaborate as cvs2svn. So my arguments hold true
for it as well.

IIRC that was the only one people could make working with incremental
updates.

Understood.

Regards

Markus Wanner

On Wednesday 27 May 2009 13:25:14 Markus Wanner wrote:

Hi,

Quoting "Peter Eisentraut" <peter_e@gmx.net>:

.. some of our tags are broken. Any conversion will either have to drop
or repair some tags; see my previous posts to hackers about this.

Can you please point me to that post? I didn't find it.

http://archives.postgresql.org/pgsql-hackers/2008-12/msg01879.php

In what way do you consider the tags "broken"?

The tag applies to different commits on different files.

(As CVS does not
guarantee any inter-file consistency, I don't think one can speak of
brokenness at all.

Just because CVS doesn't guarantee it, it doesn't mean it's not broken.
Otherwise, any possible random permutation of files would be a non-broken
checkout by your definition.

On Wednesday 27 May 2009 00:54:52 Marc G. Fournier wrote:

So, you are suggesting:

cvs -q tag -d REL7_1_BETA2 .
cvs -q tag -d REL7_1_BETA3 .

Note that there are actually two different issues related to tags:

One is, the tags REL7_1_BETA2 and REL7_1_BETA3 cannot be parsed by cvsps. But
no one has analyzed why that is. Nor is there any proof that they are wrong
or broken.

The other is, the tag REL7_1 produces different files than were actually in
the release. cvsps warns about this. I had posted a patch to fix this.

* Markus Wanner <markus@bluegap.ch> [090527 06:20]:

Has anybody ever tried using cvs2git? Being based on cvs2svn, it should
yield better results than cvsps. It's even recommended from the issues
section of the git-cvsimport man page [1]. And git-cvsimport seems to be
able to continue from an initial import with cvs2git.

cvs2svn (and hence cvs2git certainly has *oodles* of code explicitly to
try and deal with "weird" (non-linear) cvs histories (a la type of the
PG repo)... I'm not sure I would take the reference to "the old cvs2git
tool" to be the cvs2git that's currently active and based on cvs2svn...
If anybody can confirm that the incremental git cvsimport can follow a
recent cvs2git conversion, that would definitely be awesome! If I can
come across a few free hours some time, I might even try it myself!

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

* Marc G. Fournier <scrappy@hub.org> [090526 18:00]:

Specifically, it's 2 tags, and I just remove them:
REL7_1_BETA2
REL7_1_BETA3

So, you are suggesting:

cvs -q tag -d REL7_1_BETA2 .
cvs -q tag -d REL7_1_BETA3 .

correct?

Not directly, I claim *no* knowledge of the safety of any CVS commands
;-)

But whatever you do, please, lock us our of the repository first (by us,
I mean any publc access to it, via anoncvs, rsync, mirror, anything),
notify us first, (give us at *least* day warning so we can disable any
automatic conversions), and let us know when it's done, so we can watch
the first run after any conversion carefully.

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

* Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> [090527 07:29]:

OTOH, there's some value in staying with current GIT repository. In
EnterpriseDB, we maintain all the Oracle-compatibility stuff in a GIT
repository that's based on the PostgreSQL mirror. If PostgreSQL switches
to a new GIT repository/mirror, I'll have to rebase all that, and I'm
not sure how well that works with all the merges and stuff. I'm probably
the one with most complex situation, but others who have
work-in-progress patches in local repositories will face the same issue
at a smaller scale.

But there are oodles of options in git available to handle a cutover
like that:
- grafts
- filter-branch
- rebase (the new rebase toolset can even attempt to rebase a DAG onto an
existing DAG, not just linear patches))

So I'm whatever becomes the "official" git repo can simply be "grafted"
into your history, your your new development grafted on to the
"official" history...

But, if there is nothing wrong with the current repo (except that it
doesn't have tags), than we can easily add tags to it...

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.