git: uh-oh

Robert Haas <robertmhaas@gmail.com> writes:

It appears that the git conversion of the CVS repository is seriously
screwed up. For example, if you look at this:

Um ... Magnus has not given any report that he's finished running
the conversion. What exactly are you looking at?

It's too bad that nobody noticed this sooner, but I'm glad I noticed
today rather than tomorrow.

We're not going to start using the git repository until everyone is
satisfied it's OK, both as to current contents and history.

regards, tom lane

On Tue, Aug 17, 2010 at 21:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

It appears that the git conversion of the CVS repository is seriously
screwed up.  For example, if you look at this:

Um ... Magnus has not given any report that he's finished running
the conversion.  What exactly are you looking at?

That's the previous conversion. The one that we used to verify that
things looked ok. Seems nobody caught this :S

The new migration looks similarly weird.

Does anybody with some more git-fu have any clue how this can be?

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

It's too bad that nobody noticed this sooner, but I'm glad I noticed
today rather than tomorrow.

We're not going to start using the git repository until everyone is
satisfied it's OK, both as to current contents and history.

Yeah..

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus@hagander.net> wrote:

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

It seems to me that what we'll need to do here is write a script to
look through the CVS history of each file and make sure that the
versions of that file which appear on each branch match the revs in
CVS in content, order, and the commit message associated with any
changes. However, that's not going to do get done today.

It's too bad that nobody noticed this sooner, but I'm glad I noticed
today rather than tomorrow.

We're not going to start using the git repository until everyone is
satisfied it's OK, both as to current contents and history.

Duh. But obviously no one's checked that carefully enough up until now.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Magnus Hagander <magnus@hagander.net> writes:

On Tue, Aug 17, 2010 at 21:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Um ... Magnus has not given any report that he's finished running
the conversion. �What exactly are you looking at?

That's the previous conversion. The one that we used to verify that
things looked ok. Seems nobody caught this :S

The new migration looks similarly weird.

Does anybody with some more git-fu have any clue how this can be?

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD. For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

regards, tom lane

On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus@hagander.net> wrote:

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

It seems to me that what we'll need to do here is write a script to
look through the CVS history of each file and make sure that the
versions of that file which appear on each branch match the revs in
CVS in content, order, and the commit message associated with any
changes.  However, that's not going to do get done today.

Yeah. Unless someone comes up with a good way to fix this, or even
better an explanation why it's actually ont broken and we're looking
at things wrong :D, I think we have no choice but aborting the
conversion for now and come back to it later.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Aug 17, 2010, at 2:29 PM, Magnus Hagander wrote:

On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus@hagander.net> wrote:

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

It seems to me that what we'll need to do here is write a script to
look through the CVS history of each file and make sure that the
versions of that file which appear on each branch match the revs in
CVS in content, order, and the commit message associated with any
changes. However, that's not going to do get done today.

Yeah. Unless someone comes up with a good way to fix this, or even
better an explanation why it's actually ont broken and we're looking
at things wrong :D, I think we have no choice but aborting the
conversion for now and come back to it later.

Can you post the cvs2svn command line used for conversion?

Regards,

David
--
David Christensen
End Point Corporation
david@endpoint.com

On Tue, Aug 17, 2010 at 3:29 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD.  For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD.  It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

It looks to me like the commit I referenced in my original email is a
manufactured merge commit that completely rewrites the tree while
asserting that it doesn't do any such thing.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

No. The repository itself has the same problem, or at least my clone
of it does. I have to say I am totally underwhelmed by the quality of
the CVS-to-git conversion tools I've seen so far. It's fine for Linus
to say that CVS will eat your data, but these tools were evidently
written with grossly inadequate error and sanity checks. Whatever
we've been using for the incremental conversions doesn't seem to think
it's a problem if the new commit it's pushing doesn't make the head of
the tree match CVS HEAD, which seeems like a pretty darn obvious thing
to check for, and this tool evidently can't follow branch history
properly.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

* Tom Lane <tgl@sss.pgh.pa.us> [100817 15:30]:

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD. For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

No, that git branch is definately strange. The commit Robert pointed
out is a merge commit.

But looking at your explanation of when similar commits with the same
message were committed, I'm guessng the "timestamp fudge factor" along
with the "look for same commit message" behaviour of Magnus's cvs2git
conversion is trying "too hard" to make "atomic" commits of non-atomic
commits.

If you use a git viewer that shows the fork/merge points, you can see
that there are lots of these little "common" commits that have been
"unified" onto multiple brances.

Magnus, can you check if you can reduce the time fudge?

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

BTW, those two "manufactured" commits seem to directly follow commits
into HEAD that added files that were later also added on the branch.
I dunno exactly how git represents that type of event, but maybe an
extra commit is needed? It'd be interesting to look into the cvs2git
source code to see exactly what causes it to add a commit message
like that.

regards, tom lane

On Tue, Aug 17, 2010 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

 It'd be interesting to look into the cvs2git
source code to see exactly what causes it to add a commit message
like that.

I vigorously agree.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Tue, Aug 17, 2010 at 21:35, David Christensen <david@endpoint.com> wrote:

On Aug 17, 2010, at 2:29 PM, Magnus Hagander wrote:

On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus@hagander.net> wrote:

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

It seems to me that what we'll need to do here is write a script to
look through the CVS history of each file and make sure that the
versions of that file which appear on each branch match the revs in
CVS in content, order, and the commit message associated with any
changes.  However, that's not going to do get done today.

Yeah. Unless someone comes up with a good way to fix this, or even
better an explanation why it's actually ont broken and we're looking
at things wrong :D, I think we have no choice but aborting the
conversion for now and come back to it later.

Can you post the cvs2svn command line used for conversion?

Sure:
cvs2git --options=/root/cvs2git.options

Attached is the options file.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 17, 2010 at 13:43, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Aug 17, 2010 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

 It'd be interesting to look into the cvs2git
source code to see exactly what causes it to add a commit message
like that.

I vigorously agree.

How sure are we that its not the cvs2svn step that is screwing it up?
I know way back when I tried to convert a cvs tree to svn it fell over
horribly. Course the same was true when we went from svn to git...
(due to how we organized things in svn mainly)

On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex@gmail.com> wrote:

How sure are we that its not the cvs2svn step that is screwing it up?

urp, I jumped to a conclusion while skimming the cvs2git.options file
Magnus posted. With all the references to svn and things like
"GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
like it converts to svn first and then to git... im not sure what it
does.

Robert Haas <robertmhaas@gmail.com> writes:

It looks to me like the commit I referenced in my original email is a
manufactured merge commit that completely rewrites the tree while
asserting that it doesn't do any such thing.

AFAICS, the commits in the 8.3 history *after* that point are sane;
at least there's one for each actual 8.3 commit in the CVS logs.
Before that point, though, the history shown for 8.3 seems to include
all HEAD commits as well. The merge commit is probably cleaning up
those incorrectly included HEAD commits.

I concur that we gotta abort the git conversion. This looks like it
might be a pretty simple bug in the converter, but we can't block
Postgres development while we look for it.

regards, tom lane

On Tue, Aug 17, 2010 at 01:57:02PM -0600 I heard the voice of
Alex Hunsaker, and lo! it spake thus:

On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex@gmail.com> wrote:

How sure are we that its not the cvs2svn step that is screwing it up?

urp, I jumped to a conclusion while skimming the cvs2git.options
file Magnus posted. With all the references to svn and things like
"GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
like it converts to svn first and then to git... im not sure what
it does.

It's not that it converts to svn, but that it's built on (/part of)
cvs2svn, so presumably a lot of the "figure out changesets and branch
membership" logic and the "get things in the shape svn wants" logic
are intertwined.

--
Matthew Fuller (MF4839) | fullermd@over-yonder.net
Systems/Network Administrator | http://www.over-yonder.net/~fullermd/
On the Internet, nobody can hear you scream.

Alex Hunsaker wrote:

On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex@gmail.com> wrote:

How sure are we that its not the cvs2svn step that is screwing it up?

urp, I jumped to a conclusion while skimming the cvs2git.options file
Magnus posted. With all the references to svn and things like
"GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
like it converts to svn first and then to git... im not sure what it
does.

cvs2git converts directly from CVS to git. There is no intermediate SVN
step. However, given that cvs2git is built on top of cvs2svn, it is
true that some subversion terminology appears in the configuration file
and even in some of the manufactured commit messages.

Michael
the cvs2svn/cvs2git maintainer

Tom Lane wrote:

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD. For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

Please don't panic :-)

The problem is that it is *impossible* to faithfully represent a CVS or
Subversion history with its ancestry information in a git repository (or
AFAIK any of the DVCS repositories). The reason is that CVS
fundamentally records the history of single files, and each file can
have a branching history that is incompatible with those of other files.
For example, in CVS, a file can be added to a branch after the branch
already exists, different files can be added to a branch from multiple
parent branches, and even more perverse things are allowed. The CVS
history can record this mish-mash (albeit with much ambiguity).

Git, on the other hand, fundamentally only records a single history that
is considered to apply to the entire source tree. If a commit is
created with more than one parent, git treats it as a merge and
implicitly assumes that all of the contents of all of the ancestor
commits of all of the parents have been merged into the new version of
the source tree.

See [1]http://softwareswirl.blogspot.com/2009/08/git-mercurial-and-bazaarsimplicity.html for more discussion of the impedance mismatch between the
branching model of CVS/Subversion vs. that of the DVCSs.

So let's take the simplest example: a branch BRANCH1 is created from
trunk commit T1, then some time later another FILE1 from trunk commit T3
is added to BRANCH1 in commit B4. How should this series of events be
represented in a git repository?

The "inclusive" possibility is to say that some content was merged from
trunk to BRANCH1, and therefore to treat B4 as a merge commit:

T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
\ \
B1 -- B2 -- B3 -- B4 BRANCH1

This is wrong because there might be other changes in T2 and T3 (besides
the addition of FILE1) that were *not* merged to BRANCH1.

The "exclusive" possibility is to ignore the fact that some of the
content of B4 came from trunk and to pretend that FILE1 just appeared
out of nowhere in commit B4 independent of the FILE1 in TRUNK:

T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
\
B1 -- B2 -- B3 -- B4 BRANCH1

This is also wrong, because it doesn't reflect the true lineage of FILE1.

Given the choice between two wrong histories, cvs2git uses the
"inclusive" style. The result is that the ancestors of B4 include not
only T0, T1, B1, B2, and B3 (as might be expected), but also T2 and T3.
The display in the website that was quoted [2] seems to mash all of the
ancestors together without showing the topology of the history, making
the result quite confusing. The true history looks more like this:

$ git log --oneline --graph REL8_3_10 master
[...]
| * 2a91f07 tag 8.3.10
| * eb1b49f Preliminary release notes for releases 8.4.3, 8.3
| * dcf9673 Use SvROK(sv) rather than directly checking SvTYP
| * 1194fb9 Update time zone data files to tzdata release 201
| * fdfd1ec Return proper exit code (3) from psql when ON_ERR
| * 77524a1 Backport fix from HEAD that makes ecpglib give th
| * 55391af Add missing space in example.
| * 982aa23 Require hostname to be set when using GSSAPI auth
| * cb58615 Update time zone data files to tzdata release 201
| * ebe1e29 When reading pg_hba.conf and similar files, do no
| * 5a401e6 Fix a couple of places that would loop forever if
| * 5537492 Make contrib/xml2 use core xml.c's error handler,
| * c720f38 Export xml.c's libxml-error-handling support so t
| * 42ac390 Make iconv work like other optional libraries for
| * b03d523 pgindent run on xml.c in 8.3 branch, per request
| * 7efcdaa Add missing library and include dir for XSLT in M
| * 6ab1407 Do not run regression tests for contrib/xml2 on M
| * fff18e6 Backpatch MSVC build fix for XSLT
| * 7ae09ef Fix numericlocale psql option when used with a nu
| * de92a3d Fix contrib/xml2 so regression test still works w
| * 80f81c3 This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | a08b04f Fix contrib/xml2 so regression test still works w
* | 0d69e0f It's clearly now pointless to do backwards compat
* | 4ad348c Buildfarm still unhappy, so I'll bet it's EACCES
* | 6e96e1b Remove xmlCleanupParser calls from contrib/xml2.
* | 5b65b67 add EPERM to the list of return codes to expect f
| * a4067b3 Remove xmlCleanupParser calls from contrib/xml2.
| * 91b76a4 Back-patch today's memory management fixups in co
| * 5e74f21 Back-patch changes of 2009-05-13 in xml.c's memor
| * 043041e This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | 98cc16f Fix up memory management problems in contrib/xml2
* | 17e1420 Second try at fsyncing directories in CREATE DATA
* | a350f70 Assorted code cleanup for contrib/xml2. No chang
* | 3524149 Update complex locale example in the documentatio
[...]

The left branch is master, the right branch is the one leading to
REL8_3_10. You can see that there are multiple merges from master to
the branch, presumably when new files from trunk were ported to the
branch. This is even easier to see using a graphical history browser
like gitk.

There are good arguments for both the "inclusive" and the "exclusive"
representation of history. The ideal would require a lot more
intelligence and better heuristics (and slow down the conversion
dramatically). But even the smartest conversion would still be wrong,
because git is simply incapable of representing an arbitrary CVS
history. The main practical result of the impedance mismatch is that it
will be more difficult to merge between branches that originated in CVS
(but that is no surprise!)

Michael
the cvs2svn/cvs2git maintainer

[1]: http://softwareswirl.blogspot.com/2009/08/git-mercurial-and-bazaarsimplicity.html
http://softwareswirl.blogspot.com/2009/08/git-mercurial-and-bazaarsimplicity.html

On Wed, Aug 18, 2010 at 08:25:45AM +0200, Michael Haggerty wrote:

So let's take the simplest example: a branch BRANCH1 is created from
trunk commit T1, then some time later another FILE1 from trunk commit T3
is added to BRANCH1 in commit B4. How should this series of events be
represented in a git repository?

<snip>

The "exclusive" possibility is to ignore the fact that some of the
content of B4 came from trunk and to pretend that FILE1 just appeared
out of nowhere in commit B4 independent of the FILE1 in TRUNK:

T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
\
B1 -- B2 -- B3 -- B4 BRANCH1

This is also wrong, because it doesn't reflect the true lineage of FILE1.

But the "true lineage" is not stored anywhere in CVS so I don't see why
you need to fabricate it for git. Sure, it would be really nice if you
could, but if you can't do it reliably, you may as well not do it at
all. What's the loss?

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Wed, Aug 18, 2010 at 08:25, Michael Haggerty <mhagger@alum.mit.edu> wrote:

Tom Lane wrote:

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD.  For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD.  It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

Please don't panic :-)

We're not panic'ing just yet :-)

The problem is that it is *impossible* to faithfully represent a CVS or
Subversion history with its ancestry information in a git repository (or
AFAIK any of the DVCS repositories).  The reason is that CVS
fundamentally records the history of single files, and each file can
have a branching history that is incompatible with those of other files.
 For example, in CVS, a file can be added to a branch after the branch
already exists, different files can be added to a branch from multiple
parent branches, and even more perverse things are allowed.  The CVS
history can record this mish-mash (albeit with much ambiguity).

It can. IIRC we have cleaned a couple of such things out.

<snip some good descriptions of how git works>

Given the choice between two wrong histories, cvs2git uses the
"inclusive" style.  The result is that the ancestors of B4 include not
only T0, T1, B1, B2, and B3 (as might be expected), but also T2 and T3.
 The display in the website that was quoted [2] seems to mash all of the
ancestors together without showing the topology of the history, making
the result quite confusing.  The true history looks more like this:

$ git log --oneline --graph REL8_3_10 master
[...]
| * 2a91f07 tag 8.3.10
| * eb1b49f Preliminary release notes for releases 8.4.3, 8.3
| * dcf9673 Use SvROK(sv) rather than directly checking SvTYP
| * 1194fb9 Update time zone data files to tzdata release 201
| * fdfd1ec Return proper exit code (3) from psql when ON_ERR
| * 77524a1 Backport fix from HEAD that makes ecpglib give th
| * 55391af Add missing space in example.
| * 982aa23 Require hostname to be set when using GSSAPI auth
| * cb58615 Update time zone data files to tzdata release 201
| * ebe1e29 When reading pg_hba.conf and similar files, do no
| * 5a401e6 Fix a couple of places that would loop forever if
| * 5537492 Make contrib/xml2 use core xml.c's error handler,
| * c720f38 Export xml.c's libxml-error-handling support so t
| * 42ac390 Make iconv work like other optional libraries for
| * b03d523 pgindent run on xml.c in 8.3 branch, per request
| * 7efcdaa Add missing library and include dir for XSLT in M
| * 6ab1407 Do not run regression tests for contrib/xml2 on M
| * fff18e6 Backpatch MSVC build fix for XSLT
| * 7ae09ef Fix numericlocale psql option when used with a nu
| * de92a3d Fix contrib/xml2 so regression test still works w
| *   80f81c3 This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | a08b04f Fix contrib/xml2 so regression test still works w
* | 0d69e0f It's clearly now pointless to do backwards compat
* | 4ad348c Buildfarm still unhappy, so I'll bet it's EACCES
* | 6e96e1b Remove xmlCleanupParser calls from contrib/xml2.
* | 5b65b67 add EPERM to the list of return codes to expect f
| * a4067b3 Remove xmlCleanupParser calls from contrib/xml2.
| * 91b76a4 Back-patch today's memory management fixups in co
| * 5e74f21 Back-patch changes of 2009-05-13 in xml.c's memor
| *   043041e This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | 98cc16f Fix up memory management problems in contrib/xml2
* | 17e1420 Second try at fsyncing directories in CREATE DATA
* | a350f70 Assorted code cleanup for contrib/xml2.  No chang
* | 3524149 Update complex locale example in the documentatio
[...]

The left branch is master, the right branch is the one leading to
REL8_3_10.  You can see that there are multiple merges from master to
the branch, presumably when new files from trunk were ported to the
branch.  This is even easier to see using a graphical history browser
like gitk.

Yeah, this is clearly the problem.

There are good arguments for both the "inclusive" and the "exclusive"
representation of history.  The ideal would require a lot more
intelligence and better heuristics (and slow down the conversion
dramatically).  But even the smartest conversion would still be wrong,
because git is simply incapable of representing an arbitrary CVS
history.  The main practical result of the impedance mismatch is that it
will be more difficult to merge between branches that originated in CVS
(but that is no surprise!)

Our requirements are simple: our cvs history is linear, the git
history should be linear. It is *not* the same commit that's on head
and the branch. They are two different commits, that happen to have
the same commit message and mostly the same content.

Bottom line is, we want zero merge commits in the git repository. We
may start using that sometime in the future (but for now, we've
decided we don't want that even in the future), but we most
*definitely* don't want it in the past. We don't care about
"representing the proper heritage of FILE1" in git, because we never
did in cvs.

Is there some way to make cvs2git work this way, and just not bother
even trying to create merge commits, or is that fundamentally
impossible and we need to look at another tool?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/