Hacking on PostgreSQL via GIT

Martin Langhoff wrote:

Hi Florian,

I am right now running an rsync of the Pg CVS repo to my work machine to
get a git import underway. I'm rather keen on seeing your cool PITR Pg
project go well and I have some git+cvs fu I can apply here (being one
of the git-cvsimport maintainers) ;-)

For the kind of work you'll be doing (writing patches that you'll want
to be rebasing onto the latest HEAD for merging later) git is probably
the best tool. That's what I use it for... tracking my experimental /
custom branches of projects that use CVS or SVN :-)

Initially, I'll post it on http://git.catalyst.net.nz/ and I can run a
daily import for you - once that's in place you can probably get a repo
with your work on http://repo.or.cz/

Well, now that more than one of us are working with git on PostgreSQL...

I've had a repo conversion running for a while... I've only got it to what
I consider "stable" last week:
http://repo.or.cz/w/PostgreSQL.git
git://repo.or.cz/PostgreSQL.git

Note that this is a "special" conversion - I intentionally "unmunge" all the
$PostgreSQL$ tags in this repo.

I hate the Keyword expansion, and it only servers to make otherwise
automatically merging a manual process. So I specifically go through and
un-munge any keyword a-like things before stomping it into GIT.

For those interested int he conversion process, I've used a slightly
modified version of fromcvs (A ruby cvs to git/Hg tool), and it runs on all
of pgsql in about 20 minutes.

I gave up on git-svn (because of both speed and my in-ablility to
easy "filter" out Keywords, etc) and git-cvsimport (because cvsps doesn't
seem to like pgsql's repo)

I "update" the git repo daily, based on an anonymous rsync of the cvsroot.
If the anon-rsync is updated much more frequently, and people think my git
conversion should match it, I have no problem having cron run it more than
daily.

Also - note that I give *no* guarentees of it's integrity, etc.

I've "diffed" a CVS checkout and a git checkout, and the are *almost*
identical. Almost, because it seems like my git repository currently has 3
files that a cvs checkout doesn't:
backend/parser/gram.c |12088 +++++++++++++++++++++++++++
interfaces/ecpg/preproc/pgc.c | 2887 ++++++
interfaces/ecpg/preproc/preproc.c |16988 ++++++++++++++++++++++++++++++++++

And at this point, I haven't been bothered to see where those files came
from (and where they dissapear) in CVS and why my import isn't picking that
up... I could probably be pushed if others find this repo really useful,
but those files problematic...

* Aidan Van Dyk <aidan@highrise.ca> [070416 14:08]:

Note that this is a "special" conversion - I intentionally "unmunge" all the
$PostgreSQL$ tags in this repo.

Blah - and I just noticed that I actually "missed" the $PostgreSQL$
(although I did catch the Date/Modified/From/etc)...

I hate the Keyword expansion, and it only servers to make otherwise
automatically merging a manual process. So I specifically go through and
un-munge any keyword a-like things before stomping it into GIT.

Expect it to change in the next little while once more ;-)

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Martin Langhoff wrote:

Hi Florian,

I am right now running an rsync of the Pg CVS repo to my work machine to
get a git import underway. I'm rather keen on seeing your cool PITR Pg
project go well and I have some git+cvs fu I can apply here (being one
of the git-cvsimport maintainers) ;-)

Cool - I'm new to git, so I really appreciate any help that I can get.

For the kind of work you'll be doing (writing patches that you'll want
to be rebasing onto the latest HEAD for merging later) git is probably
the best tool. That's what I use it for... tracking my experimental /
custom branches of projects that use CVS or SVN :-)

Thats how I figured I'd work - though I don't yet understand what
the advantage of "rebase" is over "merge".

Currently, I've setup a git repo that pulls in the changes from the SVN
repo, and pushed them to my main soc git repo. On that main repo I have
two branches, master and pgsql-head, and I call "cg-merge pgsql-head"
if I want to merge with CVS HEAD.

Initially, I'll post it on http://git.catalyst.net.nz/ and I can run a
daily import for you - once that's in place you can probably get a repo
with your work on http://repo.or.cz/

Having a git mirror of the pgsql CVS would be great.
BTW, I've just check out repo.or.cz, and noticed that there is already a
git mirror of the pgsql CVS: http://repo.or.cz/w/PostgreSQL.git

greetings + thanks
Florian Pflug

Aidan Van Dyk wrote:

Martin Langhoff wrote:
Well, now that more than one of us are working with git on PostgreSQL...

I've had a repo conversion running for a while... I've only got it to what
I consider "stable" last week:
http://repo.or.cz/w/PostgreSQL.git
git://repo.or.cz/PostgreSQL.git

Ah - thats what I just stumbled over ;-)

For those interested int he conversion process, I've used a slightly
modified version of fromcvs (A ruby cvs to git/Hg tool), and it runs on all
of pgsql in about 20 minutes.

I gave up on git-svn (because of both speed and my in-ablility to
easy "filter" out Keywords, etc) and git-cvsimport (because cvsps doesn't
seem to like pgsql's repo)

Yeah, git-cvsimport didn't work for me either...

I "update" the git repo daily, based on an anonymous rsync of the cvsroot.
If the anon-rsync is updated much more frequently, and people think my git
conversion should match it, I have no problem having cron run it more than
daily.

Also - note that I give *no* guarentees of it's integrity, etc.

I've "diffed" a CVS checkout and a git checkout, and the are *almost*
identical. Almost, because it seems like my git repository currently has 3
files that a cvs checkout doesn't:
backend/parser/gram.c |12088 +++++++++++++++++++++++++++
interfaces/ecpg/preproc/pgc.c | 2887 ++++++
interfaces/ecpg/preproc/preproc.c |16988 ++++++++++++++++++++++++++++++++++

And at this point, I haven't been bothered to see where those files came
from (and where they dissapear) in CVS and why my import isn't picking that
up... I could probably be pushed if others find this repo really useful,
but those files problematic...

Thats interesting - the SVN mirror of the pgsql CVS at
http://projects.commandprompt.com/public/pgsql/browser
has exactly the same problem with those 3 files, as I found out the hard way ;-)

In the case of pgc.c, I've compared that revisions in CVS with the one in
SVN. SVN include the cvs-version 1.5 if this file in trunk, which seems to
be the last version of that file in CVS HEAD. Interestingly,
http://developer.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/ecpg/preproc/Attic/pgc.c
shows no trace of the file being deleted from HEAD either - it just shows
that it was removed from WIN32_DEV. But still a CVS checkout doesn't include
that file...

Since 3 tools (cvsweb, git-cvsimport and whatever commandprompt uses to create
the SVN mirror) all come to the same conclusion regarding this file, I think
that this is caused by some corruption of the CVS repository - but I don't have
the cvs-fu to debug this...

greetings, Florian Pflug

Aidan Van Dyk wrote:

I've "diffed" a CVS checkout and a git checkout, and the are *almost*
identical. Almost, because it seems like my git repository currently has 3
files that a cvs checkout doesn't:
backend/parser/gram.c |12088 +++++++++++++++++++++++++++
interfaces/ecpg/preproc/pgc.c | 2887 ++++++
interfaces/ecpg/preproc/preproc.c |16988 ++++++++++++++++++++++++++++++++++

And at this point, I haven't been bothered to see where those files came
from (and where they dissapear) in CVS and why my import isn't picking that
up... I could probably be pushed if others find this repo really useful,
but those files problematic...

These files are generated (from gram.y, pgc.l and preproc.y
respectievly) and are not present in the CVS repo, though I think they
have been at some point.

It's strange that other generated files (that have also been in the repo
in the past) like preproc.h are not showing up.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:

These files are generated (from gram.y, pgc.l and preproc.y
respectievly) and are not present in the CVS repo, though I think they
have been at some point.

It's strange that other generated files (that have also been in the repo
in the past) like preproc.h are not showing up.

The weird thing about these files is that the CVS history shows commits
on HEAD later than the file removal commit. I don't recall if Vadim
unintentionally re-added the files before making those commits ... but
if he did, you'd think it'd have taken another explicit removal to get
rid of them in HEAD. More likely, there was some problem in his local
tree that allowed a "cvs commit" to think it should update the
repository with copies of the derived files he happened to have.

I think this is a corner case that CVS handles in a particular way and
the tools people are using to read the repository handle in a different
way. Which would be a bug in those tools, since CVS's interpretation
must be right by definition.

regards, tom lane

Tom Lane wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

These files are generated (from gram.y, pgc.l and preproc.y
respectievly) and are not present in the CVS repo, though I think they
have been at some point.

It's strange that other generated files (that have also been in the repo
in the past) like preproc.h are not showing up.

The weird thing about these files is that the CVS history shows commits
on HEAD later than the file removal commit. I don't recall if Vadim
unintentionally re-added the files before making those commits ... but
if he did, you'd think it'd have taken another explicit removal to get
rid of them in HEAD. More likely, there was some problem in his local
tree that allowed a "cvs commit" to think it should update the
repository with copies of the derived files he happened to have.

I think this is a corner case that CVS handles in a particular way and
the tools people are using to read the repository handle in a different
way. Which would be a bug in those tools, since CVS's interpretation
must be right by definition.

The question is if it'd be acceptable to manually remove that last commit
from the repository. I guess simply readding, and then removing the files
again should do the trick, though I'd be cleaner to fix remove the
offending commit in the first place. Should postgres ever decide to switch
to another version control system (which I don't advocate), that'd be
one obstacle less to deal with...

Or is the risk of causing breakage too high?

greetings, Florian Pflug

aidan@highrise.ca (Aidan Van Dyk) writes:

I've "diffed" a CVS checkout and a git checkout, and the are *almost*
identical. Almost, because it seems like my git repository currently has 3
files that a cvs checkout doesn't:
backend/parser/gram.c |12088 +++++++++++++++++++++++++++
interfaces/ecpg/preproc/pgc.c | 2887 ++++++
interfaces/ecpg/preproc/preproc.c |16988 ++++++++++++++++++++++++++++++++++

And at this point, I haven't been bothered to see where those files came
from (and where they dissapear) in CVS and why my import isn't picking that
up... I could probably be pushed if others find this repo really useful,
but those files problematic...

Those three files are normally generated by either flex or bison
(gram.c depends on gram.y, pgc.c on pgc.l, and preproc.c on
preproc.y); I'd suggest removing those three files from your git
repository.
--
"cbbrowne","@","acm.org"
http://cbbrowne.com/info/rdbms.html
"They laughed at Columbus, they laughed at Fulton, they laughed at the
Wright brothers. But they also laughed at Bozo the Clown."
-- Carl Sagan

fgp@phlo.org ("Florian G. Pflug") writes:

Martin Langhoff wrote:

Hi Florian,
I am right now running an rsync of the Pg CVS repo to my work
machine to
get a git import underway. I'm rather keen on seeing your cool PITR Pg
project go well and I have some git+cvs fu I can apply here (being one
of the git-cvsimport maintainers) ;-)

Cool - I'm new to git, so I really appreciate any help that I can get.

For the kind of work you'll be doing (writing patches that you'll want
to be rebasing onto the latest HEAD for merging later) git is probably
the best tool. That's what I use it for... tracking my experimental /
custom branches of projects that use CVS or SVN :-)

Thats how I figured I'd work - though I don't yet understand what
the advantage of "rebase" is over "merge".

Currently, I've setup a git repo that pulls in the changes from the SVN
repo, and pushed them to my main soc git repo. On that main repo I have
two branches, master and pgsql-head, and I call "cg-merge pgsql-head"
if I want to merge with CVS HEAD.

Initially, I'll post it on http://git.catalyst.net.nz/ and I can run a
daily import for you - once that's in place you can probably get a repo
with your work on http://repo.or.cz/

Having a git mirror of the pgsql CVS would be great.
BTW, I've just check out repo.or.cz, and noticed that there is already a
git mirror of the pgsql CVS: http://repo.or.cz/w/PostgreSQL.git

This strikes me as being a really super thing, having both Subversion
and Git repositories publicly available that are tracking the
PostgreSQL sources.

Stepping back to the SCM discussion, people were interested in finding
out what merits there were in having these sorts of SCMs, and in
finding out what glitches people might discover (e.g. - like the files
where the CVS repository is a bit schizophrenic as to whether they are
still there or not...). Having these repositories should allow some
of this experimentation to take place now.

I'd be interested in fiddling with a Git repository, at some point;
I'll happily wait a bit to start drawing from one of these existing
ones, to let the dust settle and to let things stabilize a bit.
--
(reverse (concatenate 'string "moc.enworbbc" "@" "enworbbc"))
http://linuxdatabases.info/info/emacs.html
"Support your local medical examiner - die strangely."
-- Blake Bowers

* Florian G. Pflug <fgp@phlo.org> [070416 16:16]:

I think this is a corner case that CVS handles in a particular way and
the tools people are using to read the repository handle in a different
way. Which would be a bug in those tools, since CVS's interpretation
must be right by definition.

;-)

Would anyone know if these were "hand moved" to Attic? For instance, I
*can't* seem to get non-dead files into Attic, no matter what I try with
my cvs (on debian). But I haven't gone through the last 8 years of
CVS's CVS logs to see if they fixed a bug in the cvs server code that
would allow a non-dead HEAD rcs to be in the Attic...

The question is if it'd be acceptable to manually remove that last commit
from the repository. I guess simply readding, and then removing the files
again should do the trick, though I'd be cleaner to fix remove the
offending commit in the first place. Should postgres ever decide to switch
to another version control system (which I don't advocate), that'd be
one obstacle less to deal with...

Or is the risk of causing breakage too high?

Well, I've "hand fixed" this in my conversion process so my git
conversion should not have this problem...

I'm not a fan of mucking around by hand in CVS. It's only because of
the short comings of CVS that it's necessary to every resort to that.
So I don't think re-adding/deleting it is worth it...

I've updated the repo.or.cz/PostgreSQL.git again - and this time it
should be pretty good. Consider it "usable" to clone off and follow CVS
development with... I won't re-convert the whole thing again, and will
just provide daily updates to it now. Unless anybody finds issues with
it...

Ignore the "public" branch in there - that got in in an errant push, and
I don't know how to remove branches on repo.or.cz. I'm now just putting
"conversion notes" up in the public branch... IT's *not* a PostgreSQL
branch.

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Aidan Van Dyk <aidan@highrise.ca> writes:

Would anyone know if these were "hand moved" to Attic?

Seems unlikely, since there's a commit log entry for the removal. But
this all happened seven-plus years ago and I'm sure there's an old CVS
bug involved *somewhere*.

I like the idea of re-adding and then re-removing the files on HEAD.
Does anyone think that poses any real risk?

regards, tom lane

* Tom Lane <tgl@sss.pgh.pa.us> [070416 19:03]:

Aidan Van Dyk <aidan@highrise.ca> writes:

Would anyone know if these were "hand moved" to Attic?

Seems unlikely, since there's a commit log entry for the removal. But
this all happened seven-plus years ago and I'm sure there's an old CVS
bug involved *somewhere*.

I like the idea of re-adding and then re-removing the files on HEAD.
Does anyone think that poses any real risk?

No - it even fixed the "hand moved" test I had done trying to create an
Attic with, when trying to figure out how they got that way in the first
place...

What I did when I converted the repo was just hand edit those files to
have a state of "dead" to match their position in Attic for those RCS
revs. If you "add" them and remove them, I believe my GIT conversion
will actually "follow" that correctly... If not - I just rm -Rf it and
let it go from scratch "one more time"... I'm glad computers are good
at that type of repetitive task...

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Aidan Van Dyk <aidan@highrise.ca> writes:

* Tom Lane <tgl@sss.pgh.pa.us> [070416 19:03]:

I like the idea of re-adding and then re-removing the files on HEAD.
Does anyone think that poses any real risk?

No - it even fixed the "hand moved" test I had done trying to create an
Attic with, when trying to figure out how they got that way in the first
place...

Well, it doesn't work :-(. CVS is definitely a bit confused about the
status of these files:

$ touch gram.c
$ cvs add gram.c
cvs add: gram.c added independently by second party
$ cvs remove gram.c
cvs remove: file `gram.c' still in working directory
cvs remove: 1 file exists; remove it first
$ rm gram.c
rm: remove regular empty file `gram.c'? y
$ cvs remove gram.c
cvs remove: nothing known about `gram.c'

So there's no way, apparently, to fix the state of these files through
the "front door". Shall we try the proposed idea of hand-moving the
files out of the Attic subdirectory, whereupon they should appear live
and we can cvs remove them again? I have login on cvs.postgresql.org
and can try this, but I'd like confirmation from someone that this is
unlikely to break things. Is there any hidden state to be fixed in the
CVS repository? I don't see any ...

regards, tom lane

I wrote:

So there's no way, apparently, to fix the state of these files through
the "front door".

I take that back: the right sequence involving a "cvs update" got me
into a state where it thought the files were "locally modified", and
then I could commit and "cvs remove" and commit again. So hopefully
it's all cleaned up now --- at least the states of the files look
reasonable in cvsweb.

regards, tom lane

Tom Lane wrote:

Aidan Van Dyk <aidan@highrise.ca> writes:

* Tom Lane <tgl@sss.pgh.pa.us> [070416 19:03]:

I like the idea of re-adding and then re-removing the files on HEAD.
Does anyone think that poses any real risk?

No - it even fixed the "hand moved" test I had done trying to create an
Attic with, when trying to figure out how they got that way in the first
place...

Well, it doesn't work :-(. CVS is definitely a bit confused about the
status of these files:

$ touch gram.c
$ cvs add gram.c
cvs add: gram.c added independently by second party
$ cvs remove gram.c
cvs remove: file `gram.c' still in working directory
cvs remove: 1 file exists; remove it first
$ rm gram.c
rm: remove regular empty file `gram.c'? y
$ cvs remove gram.c
cvs remove: nothing known about `gram.c'

So there's no way, apparently, to fix the state of these files through
the "front door". Shall we try the proposed idea of hand-moving the
files out of the Attic subdirectory, whereupon they should appear live
and we can cvs remove them again? I have login on cvs.postgresql.org
and can try this, but I'd like confirmation from someone that this is
unlikely to break things. Is there any hidden state to be fixed in the
CVS repository? I don't see any ...

Forgive my caution, but I'd suggest trying on a copy first.

cheers

andrew

Andrew Dunstan wrote:

Tom Lane wrote:

So there's no way, apparently, to fix the state of these files through
the "front door". Shall we try the proposed idea of hand-moving the
files out of the Attic subdirectory, whereupon they should appear live
and we can cvs remove them again? I have login on cvs.postgresql.org
and can try this, but I'd like confirmation from someone that this is
unlikely to break things. Is there any hidden state to be fixed in the
CVS repository? I don't see any ...

Forgive my caution, but I'd suggest trying on a copy first.

Too late ;-)

FWIW my CVSup copy seems happy with the change; it reported this when I
updated it:

$ pgcvsup
Connected to cvsup.postgresql.org
Updating collection repository/cvs
Edit pgsql/src/backend/parser/gram.c,v -> Attic
Edit pgsql/src/backend/utils/mb/encnames.c,v
Edit pgsql/src/bin/pg_dump/pg_dump.c,v
Edit pgsql/src/bin/psql/common.c,v
Edit pgsql/src/include/pg_config.h.win32,v
Edit pgsql/src/interfaces/ecpg/preproc/pgc.c,v -> Attic
Edit pgsql/src/interfaces/ecpg/preproc/preproc.c,v -> Attic
Edit pgsql/src/tools/msvc/Solution.pm,v
Rsync sup/repository/checkouts.cvs
Finished successfully

The gram.c,v file looks good -- it has the expected "state dead;" line.
A checked out tree from that updates fine. A "cvs update" to a checked
out tree direct from the main CVS server also updates fine.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Florian G. Pflug wrote:

Initially, I'll post it on http://git.catalyst.net.nz/ and I can run a
daily import for you - once that's in place you can probably get a repo
with your work on http://repo.or.cz/

Ok - you can now clone from http://git.catalyst.net.nz/postgresql.git
viewable from http://git.catalyst.net.nz/gitweb too. It's 24hs behind,
and I'm sorting the updating scripts that will run daily.

The HEAD of CVS is renamed to cvshead there. All the other branches and
tags are untouched. Please DO check that the tip of cvshead matches a
CVS checkout with -kk. I've had limited time to sanitycheck the import ;-)

cheers,

m
--
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington
WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224 UK: 0845 868 5733 ext 7224 MOB: +64(21)364-017
Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

* Tom Lane <tgl@sss.pgh.pa.us> [070416 21:11]:

I wrote:

So there's no way, apparently, to fix the state of these files through
the "front door".

I take that back: the right sequence involving a "cvs update" got me
into a state where it thought the files were "locally modified", and
then I could commit and "cvs remove" and commit again. So hopefully
it's all cleaned up now --- at least the states of the files look
reasonable in cvsweb.

And my GIT conversion handled that nicely too. Looks good (at least
from the GIT PoV).

Now, on my hand-crafted GIT repo - you see them in and out now with
Tom's commits. But any *real* conversion tracking the *actual* RCS cvs
states should have them checked out from 1999 to now in the state they
were from vadim's last changes, and Tom's first commit will "truncate"
them (because he checked them in as empty files), and the 2nd commit
will remove them again.

So it's still a "gotcha" if you're trying to get a copy of CVS from ages
ago via one of the alternative SCM conversions...

But my git one works, so I'll let others worry about the others ;-)

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Aidan Van Dyk <aidan@highrise.ca> writes:

Now, on my hand-crafted GIT repo - you see them in and out now with
Tom's commits. But any *real* conversion tracking the *actual* RCS cvs
states should have them checked out from 1999 to now in the state they
were from vadim's last changes, and Tom's first commit will "truncate"
them (because he checked them in as empty files), and the 2nd commit
will remove them again.

So it's still a "gotcha" if you're trying to get a copy of CVS from ages
ago via one of the alternative SCM conversions...

It shouldn't be a big problem, assuming the checkout preserves the file
dates --- they'll look older than the source files and so a rebuild will
happen anyway in such a checkout.

regards, tom lane