New script to convert p4 repositories to git - git-p4c version 1.

I couldn't use git-p4 on my system because I kept running out of memory,
and I didn't like the workflow it imposed.
Also, it had various other issues with the repo I was trying to use,
mainly because it is not an ideal repository, however those are
(generally) the fault of the particular repo I was using, and not
git-p4. (Which is an excellent script by itself).

This script is severely crippled in that it doesn't (yet) allow one to
contribute changesets back to perforce, however it manages to read from
perforce with:
* No need to rebase.
* Mangling of file names. (Especially with regards to case sensitivity).
* Tagging of revisions with the perforce changesets.
* Ability to handle branches with spaces in the name.
* Ability to pretend that perforce doesn't exist. (That's the plan,
anyway).
* Be extremely memory efficient. It does NOT require as much memory as
does git-p4, even when the size of the change is large.
* Be easy to manually modify the repository, particularly if bad things
happen.

Unfortunately, not all of the above features may be reliable yet,
however I offer this script in order to obtain hopefully constructive
feedback so that I may improve the script and make it work very well.

Once I perfect this script, I plan to work on getting changes from git
back into perforce, which I have a few ideas as to how I might do it.
(None of which require rebasing).

It requires an OS that can efficiently utilise many open files and
pipes, and can run many processes. Such as Linux.  I seriously doubt it
can work on Windows.

It is called git-p4c, because 'git-p4' was taken, and I intended to
write it in C++.  I may still rewrite it in C++ if it is found
neccessary to use it on windows. (The Perforce C++ ABI will remove the
need to fork so many processes), but I won't be doing that before I
implement the write to perforce support.

Consider this to be experimental, not yet worthy of a version number.

Remember, I crave (constructive) feedback.

Thankyou.

John Chapman [off-list ref] writes:

It is called git-p4c, because 'git-p4' was taken, and I intended to
write it in C++.  I may still rewrite it in C++ if it is found
neccessary to use it on windows. (The Perforce C++ ABI will remove the
need to fork so many processes), but I won't be doing that before I
implement the write to perforce support.

Consider this to be experimental, not yet worthy of a version number.

Do you feel it would be worth adding it below git-p4 (and other
Perforce importers) on git wiki page:
  http://git.or.cz/gitwiki/InterfacesFrontendsAndTools

Or would you rather wait a bit to at least version 0.1?
-- 
Jakub Narebski
Poland
ShadeHawk on #git

thestar@fussycoder.id.au wrote on Tue, 09 Dec 2008 21:25 +1100:

I couldn't use git-p4 on my system because I kept running out of memory,
and I didn't like the workflow it imposed.
Also, it had various other issues with the repo I was trying to use,
mainly because it is not an ideal repository, however those are
(generally) the fault of the particular repo I was using, and not
git-p4. (Which is an excellent script by itself).

This script is severely crippled in that it doesn't (yet) allow one to
contribute changesets back to perforce, however it manages to read from
perforce with:
* No need to rebase.
* Mangling of file names. (Especially with regards to case sensitivity).
* Tagging of revisions with the perforce changesets.
* Ability to handle branches with spaces in the name.
* Ability to pretend that perforce doesn't exist. (That's the plan,
anyway).
* Be extremely memory efficient. It does NOT require as much memory as
does git-p4, even when the size of the change is large.
* Be easy to manually modify the repository, particularly if bad things
happen.

I like how your script imports one change at a time, as the initial
import using git-p4 here does indeed get close to exhasting virtual
memory, but I'm running into a different limitation with p4c.

The command:

    p4 -G changes -l -t

is adminstratively limited to a paltry six-digit number, and
produces only an error message.

The other feature I need is the ability to use a client
specification.  We merge together 40-odd different chunks of //depot
into a single checked-out client, and use some other number of
"-//depot/..." rules to exclude some parts of the full depot.

If I hack p4c to limit the changes with "-m 10" or so, then things
are a bit better in that I get two objects (changesets) but no
diffs.  Had to hack the on_branch() code somewhat, in that no form
of --branches seemed to produce an "interesting" changeset by your
definition.  Could be my lack of understanding here.

If you think you want to handle client specifications, and can think
of a way around the "p4 changes" limitation, I'll be happy to poke
at your next version.  Perhaps I'm not in your target audience,
though.  I don't necessarily need to have a full git history of the
entire p4, but this seems to be a fundamental part of your approach.

		-- Pete

On Mon, 2008-12-15 at 14:30 -0500, Pete Wyckoff wrote:
<snip>

I'm running into a different limitation with p4c.

The command:

    p4 -G changes -l -t

is adminstratively limited to a paltry six-digit number, and
produces only an error message.

Hmm, I'm not sure how to resolve this, the repo I use this with so far
only uses 6 digit numbers, will have to review the p4 client documents
on this.

The other feature I need is the ability to use a client
specification.  We merge together 40-odd different chunks of //depot
into a single checked-out client, and use some other number of
"-//depot/..." rules to exclude some parts of the full depot.

If I hack p4c to limit the changes with "-m 10" or so, then things
are a bit better in that I get two objects (changesets) but no
diffs.  Had to hack the on_branch() code somewhat, in that no form
of --branches seemed to produce an "interesting" changeset by your
definition.  Could be my lack of understanding here.

Modifying the on_branch code is the right place to do what you want,
however I was hoping that specifying your 'interesting' branches by
regexp would be sufficient?  All those places that do not get matched
are effectively ignored and do not become part of the git repo.

There is a bug in which the case of the branch name sometimes seems to
change in my repo, and thus my newer version always lower-cases the
names.

Do you need to use a client spec here, or is it possible to just use
regexps?  I'd like to try and avoid the need to parse the clientspec,
that's all.

If you think you want to handle client specifications, and can think
of a way around the "p4 changes" limitation, I'll be happy to poke
at your next version.  Perhaps I'm not in your target audience,
though.  I don't necessarily need to have a full git history of the
entire p4, but this seems to be a fundamental part of your approach.

I'm not sure about client specifications, but I definetly do want to
think of a way around the p4 changes limitation.
I'll get a new version out to you soon (Which includes several bug
fixes), however before I go, I should mention that a deliberate design
decision was to allow the git administrator maximum flexibility.

The script does try to import the full history, however it starts that
history only at the most recent change, which is defined as a tag.

Thus, if your main branch is 'trunk', then you should be able to
manually import your changes, tag it as 'trunk/102388', to indicate
that's p4's revision 102388 of trunk, and the script should then begin
checking out changeset 102389.  I haven't had a chance to test that
theory yet, but it should work.

Thankyou for your valuable feedback.

		-- Pete
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

thestar@fussycoder.id.au wrote on Tue, 16 Dec 2008 08:52 +1100:

On Mon, 2008-12-15 at 14:30 -0500, Pete Wyckoff wrote:
Modifying the on_branch code is the right place to do what you want,
however I was hoping that specifying your 'interesting' branches by
regexp would be sufficient?  All those places that do not get matched
are effectively ignored and do not become part of the git repo.

[..]

Do you need to use a client spec here, or is it possible to just use
regexps?  I'd like to try and avoid the need to parse the clientspec,
that's all.

There is code to parse the client spec in git-p4, and I did hack it
to put the various depot contents in the git tree according to where
the spec says they go.  Not a big hassle, and will do the same for
git-p4c if needed.  (Not clean enough or tested with non-client-spec
configs to submit upstream for git-p4 though.)

The script does try to import the full history, however it starts that
history only at the most recent change, which is defined as a tag.

Thus, if your main branch is 'trunk', then you should be able to
manually import your changes, tag it as 'trunk/102388', to indicate
that's p4's revision 102388 of trunk, and the script should then begin
checking out changeset 102389.  I haven't had a chance to test that
theory yet, but it should work.

Okay, this is interesting.  Would like to pursue that approach.
Except for the "manually import" part.  Maybe bits of git-p4's
full-checkout model could be borrowed here?  Anyway, will take a
look when you're ready.

		-- Pete