[git-p4] Trouble importing all perforce branches into git repository

I'm trying to import a project from my company's Perforce depot into a git repository. I've been successful getting a single branch into a single git repo, but what I'd like to do is to pull in the entire project plus all branches, but I can't get it working correctly.

I have my .p4settings file:

P4PORT=perforce-server.local:1666
P4USER=my.username
P4CLIENT=my.clientspec.name

Clientspec is fairly simple:

//project/... //my.clientspec.name/project/...
-//project/External/... //my.clientspec.name/project/External/...

The second line is to exclude the external library binaries from the checkout.

I use the following command line to start git-p4 importing:

git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all

This goes through and checks all the branch names, etc, and attempts to import starting at the first commit (in this case its changelists 771 for this project).

Importing from //project/...@all into project
Initialized empty Git repository in /Users/grant/Development/git_test/project/.git/
Reading pipe: git config git-p4.useclientspec
Reading pipe: git config git-p4.user
Reading pipe: git config git-p4.password
Reading pipe: git config git-p4.port
Reading pipe: git config git-p4.host
Reading pipe: git config git-p4.client
p4 -G branches
Opening pipe: p4 -G branches
p4 -G branch -o branch-1
Opening pipe: p4 -G branch -o branch-1
Reading pipe: git config --bool core.ignorecase
p4 -G branch -o branch-2
...
Opening pipe: p4 -G branch -o branch-n
p4-git branches: []
initial parents: {}

I'm not sure if p4-git branches, and initial parents are supposed to be empty here, but they are.

Finally we get to the point where it's about to start importing changelists into git and the following happens:

Getting p4 changes for //project/...
p4 changes //project/...
Reading pipe: p4 changes //project/...
p4 -G describe 771
Opening pipe: p4 -G describe 771
Importing revision 771 (1%)branch is MAIN

    Importing new branch RCMerge/MAIN
p4 changes //RCMerge/MAIN/...@1,770
Reading pipe: p4 changes //RCMerge/MAIN/...@1,770

    Resuming with change 771
parent determined through known branches: Branch-foo
looking for initial parent for refs/remotes/p4/project/MAIN; current parent is refs/remotes/p4/project/Branch-foo
commit into refs/remotes/p4/project/MAIN
parent refs/remotes/p4/project/Branch-foo
p4 -G -x - print
Opening pipe: p4 -G -x - print
Glue/source/.empty
fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo
fast-import: dumping crash report to .git/fast_import_crash_26002

Here's the file referenced above:

fast-import crash report:
    fast-import process: 26002
    parent process     : 26000
    at Thu May 19 11:51:54 2011

fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo

Most Recent Commands Before Crash
---------------------------------
  checkpoint
  commit refs/remotes/p4/project/MAIN
  committer Some User [off-list ref] 1253574589 -0800
  data <<EOT
* from refs/remotes/p4/project/Branch-foo

Active Branch LRU
-----------------
    active_branches = 0 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Inactive Branches
-----------------
refs/remotes/p4/project/MAIN:
  status      : dirty
  tip commit  : 0000000000000000000000000000000000000000
  old tree    : 0000000000000000000000000000000000000000
  cur tree    : 0000000000000000000000000000000000000000
  commit clock: 0
  last pack   : 


Marks
-----

-------------------
END OF CRASH REPORT

Now, being a perforce novice, I have no idea what this means and have no idea how to resolve this issue, or if its even possible. Has anyone run into a similar issue before? If so, how did you resolve it?


Grant Limberg
glimberg@gmail.com

glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:

[git-p4 automatic branch detection]

quoted

git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all

I worry that --max-changes prevents it from pulling in the
ref for Branch-foo, and this confuses the automatic branch
detection that assumes it has all its parents.

It would certainly be more reliable if you did "git-p4 sync
--branch foo //project/foo@all" for each branch you care about.

Vitor has some modifications to auto-branch detection that might
be useful, or maybe some thoughts as to what's going on.

		-- Pete

(Vitor: full mail is here:
http://article.gmane.org/gmane.comp.version-control.git/173996/ )

On May 22, 2011, at 4:49 AM, Pete Wyckoff wrote:

glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:

quoted

[git-p4 automatic branch detection]

quoted

git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all

I worry that --max-changes prevents it from pulling in the
ref for Branch-foo, and this confuses the automatic branch
detection that assumes it has all its parents.

It would certainly be more reliable if you did "git-p4 sync
--branch foo //project/foo@all" for each branch you care about.

Vitor has some modifications to auto-branch detection that might
be useful, or maybe some thoughts as to what's going on.

		-- Pete

(Vitor: full mail is here:
http://article.gmane.org/gmane.comp.version-control.git/173996/ )

Removing --max-changes=100 has no effect on the crash I'm experiencing.   Same crash still happens on the very first imported commit.

If i do "git-p4 sync --branch foo //project/foo@all" for each branch, does that convert 'p4 integrate' commands into git merges?  I"m trying to keep all history intact when importing from p4 into git.

Thanks,

-Grant

Hi Grant,

I think I saw that happening when there is nothing to import or in
situations where the parent of a certain branch does not exist.

Perforce does not strictly require branch definitions to allow a branch
structure/flow. It is quite possible that you are integrating stuff
around without using "branch specs". When git-p4 is importing your P4
database it will look at all branches available in the server, including
ones from other depots. So it is possible that while you are seeing a
big list of branches being processed, none apply to the branch structure
you are trying to import.

Could you please confirm that you have Perforce branch specs for all
branches that you are trying to import?

Thanks,
Vitor

On May 23, 2011, at 3:54 AM, Vitor Antunes wrote:

Hi Grant,

I think I saw that happening when there is nothing to import or in
situations where the parent of a certain branch does not exist.

Perforce does not strictly require branch definitions to allow a branch
structure/flow. It is quite possible that you are integrating stuff
around without using "branch specs". When git-p4 is importing your P4
database it will look at all branches available in the server, including
ones from other depots. So it is possible that while you are seeing a
big list of branches being processed, none apply to the branch structure
you are trying to import.

Could you please confirm that you have Perforce branch specs for all
branches that you are trying to import?

Thanks,
Vitor

Vitor, 

The "Branch-foo" that git-p4 is crashing on does have a branchspec in Perforce.  I cannot, however guarantee that branchspecs were correctly created and used for all integrates in the repository.  It was created long before I started working here.  Is there a way to limit --detect-branches to only certain branches that I can ensure do have branchspecs?

- Grant

Hi Grant,

The "Branch-foo" that git-p4 is crashing on does have a branchspec in
Perforce.  I cannot, however guarantee that branchspecs were
correctly created and used for all integrates in the repository.  It
was created long before I started working here.

You can create the branch specs now. There is no requirement for them to
have been created at the time of the integration.

I've sent a patch to this mailing list [1] that adds the possibility of
creating these branches in a gitconfig file. If you feel brave enough,
please give it a try. The patch is missing a test case, which I still
did not have time to review and send

Is there a way to limit --detect-branches to only certain branches
that I can ensure do have branchspecs?

If you have an incremental structure you could limit the cloning to a
subset of the directories. I am not aware of any other way. Maybe
someone else in the mailing list can shed some light...? :)

Vitor

[1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000

On May 23, 2011, at 10:38 AM, Vitor Antunes wrote:

Hi Grant,

quoted

The "Branch-foo" that git-p4 is crashing on does have a branchspec in
Perforce.  I cannot, however guarantee that branchspecs were
correctly created and used for all integrates in the repository.  It
was created long before I started working here.

You can create the branch specs now. There is no requirement for them to
have been created at the time of the integration.

I've sent a patch to this mailing list [1] that adds the possibility of
creating these branches in a gitconfig file. If you feel brave enough,
please give it a try. The patch is missing a test case, which I still
did not have time to review and send

quoted

Is there a way to limit --detect-branches to only certain branches
that I can ensure do have branchspecs?

If you have an incremental structure you could limit the cloning to a
subset of the directories. I am not aware of any other way. Maybe
someone else in the mailing list can shed some light...? :)

Vitor

[1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000

I think I just figured out the problem.  The branch mapping between //project/MAIN/... and //project/Branch-foo/... was done backwards, thus git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse was actually true.  MAIN's first changelist number was 771.  Branch-foo's first changelist number was 7652.  That's why git-p4 was getting confused.  I fixed the branch mapping in Perforce and the clone with --detect-branches seems to be working fine now. 

Grant Limberg
glimberg@gmail.com

Grant Limberg wrote:

I think I just figured out the problem.  The branch mapping between
//project/MAIN/... and //project/Branch-foo/... was done backwards, thus
git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse
was actually true.

Did you get it working so that the branch integrates at Perforce show up as
merges in git? I have been trying to do this for tens of hours now and just
can't get it working.. I have tried making the branch mapping both ways, and
even both at the same time but with no success.

I have succesfully imported the two testing branches to git, but if I look
at it in gitk for example, the history is linear and there are no merges
from the other branch (it shows up as a normal commit, without the parent
from other branch, just adding all lines that the merging did). in p4v it
shows up as it should!

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482174.html
Sent from the git mailing list archive at Nabble.com.

On Thu, Jun 16, 2011 at 10:28 AM, tumik [off-list ref] wrote:

Did you get it working so that the branch integrates at Perforce show up as
merges in git? I have been trying to do this for tens of hours now and just
can't get it working.. I have tried making the branch mapping both ways, and
even both at the same time but with no success.

I have succesfully imported the two testing branches to git, but if I look
at it in gitk for example, the history is linear and there are no merges
from the other branch (it shows up as a normal commit, without the parent
from other branch, just adding all lines that the merging did). in p4v it
shows up as it should!

git-p4 only uses the branch mapping for the initial commit on a
branch. It does not attempt to detect merges after that point. After
importing to git, merges just appear as regular commits in the
history.

Unfortunately, implementing a general solution for this is
non-trivial. Perforce branches and merges per file, meaning that
partial merges are possible in Perforce. Such states can not be
correctly represented in Git, and depending on your needs, you may or
may not want a particular partial merge to be considered as a git
merge.

There was another script called git-p4raw for importing Perforce
history to git, which does merge detection. It solved the problem by
having configurable rules for what should be regarded as a merge, and
it supported manually reviewing the decisions afterwards (iirc).

One possibly useful feature in git-p4 would be an option to regard
every integration as a complete merge, which shouldn't be too hard to
implement. But this would break every time someone did a single-file
integration between two branches (which happens quite often in my
experience, for cherry-picking changes etc.).

Thomas

Thomas Berg wrote:

git-p4 only uses the branch mapping for the initial commit on a
branch. It does not attempt to detect merges after that point. After
importing to git, merges just appear as regular commits in the
history.

Thank you for the info, that's the way it seems to be working! It's a shame
if the cloning from perforce -> git can't be made succesfully keeping all
the history information :(


Thomas Berg wrote:

There was another script called git-p4raw for importing Perforce
history to git, which does merge detection. It solved the problem by
having configurable rules for what should be regarded as a merge, and
it supported manually reviewing the decisions afterwards (iirc).

Yes, I tried the git-p4raw too. Actually even several forks of it! But I
wasn't able to get any of those working, as none of them could load the
perforce database correctly. Maybe the perforce database has changed lately
or something, as git-p4raw gave errors that it's database tables don't have
as much columns as perforce's does. Tried also to change the code to add
those columns but there were so many of them and in many tables so I decided
it's too hard and time-consuming to start reverse-engineering them..

- tumik

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482333.html
Sent from the git mailing list archive at Nabble.com.

On Thu, Jun 16, 2011 at 11:35 AM, tumik [off-list ref] wrote:

Yes, I tried the git-p4raw too. Actually even several forks of it! But I
wasn't able to get any of those working, as none of them could load the
perforce database correctly. Maybe the perforce database has changed lately
or something, as git-p4raw gave errors that it's database tables don't have
as much columns as perforce's does. Tried also to change the code to add
those columns but there were so many of them and in many tables so I decided
it's too hard and time-consuming to start reverse-engineering them..

- tumik

I could maybe mention another approach: if your Perforce history isn't
too large so you can extract the info manually, or if you are able to
create a script to figure out your merges (changelist number, revision
of source and target branch in Perforce) it is easy to modify the git
history with this info afterwards.

You can add merge arrows to your git-p4 imported repository, either by
creating a .git/info/graft file and running git-filter-branch
afterwards (to make it permanent and distributable), or by using
git-replace. I may have a script somewhere that generates the graft
file from p4 changelist numbers and branch names, if you're
interested.

The graft + git-filter-branch is good for a one-time conversion, but
since it rewrites history it may not work well for branches that still
get updated in Perforce.

I have not tested git-replace myself, but it should be possible to use
this to add merge arrows too, without modifying the hashes of commits
later in the history. I'm not sure how well git-replace works with the
rest of git though.

Thomas