Re: git-p4import.py robustness changes

Scott Lamb [off-list ref] writes:

On May 31, 2007, at 4:53 PM, Junio C Hamano wrote:

quoted

Actually, my preference is to have a "patch 0" before all of the
above, that demotes git-p4import to contrib/ hierarchy.  Having
no access to p4 managed repositories (nor much inclination to
get one), I can never test nor maintain it myself, so it is just
crazy for me to be the maintainer for it.

Will do. What does that mean for Documentation/git-p4import.txt and
the git-p4 rpm (defined in git.spec.in)? Should I move them with it?
(Seems nothing else in the main tree references contrib.) If so,
maybe I should set up a common "Documentation/asciidoc.mak" or
something for building the man/html pages rather than duplicating all
that Makefile logic.

A much more preferable alternative is for you to say "Hey, don't
say you want to demote it.  I'll keep it maintained, I regularly
use p4 and have a strong incentive to keep it working".  Then we
do not have to do the "patch 0" ;-)

On Jun 2, 2007, at 2:33 PM, Junio C Hamano wrote:

A much more preferable alternative is for you to say "Hey, don't
say you want to demote it.  I'll keep it maintained, I regularly
use p4 and have a strong incentive to keep it working".  Then we
do not have to do the "patch 0" ;-)

Hmm. I'd like to say that, but keep in mind that I'd never even used  
git before Wednesday, and I'm not sure yet how well git-p4import.py  
will work out for me.

It'd be a huge leap from git-p4import.py to something that could  
remove my need to use p4 commands daily. First, I'd need something  
that could follow all upstream branches with merge history. Then I'd  
either need to convince my team to ditch p4 entirely (not easy, and  
then I wouldn't use/maintain git-p4import.py afterward anyway) or a  
way to robustly generate "p4 integrate", "p4 resolve", "p4 submit"  
command sequences to merge between upstream branches.

We're attempting to address more modest needs, like those of our off- 
site contractors who should only be sending patches anyway. (Another  
guy wrote a script that pulls changes into an svn mirror basically by  
"svn ci -m 'changed some stuff'" every half hour, but I made fun of  
it and now have to replace it. ;)

I'll at least finish up the other patches first and see how it goes.

-- 
Scott Lamb <http://www.slamb.org/>

On Saturday 02 June 2007 23:33:25 Junio C Hamano wrote:

Scott Lamb [off-list ref] writes:

quoted

On May 31, 2007, at 4:53 PM, Junio C Hamano wrote:

quoted

Actually, my preference is to have a "patch 0" before all of the
above, that demotes git-p4import to contrib/ hierarchy.  Having
no access to p4 managed repositories (nor much inclination to
get one), I can never test nor maintain it myself, so it is just
crazy for me to be the maintainer for it.

Will do. What does that mean for Documentation/git-p4import.txt and
the git-p4 rpm (defined in git.spec.in)? Should I move them with it?
(Seems nothing else in the main tree references contrib.) If so,
maybe I should set up a common "Documentation/asciidoc.mak" or
something for building the man/html pages rather than duplicating all
that Makefile logic.

A much more preferable alternative is for you to say "Hey, don't
say you want to demote it.  I'll keep it maintained, I regularly
use p4 and have a strong incentive to keep it working".  Then we
do not have to do the "patch 0" ;-)

On the topic of git integration with perforce, what are the chances of getting 
git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's contrib/fast-export 
area? :)

git-p4 can do everything git-p4import can do plus a lot more (it can track 
multiple branches, it's a hell of a lot faster, it can export back to p4 and 
it also works on Windows!).


Simon

On Jun 3, 2007, at 6:11 AM, Simon Hausmann wrote:

On the topic of git integration with perforce, what are the chances  
of getting
git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's contrib/ 
fast-export
area? :)

git-p4 can do everything git-p4import can do plus a lot more (it  
can track
multiple branches, it's a hell of a lot faster, it can export back  
to p4 and
it also works on Windows!).

I missed that one...I just saw Tailor and the Perl script someone  
else had written.

Ergh. git-p4 imports both "subprocess" and "popen2" and also uses  
"system" and "os.popen". Why use four different modules to launch git  
and p4?

The branch support's interesting. Have you considered tracking  
integration history? I was pondering it and am not sure if it's  
feasible. Perforce doesn't seem to have an efficient way of  
displaying it (just "p4 integrates" that will fetch *all* revisions  
even if you want incremental results and "p4 filelog" which would  
have to be done on each file). Also, I think there's some mismatch  
between the Perforce and git models.

git-p4import.py should work fine on Windows, too - the binary mode on  
the pipe should be all handled by "subprocess", and git-p4's  
data.replace("\r\n", "\n") is not necessary if you use "LineEnd:  
unix" or "share" in the Perforce client specification.

As for performance...hmm. Looks like git-p4import.py runs these  
commands for each Perforce revision:

     realtime  operation
         3.4%  p4 describe -s N
        66.6%  p4 sync ...@N
    [*] 10.2%  git ls-files -m -d -o -z | git update-index --add -- 
remove -z --stdin
         2.6%  git rev-parse --verify HEAD
         4.2%  git write-tree
         2.8%  git commit-tree xxxxxx
         7.5%  git tag -f p4/N xxxxxx
         2.7%  git update-ref HEAD xxxxxx

That's with Perforce running over the network. Are you running locally?

git-p4 seems to use "git fast-import". I guess the big performance  
improvement there is removing the ls-files operation? So we're  
talking about a 0-10% speedup, right? Plus some fork()/exec() overhead.

[*] - Note that I just discovered a big performance regression in my  
patches. Reading the ls-files into Python, through a regexp, and back  
out through update-index was a horrible idea. The times above are  
with that fixed.

-- 
Scott Lamb <http://www.slamb.org/>

Scott Lamb [off-list ref] wrote:

On Jun 3, 2007, at 6:11 AM, Simon Hausmann wrote:

quoted

On the topic of git integration with perforce, what are the chances  
of getting
git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's contrib/ 
fast-export
area? :)

I missed that one...I just saw Tailor and the Perl script someone  
else had written.

Perhaps why it should be in contrib/fast-import?  ;-)
 

As for performance...hmm. Looks like git-p4import.py runs these  
commands for each Perforce revision:

    realtime  operation
        3.4%  p4 describe -s N
       66.6%  p4 sync ...@N
   [*] 10.2%  git ls-files -m -d -o -z | git update-index --add -- 
remove -z --stdin
        2.6%  git rev-parse --verify HEAD
        4.2%  git write-tree
        2.8%  git commit-tree xxxxxx
        7.5%  git tag -f p4/N xxxxxx
        2.7%  git update-ref HEAD xxxxxx

...

git-p4 seems to use "git fast-import". I guess the big performance  
improvement there is removing the ls-files operation? So we're  
talking about a 0-10% speedup, right? Plus some fork()/exec() overhead.

fast-import folds all of the git commands you list above behind
a single engine that is *fast*.  So its actually a 0-30% gain
that is available by using the fast-import backend, with a single
fork()/exec() for the *entire import*.  The local object IO performed
by Git is also minimized, so large imports have much better IO
behavior from the Git perspective.  Its not something to sneeze at.

fast-import also can run in parallel with the frontend process,
allowing you to use a dual-core system, to the extent that your
disk(s) and network can keep up.  Generally p4 is going to be
the bottleneck.

I think writing data to fast-import is much easier than running
the raw Git commands, especially when you are talking about an
import engine where you need to set all of the special environment
variables for git-commit-tree or git-tag to do its job properly.
Its a good tool that simply doesn't get enough use, partly because
nobody is using it...

-- 
Shawn.

Simon Hausmann [off-list ref] wrote:

On the topic of git integration with perforce, what are the chances of getting 
git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's contrib/fast-export 
area? :)

git-p4 can do everything git-p4import can do plus a lot more (it can track 
multiple branches, it's a hell of a lot faster, it can export back to p4 and 
it also works on Windows!).

I was sort of hoping we could fold the fast-export Git repository
on repo.or.cz into core Git at some point.  Right now the only
thing in contrib/fast-export is the import-tars.perl script that
I maintain in my fastimport repository...  ;-)

Like Junio I don't use Perforce, and can't test against it, but
if you can maintain git-p4 (and I think the history on repo.or.cz
shows that you do) then it may be a good idea to add it to core Git.

Send a patch to add it.  Worst that happens is both Junio and I
decide not to apply it.  Or I apply it, but Junio refuses to pull
from me afterwards.  ;-)

-- 
Shawn.

On 6/3/07, Shawn O. Pearce [off-list ref] wrote:

I think writing data to fast-import is much easier than running
the raw Git commands, especially when you are talking about an
import engine where you need to set all of the special environment
variables for git-commit-tree or git-tag to do its job properly.
Its a good tool that simply doesn't get enough use, partly because
nobody is using it...

Well,  perhaps they use it *once*,  in that they write a wrapper script for
it and then forget about it.  At least that's what I did.  And the _only_
annoyance was the trailing NL requirement on the delimited "data" statement,
so you don't get much noise/complaints when people use it.

Thanks,
-- 
Dana L. How  danahow@gmail.com  +1 650 804 5991 cell

Dana How [off-list ref] wrote:

On 6/3/07, Shawn O. Pearce [off-list ref] wrote:

quoted

I think writing data to fast-import is much easier than running
the raw Git commands, especially when you are talking about an
import engine where you need to set all of the special environment
variables for git-commit-tree or git-tag to do its job properly.
Its a good tool that simply doesn't get enough use, partly because
nobody is using it...

Well,  perhaps they use it *once*,  in that they write a wrapper script for
it and then forget about it.  At least that's what I did.  And the _only_
annoyance was the trailing NL requirement on the delimited "data" statement,
so you don't get much noise/complaints when people use it.

True.  I did try to make fast-import take a simple enough format
that you could write throwaway code against it, run it, and never
look back...

The trailing NL after data was because of cvs2svn.  The SVN dump
file format apparently does something like this, and the version
of cvs2svn that Jon Smirl was working on output that trailing NL.
Accepting it in fast-import was easier than fixing cvs2svn to not
create it.

I'll admit the error handling in fast-import could probably
be easier, and that NL after data probably could be optional.
I don't think the input stream parser needs it to understand what
is going on.  Its just sheer laziness on my part that the code
requires it there.

-- 
Shawn.

On Jun 3, 2007, at 10:54 PM, Shawn O. Pearce wrote:

I think writing data to fast-import is much easier than running
the raw Git commands, especially when you are talking about an
import engine where you need to set all of the special environment
variables for git-commit-tree or git-tag to do its job properly.
Its a good tool that simply doesn't get enough use, partly because
nobody is using it...

Yeah, I'm sold. I read git-p4 more thoroughly and tried it out...it's  
pretty nice. The P4Sync command has a simpler, more trustworthy flow  
than git-p4import.py.

On the Perforce side, I particularly like the use of "p4 print" to  
grab the files instead of "p4 sync". It avoids playing weird games  
with the client - I think nothing good can come of git-p4import.py's  
"p4 sync -k" and symlinks to map multiple branches into the same  
directory, which is not the Perforce way. Makes me nervous that  
what's submitted to git won't be the same as what's in the Perforce  
depot.

I would have thought launching a "p4 print" on each file would be  
horribly slow with the network latency of each request, but...well,  
apparently not.

Maybe I'll work up git-p4 patches for subcommand error handling, like  
my git-p4import.py ones. And fix some style - seriously, who puts  
semicolons at the end of Python commands? *grumble*

Best regards,
Scott

-- 
Scott Lamb <http://www.slamb.org/>

git-p4import.py should work fine on Windows, too - the binary mode on  
the pipe should be all handled by "subprocess", and git-p4's  
data.replace("\r\n", "\n") is not necessary if you use "LineEnd:  
unix" or "share" in the Perforce client specification.

The problem is that you cannot set the LineEnd when using the 'p4 
print' command, since it doesn't use the client spec; so Perforce the 
uses the platform default when printing the file.

git-p4 seems to use "git fast-import". I guess the big performance
improvement there is removing the ls-files operation? So we're 
talking about a 0-10% speedup, right? Plus some fork()/exec()
overhead.

With git-p4 the performance bottleneck is from what we can see the 
Perforce server, on non-Windows machines.

-- 
.marius

On Monday 04 June 2007 09:19:56 Scott Lamb wrote:

On Jun 3, 2007, at 10:54 PM, Shawn O. Pearce wrote:

quoted

I think writing data to fast-import is much easier than running
the raw Git commands, especially when you are talking about an
import engine where you need to set all of the special environment
variables for git-commit-tree or git-tag to do its job properly.
Its a good tool that simply doesn't get enough use, partly because
nobody is using it...

Yeah, I'm sold. I read git-p4 more thoroughly and tried it out...it's
pretty nice. The P4Sync command has a simpler, more trustworthy flow
than git-p4import.py.

On the Perforce side, I particularly like the use of "p4 print" to
grab the files instead of "p4 sync". It avoids playing weird games
with the client - I think nothing good can come of git-p4import.py's
"p4 sync -k" and symlinks to map multiple branches into the same
directory, which is not the Perforce way. Makes me nervous that
what's submitted to git won't be the same as what's in the Perforce
depot.

I would have thought launching a "p4 print" on each file would be
horribly slow with the network latency of each request, but...well,
apparently not.

I've found it to be fast enough for "standard software development". When 
importing big changes like integrations of an entire branch then it naturally 
slows down. The workaround me and my colleague have come up with is to 
combine git-p4 usage with the regular git protocol:

For imports of simple projects from perforce the direct use of git-p4 clone 
and sync/rebase is good enough.

For big projects we have set up a dedicated (recycled old) machine that 
continuously imports from the perforce server. That makes the initial clone 
very fast thanks to the use of the git protocol, it still allows imports from 
perforce afterwards and when the developer syncs the chances are very high 
that the dedicated machine already imported the necessary changes/objects 
from the perforce server and the faster git protocol instead of "p4 print" on 
a lot of files can be used.

In order to avoid that machine constantly polling the p4 server we've come up 
with a neat little trick by adding a change-commit trigger on the p4 server 
that consists of a little perl script that just sends a single udp packet 
with the latest change number as notification to the git machine, which upon 
reception imports then.

That is why git-p4 sync/rebase call "git fetch" by default (configurable 
through config key) if there is an origin remote present.

Maybe I'll work up git-p4 patches for subcommand error handling, like
my git-p4import.py ones. And fix some style - seriously, who puts
semicolons at the end of Python commands? *grumble*

I'd be more than happy to apply style patches. I'm not a very experienced 
python programmer and I admit that I certainly lack the style there :)

Simon

On Monday 04 June 2007 07:56:00 Shawn O. Pearce wrote:

Simon Hausmann [off-list ref] wrote:

quoted

On the topic of git integration with perforce, what are the chances of
getting git-p4 ( http://repo.or.cz/w/fast-export.git ) into git's
contrib/fast-export area? :)

git-p4 can do everything git-p4import can do plus a lot more (it can
track multiple branches, it's a hell of a lot faster, it can export back
to p4 and it also works on Windows!).

I was sort of hoping we could fold the fast-export Git repository
on repo.or.cz into core Git at some point.  Right now the only
thing in contrib/fast-export is the import-tars.perl script that
I maintain in my fastimport repository...  ;-)

Like Junio I don't use Perforce, and can't test against it, but
if you can maintain git-p4 (and I think the history on repo.or.cz
shows that you do) then it may be a good idea to add it to core Git.

Send a patch to add it.  Worst that happens is both Junio and I
decide not to apply it.  Or I apply it, but Junio refuses to pull
from me afterwards.  ;-)

Ok, I'll give it a try :)

I've used git-filter-branch to rewrite the history in fast-export to include 
only changes relevant to git-p4 and at the same time move all files into 
contrib/fast-import. The result is available as separate branch at

	git://repo.or.cz/fast-export.git git-p4

and technically merges fine into git.git's contrib/fast-import directory with 
three files (git-p4, git-p4.txt and git-p4.bat for windows convenience).

Please let me know if there's anything missing or if you prefer a different 
format or so. I also realized that I haven't really used the 'Signed-off-by' 
tags in the past but I'd be happy to adopt it for git inclusion if you prefer 
that :)


_If_ one of you decides to pull then my plan is to discontinue the git-p4 
branch in the fast-export repository and instead work in a git.git fork on 
repo.or.cz (similar to the fastimport repository).


Simon

Simon Hausmann wrote:

_If_ one of you decides to pull then my plan is to discontinue the git-p4 
branch in the fast-export repository and instead work in a git.git fork on 
repo.or.cz (similar to the fastimport repository).

So you'll continue maintain this code and others should submit changes 
through you? What is the best way to do so? (Not sure what it was 
before, or if it would change under this plan.) Email a format-patch To: 
you? Cc: this list? some other list? no list?

-- 
Scott Lamb <http://www.slamb.org/>

On Wednesday 13 June 2007 23:06:49 Scott Lamb wrote:

Simon Hausmann wrote:

quoted

_If_ one of you decides to pull then my plan is to discontinue the git-p4
branch in the fast-export repository and instead work in a git.git fork
on repo.or.cz (similar to the fastimport repository).

So you'll continue maintain this code and others should submit changes
through you? What is the best way to do so? (Not sure what it was
before, or if it would change under this plan.) Email a format-patch To:
you? Cc: this list? some other list? no list?

I would say whichever you prefer :)

For the fast-export repository multiple people have access and for example 
after Han-Wen made a lot of patches I asked Chris Lee (owner of the module on 
repo.or.cz) to add Han-Wen to the list of people with push access and he 
pushed his changes directly. I actually like working that way for a project 
that is as simple as that, I don't mind if somebody pushes simple changes 
directly as much as I like discussing bigger plans if they potentially clash 
with somebody else's work or use-case.

So if git-p4 continues to live in fast-export I'll continue to encourage Chris 
Lee to give git-p4 contributors push access, if it's in a git.git fork I'd be 
happy to do so myself (give access).

If git-p4 also ends up in git/contrib/fastimport and somebody likes to send 
patches to Junio or somebody else and CC this list that's fine with me, too.

It's just a few lines of python code after all ;-)

Simon

Simon Hausmann [off-list ref] wrote:

I've used git-filter-branch to rewrite the history in fast-export to include 
only changes relevant to git-p4 and at the same time move all files into 
contrib/fast-import. The result is available as separate branch at

	git://repo.or.cz/fast-export.git git-p4

and technically merges fine into git.git's contrib/fast-import directory with 
three files (git-p4, git-p4.txt and git-p4.bat for windows convenience).

Please let me know if there's anything missing or if you prefer a different 
format or so. I also realized that I haven't really used the 'Signed-off-by' 
tags in the past but I'd be happy to adopt it for git inclusion if you prefer 
that :)

Yes.  The SBO line is your assertion that you own the rights to the
code and can release it under the license you are offering it under.
One of the issues I have with this git-p4 history you have built
is the lack of the SBO line on all 255 commits.

Of course an SBO line doesn't carry that much weight, its just a line
after all, but according to Git's project standards it should be there
if you are agreeing to release it.  See Documentation/SubmittingPatches
for details.

My other problem with this history is a commit like b79112 "a
little bit more convenience" (and there are many such commits).
This message is insanely short, doesn't really talk at all about
what a little bit is, how it is more convenient, or who it is more
convenient for.

Think about how that oneline (and the others) would look in Junio's
"What's new in git.git" emails, or in gitweb.  There is not enough
detail here to be of any value to the reader.  Expanding out to the
full message offers nothing additional either, because that is all
there is in the entire commit message body.

I do appreciate you taking the time to use filter-branch to try to
cleanup this history a bit.  I really had originally planned on
pulling your tree through to my fastimport tree and then talking
Junio into merging with me.  But after reading through this history I
don't want do that, because of the oneline summaries I just pointed
out above, and because of the missing SBO.
 
-- 
Shawn.

On Thursday 14 June 2007 07:35:38 Shawn O. Pearce wrote:

Simon Hausmann [off-list ref] wrote:

quoted

I've used git-filter-branch to rewrite the history in fast-export to
include only changes relevant to git-p4 and at the same time move all
files into contrib/fast-import. The result is available as separate
branch at

	git://repo.or.cz/fast-export.git git-p4

and technically merges fine into git.git's contrib/fast-import directory
with three files (git-p4, git-p4.txt and git-p4.bat for windows
convenience).

Please let me know if there's anything missing or if you prefer a
different format or so. I also realized that I haven't really used the
'Signed-off-by' tags in the past but I'd be happy to adopt it for git
inclusion if you prefer that :)

Yes.  The SBO line is your assertion that you own the rights to the
code and can release it under the license you are offering it under.
One of the issues I have with this git-p4 history you have built
is the lack of the SBO line on all 255 commits.

Of course an SBO line doesn't carry that much weight, its just a line
after all, but according to Git's project standards it should be there
if you are agreeing to release it.  See Documentation/SubmittingPatches
for details.

My other problem with this history is a commit like b79112 "a
little bit more convenience" (and there are many such commits).
This message is insanely short, doesn't really talk at all about
what a little bit is, how it is more convenient, or who it is more
convenient for.

Think about how that oneline (and the others) would look in Junio's
"What's new in git.git" emails, or in gitweb.  There is not enough
detail here to be of any value to the reader.  Expanding out to the
full message offers nothing additional either, because that is all
there is in the entire commit message body.

I do appreciate you taking the time to use filter-branch to try to
cleanup this history a bit.  I really had originally planned on
pulling your tree through to my fastimport tree and then talking
Junio into merging with me.  But after reading through this history I
don't want do that, because of the oneline summaries I just pointed
out above, and because of the missing SBO.

First of all thanks for looking at the branch. I agree with your concerns and 
I do admit that I've been a bit too sloppy with the log messages.

I have started cleaning up the history even more by reworking the log messages 
of my commits (git-p4-enhanced-logs branch in fast-export, starting at the 
last page). Once that is done (I expect that to take a few days) I'll add the 
missing SOB lines with git-filter-branch and see if I can get an agreement 
from Han-Wen and Marius for doing the same with their commits (adding the 
missing lines).

Would you be willing to reevaluate the situation regarding a merge once that's 
done?


Simon

Simon Hausmann [off-list ref] wrote:

On Thursday 14 June 2007 07:35:38 Shawn O. Pearce wrote:

quoted

I do appreciate you taking the time to use filter-branch to try to
cleanup this history a bit.  I really had originally planned on
pulling your tree through to my fastimport tree and then talking
Junio into merging with me.  But after reading through this history I
don't want do that, because of the oneline summaries I just pointed
out above, and because of the missing SBO.

...

I have started cleaning up the history even more by reworking the log messages 
of my commits (git-p4-enhanced-logs branch in fast-export, starting at the 
last page). Once that is done (I expect that to take a few days) I'll add the 
missing SOB lines with git-filter-branch and see if I can get an agreement 
from Han-Wen and Marius for doing the same with their commits (adding the 
missing lines).

OK.
 

Would you be willing to reevaluate the situation regarding a merge once that's 
done?

Absolutely.  I would like to see the git-p4 work in the main tree,
so it is more readily available to users, even though I'm not a p4
user myself.  ;-)

-- 
Shawn.

Simon Hausmann said the following on 14.06.2007 23:44:

First of all thanks for looking at the branch. I agree with your
concerns and I do admit that I've been a bit too sloppy with the
log messages.

I have started cleaning up the history even more by reworking the
log messages of my commits (git-p4-enhanced-logs branch in
fast-export, starting at the last page). Once that is done (I
expect that to take a few days) I'll add the missing SOB lines with
git-filter-branch and see if I can get an agreement from Han-Wen
and Marius for doing the same with their commits (adding the 
missing lines).

Simon,

Of course! Go right ahead and add the SOB for my commits while you're 
at it.

-- 
.marius