Re: Is anyone working on a next-gen Git protocol?

Steffen Prohaska [off-list ref] writes:

I've recently discovered that the current protocol can be amazingly
inefficient when it comes to transferring binary objects.  Assuming two
repositories that are in sync.  After a 'git checkout --orphan && git
commit', a subsequent transfers sends all the blobs attached to the new
commit, although the other side already has all the blobs.

I do not think it has anything to do with binary, but what you
deserve from using orphan, where you declared that the history does
not have anything to do with the original.

If both of your repositories had the two paralle lines of these
histories as branches, the transfer would have went well with or
without binary objects.

From: "Junio C Hamano" <redacted>

Steffen Prohaska [off-list ref] writes:

quoted

I've recently discovered that the current protocol can be amazingly
inefficient when it comes to transferring binary objects.  Assuming 
two
repositories that are in sync.  After a 'git checkout --orphan && git
commit', a subsequent transfers sends all the blobs attached to the 
new
commit, although the other side already has all the blobs.

I do not think it has anything to do with binary, but what you
deserve from using orphan, where you declared that the history does
not have anything to do with the original.

If both of your repositories had the two paralle lines of these
histories as branches, the transfer would have went well with or
without binary objects.
--

Steffen,
An alternative could be a shallow clone for just those branches with the 
binary objects, so that the git objects are still identical. Or use a 
replace/graft to trim the line of development. It's still a fudge, but 
something you could look at. 

On Thu, Oct 11, 2012 at 3:46 AM, Junio C Hamano [off-list ref] wrote:

Steffen Prohaska [off-list ref] writes:

quoted

I've recently discovered that the current protocol can be amazingly
inefficient when it comes to transferring binary objects.  Assuming two
repositories that are in sync.  After a 'git checkout --orphan && git
commit', a subsequent transfers sends all the blobs attached to the new
commit, although the other side already has all the blobs.

I do not think it has anything to do with binary, but what you
deserve from using orphan, where you declared that the history does
not have anything to do with the original.

If both of your repositories had the two paralle lines of these
histories as branches, the transfer would have went well with or
without binary objects.

On the same inefficient subject, git does not try to share common
objects for non-commit refs, for example tags pointing to trees. I
have such a peculiar repo and if a new tag shares 90% the tree with
existing tags, git-fetch to sends the whole tree of the new tag over
the wire. It does not seem easy to fix though and is probably rare
enough that does not justify proper support. As a work around, I
generate commits that link all these tags/trees together in a
predetermined order. Not nice but works ok.
-- 
Duy

On Wed, Oct 10, 2012 at 6:44 PM, Nguyen Thai Ngoc Duy [off-list ref] wrote:

On Thu, Oct 11, 2012 at 3:46 AM, Junio C Hamano [off-list ref] wrote:

quoted

Steffen Prohaska [off-list ref] writes:

quoted

I've recently discovered that the current protocol can be amazingly
inefficient when it comes to transferring binary objects.  Assuming two
repositories that are in sync.  After a 'git checkout --orphan && git
commit', a subsequent transfers sends all the blobs attached to the new
commit, although the other side already has all the blobs.

I do not think it has anything to do with binary, but what you
deserve from using orphan, where you declared that the history does
not have anything to do with the original.

If both of your repositories had the two paralle lines of these
histories as branches, the transfer would have went well with or
without binary objects.

On the same inefficient subject, git does not try to share common
objects for non-commit refs, for example tags pointing to trees. I
have such a peculiar repo and if a new tag shares 90% the tree with
existing tags, git-fetch to sends the whole tree of the new tag over
the wire. It does not seem easy to fix though and is probably rare
enough that does not justify proper support. As a work around, I
generate commits that link all these tags/trees together in a
predetermined order. Not nice but works ok.

Aside from saving a huge amount of CPU during the "Counting objects"
phase, the compressed bitmap work we presented in JGit solves this by
working off the complete reachability graph, and not just some subset
related to a cut made across the commit graph. Unfortunately we took a
shortcut and didn't create bitmaps for non-commits, but this is a
trivial modification to the algorithm and the storage.