From: Junio C Hamano <hidden> Date: 2016-06-15 23:01:06
Sitaram Chamarty [off-list ref] writes:
Is there a trick to optimising a push by telling the receiver to pick up
missing objects from some other repo on its own server, to cut down even
more on network traffic?
So, hypothetically,
git push user@host:repo1 --look-for-objects-in=repo2
I'm aware of the alternates mechanism, but that makes the dependency on
the other repo sort-of permanent.
In the direction of fetching, this may be give a good starting point.
http://thread.gmane.org/gmane.comp.version-control.git/243918/focus=245397
In the direction of pushing, theoretically you could:
- define a new capability "look-for-objects-in" to pass the name of
the repository from "git push" to the "receive-pack";
- have "receive-pack" temporarily borrow from the named repository
(if the policy on the server side allows it), and accept the push;
- repack in order to dissociate the receiving repository from the
other repository it temporarily borrowed from.
which would be the natural inverse of the approach suggested in the
"Can I borrow just temporarily while cloning?" thread.
But I haven't thought things through with respect to what else need
to be modified to make sure this does not have adverse interaction
with simultaneous pushes into the same repository, which would make
it harder to solve for "receive-pack" than for "clone/fetch".
Is there a trick to optimising a push by telling the receiver to pick up
missing objects from some other repo on its own server, to cut down even
more on network traffic?
So, hypothetically,
git push user@host:repo1 --look-for-objects-in=repo2
I'm aware of the alternates mechanism, but that makes the dependency on
the other repo sort-of permanent.
That's an interesting thread and it's recent too. However, it's about
clone (though the intro email mentions other commands also).
I'm specifically interested in push efficiency right now. When you
"fork" someone's repo to your own space, and you push your fork to the
same server, it ought to be able to get most of the common objects from
disk (specifically, from the repo you forked), and only what extra you
did from the network.
Clones do have a workaround (clone with --reference, then repack, as you
said in that thread), but no such workaround exists for push.
In the direction of pushing, theoretically you could:
- define a new capability "look-for-objects-in" to pass the name of
the repository from "git push" to the "receive-pack";
- have "receive-pack" temporarily borrow from the named repository
(if the policy on the server side allows it), and accept the push;
- repack in order to dissociate the receiving repository from the
other repository it temporarily borrowed from.
which would be the natural inverse of the approach suggested in the
"Can I borrow just temporarily while cloning?" thread.
But I haven't thought things through with respect to what else need
to be modified to make sure this does not have adverse interaction
with simultaneous pushes into the same repository, which would make
it harder to solve for "receive-pack" than for "clone/fetch".
I'll leave it in your capable hands :-) My C coding days are long gone!
I do have a way to do this in gitolite (haven't coded it yet; just
thinking). Gitolite lets you specify something to do before git-*-pack
runs, and I was planning something like this:
terminology: borrow, borrower repo, reference repo
"borrow = relaxed" mode
1. check if the user has read access to the reference repo; skip
the rest of this if he doesn't
2. from reference repo's "objects", find all directories and
"mkdir" them into borrower's objects directory, then find all
files and "ln" (hardlink) them. This is presumably what "clone
-l" does.
This method is close to constant time since we're not copying
objects.
It has the potential issue that if an object existed in the
reference repo that was subsequently *deleted* (say, a commit that
contained a password, which was quickly overwritten when
discovered), and the attacker knows the SHA, he can get the commit
out by sending an commit that depends on it, then fetching it back.
(He could do that to the reference repo directly if he had write
access, but we'll assume he doesn't, so this *is* a possible
attack).
"borrow = strict" mode
1. (same as for "relaxed" mode)
2. actually *fetch* all refs from the reference repo to the
borrower (into, say, 'refs/borrowed'), then delete all those
refs so you just have the objects now.
Unlike the previous method, this takes time proportional to the
delta between borrower and reference, and may load the system a bit,
but unless the reference repo is highly volatile, this will settle
down. The point is that it cannot be used to get anything that the
user doesn't already have access to anyway.
I still have to try it, but it sounds like both these would work.
I'd appreciate any comments though...
regards
sitaram
From: Storm-Olsen, Marius <hidden> Date: 2016-06-15 23:01:06
On 5/10/2014 8:04 PM, Sitaram Chamarty wrote:
On 05/11/2014 02:32 AM, Junio C Hamano wrote: That's an interesting
thread and it's recent too. However, it's about clone (though the
intro email mentions other commands also).
I'm specifically interested in push efficiency right now. When you
"fork" someone's repo to your own space, and you push your fork to
the same server, it ought to be able to get most of the common
objects from disk (specifically, from the repo you forked), and only
what extra you did from the network.
...
I do have a way to do this in gitolite (haven't coded it yet; just
thinking). Gitolite lets you specify something to do before
git-*-pack runs, and I was planning something like this:
And here you're poking the stick at the real solution to your problem.
Many of the Git repo managers will neatly set up a server-side repo
clone for you, with alternates into the original repo saving both
network and disk I/O.
So your work flow would instead be:
1. Fork repo on server
2. Remotely clone your own forked repo
I think it's more appropriate to handle this higher level operation
within the security context of a git repo manager, rather than directly
in git.
--
.marius
On 05/11/2014 07:04 AM, Storm-Olsen, Marius wrote:
On 5/10/2014 8:04 PM, Sitaram Chamarty wrote:
quoted
On 05/11/2014 02:32 AM, Junio C Hamano wrote: That's an interesting
thread and it's recent too. However, it's about clone (though the
intro email mentions other commands also).
I'm specifically interested in push efficiency right now. When you
"fork" someone's repo to your own space, and you push your fork to
the same server, it ought to be able to get most of the common
objects from disk (specifically, from the repo you forked), and only
what extra you did from the network.
...
quoted
I do have a way to do this in gitolite (haven't coded it yet; just
thinking). Gitolite lets you specify something to do before
git-*-pack runs, and I was planning something like this:
And here you're poking the stick at the real solution to your problem.
Many of the Git repo managers will neatly set up a server-side repo
clone for you, with alternates into the original repo saving both
network and disk I/O.
Gitolite already has a "fork" command that does that (though it uses
"-l", not alternates). I specifically don't want to use alternates, and
I also specifically am looking for something that activates on a push --
in the situations I am looking to optimise, the clone already happened.
So your work flow would instead be:
1. Fork repo on server
2. Remotely clone your own forked repo
I think it's more appropriate to handle this higher level operation
within the security context of a git repo manager, rather than directly
in git.
Yes, because of the "read access" check in my suggested procedure to
handle this. (Otherwise this is as valid as the plan suggested for
clone in Junior's email in [1]).
[1]: http://thread.gmane.org/gmane.comp.version-control.git/243918/focus=245397
I will certainly be doing this in gitolite. The point of my post was to
validate the flow with the *git* experts in case they catch something I
missed, not to say "this should be done *in* git".
From: Storm-Olsen, Marius <hidden> Date: 2016-06-15 23:01:06
On 5/10/2014 9:10 PM, Sitaram Chamarty wrote:
On 05/11/2014 07:04 AM, Storm-Olsen, Marius wrote:
quoted
On 5/10/2014 8:04 PM, Sitaram Chamarty wrote: Many of the Git repo
managers will neatly set up a server-side repo clone for you, with
alternates into the original repo saving both network and disk
I/O.
Gitolite already has a "fork" command that does that (though it uses
"-l", not alternates). I specifically don't want to use alternates,
and I also specifically am looking for something that activates on a
push -- in the situations I am looking to optimise, the clone already
happened.
You can probably get the managers to do a fork without alternates too.
Also, it doesn't matter if you have already cloned from the original
repo remotely. If you use the git manager to clone the original repo on
the server, and you push to your new repo, only your changes will go
back over the wire. The git protocol will figure out only which objects
are missing to complete the new HEAD, and send those.
So
1. Clone remote repo
2. Hack hack hack
3. Fork repo on server
4. Push changes to your own remote repo
is equally efficient.
quoted
So your work flow would instead be:
1. Fork repo on server
2. Remotely clone your own forked repo
I think it's more appropriate to handle this higher level operation
within the security context of a git repo manager, rather than directly
in git.
Yes, because of the "read access" check in my suggested procedure to
handle this. (Otherwise this is as valid as the plan suggested for
clone in Junior's email in [1]).
It's similar, but security issues come into play due to the swapped
direction, which is why I think it's wrong to place it in the push
command. Now, having the 'borrow' complement to 'reference' in Git seems
like a good idea, and should work for your case too, but IMO should be
configured with in the security context of the repo manager, and not on
an individual push. *shrug*
On 05/11/2014 08:41 AM, Storm-Olsen, Marius wrote:
On 5/10/2014 9:10 PM, Sitaram Chamarty wrote:
1. Clone remote repo
2. Hack hack hack
3. Fork repo on server
4. Push changes to your own remote repo
is equally efficient.
Your suggestions are good for a manual setup where the target repo
doesn't already exist.
But what I was looking for was validation from git.git folks of the idea
of replicating what "git clone -l" does, for an *existing* repo.
For example, I'm assuming that bringing in only the objects -- without
any of the refs pointing to them, making them all dangling objects --
will still allow the optimisation to occur (i.e., git will still say "oh
yeah I have these objects, even if they're dangling so I won't ask for
them from the pusher" and not "oh these are dangling objects; so I don't
recognise them from this perspective -- you'll have to send me those
again").
[1]: for any gitolite-aware folks reading this: this involves mirroring,
bringing a new mirror into play, normal repos, wild repos, and on and
on...