Hi,
I'm using git's cvs import to track the Fedora kernel rpm repository,
and it has just passed a watermark that git cannot currently cope with.
The problem is the large number of branches and tags on this repository.
A tag for every build means that the number builds up rapidly, and
git-send-pack has an internal limit of 900 refs to query when building a
pack.
I hoped this wouldn't be a problem. In theory, I should just be able to
push to a single refspec and avoid that limit entirely. But no ---
unfortunately, git is determined to push out the objects for *every* ref
that matches between src and dst when building the pack, even if it's
not actually going to update the remote ref. And in building the
git-rev-list argument list for that colossal update, it exceeds the
internal limit of 900 refs in exec_rev_list().
I think exec_rev_list() is doing the wrong thing here. If I specify an
explicit refspec list for git-send-pack, then when send_pack() calls
match_refs(), we break out to match_explicit_refs() and set
dst->peer_ref only for the ref[s] which match the refspec. send_pack()
then skips all other refs by doing a
if (!ref->peer_ref)
continue;
Unfortunately, exec_rev_list() is missing this, and it tries to ask
git-rev-list for the commit objects of *every* ref on the remote_refs
list, even if they have been explicitly excluded by match_refs() and
have no peer_ref. So with this huge repository, I can't even push a
single refspec without bumping into the limit of 900 refs.
The immediate consequence to me is that I can't push to this particular
repository. But if I'm following the code right, a consequence is that
git-send-pack is accidentally sending all objects for refs that the
remote end is prepared to receive, even if we've supplied a refspec
asking for only a subset of those refs to actually be updated.
I think we can fix this just by adding a test for ref->peer_ref to
exec_rev_list(). That seems to fix the immediate problem for me, at
least. Patch to follow.
--Stephen
send_pack() sends only those refs we've asked to be updated on the
destination---either via an explicit refspec or by matching local and
remote refs. But rev_list() builds an object list for *all* refs it
can find. For a tree with many tags/heads, this means that it is
impossible to push updates even to a single refspec, as exec_rev_list()
overflows its arg length limit.
Fix this by skipping refs with no peer_ref set in exec_rev_list().
send_pack() already skips it when sending refs; we need to skip it
when building the object list for the pack too.
Signed-off-by: Stephen Tweedie <redacted>
---
send-pack.c | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
4aa39c0920eea37987ca8a6b10861da7a87b5c14
diff --git a/send-pack.c b/send-pack.c
index 990be3f..d2a39d9 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -42,8 +42,10 @@ static void exec_rev_list(struct ref *re
args[i++] = "rev-list"; /* 0 */
args[i++] = "--objects"; /* 1 */
- while (refs) {
+ for (; refs; refs = refs->next) {
char *buf = malloc(100);
+ if (!refs->peer_ref)
+ continue;
if (i > 900)
die("git-rev-list environment overflow");
if (!is_zero_sha1(refs->old_sha1) &&@@ -56,7 +58,6 @@ static void exec_rev_list(struct ref *re
args[i++] = buf;
snprintf(buf, 50, "%s", sha1_to_hex(refs->new_sha1));
}
- refs = refs->next;
}
args[i] = NULL;
execv_git_cmd(args);
--
1.2.2.g6643-dirty
"Stephen C. Tweedie" [off-list ref] writes:
send_pack()
then skips all other refs by doing a
if (!ref->peer_ref)
continue;
Unfortunately, exec_rev_list() is missing this, and it tries to ask
git-rev-list for the commit objects of *every* ref on the remote_refs
list, even if they have been explicitly excluded by match_refs() and
have no peer_ref. So with this huge repository, I can't even push a
single refspec without bumping into the limit of 900 refs.
IIRC, the distinction was deliberate. send_pack() excludes
what did not match because it does not want to send them.
rev_list() adds what we know they have to "do not bother to
send" list to make the resulting pack smaller. The time where
it matters most is when you are pushing a new branch head (or a
tag).
I think the exec_rev_list logic should be taught to first
include all the positive refs (i.e. the ones we are going to
send), and then as many the negative refs (i.e. the ones we know
they have), from newer to older, as they fit without triggering
"argument list too long".
Another, probably conceptually cleaner alternative might be to
allow rev-list to read from its stdin so that we do not have to
worry about the argument list issues.