Dennis Kaarsemaker [off-list ref] writes:
When cloning a repo with --mirror, and adding more remotes later,
get_stale_heads for origin would mark all refs from other repos as stale. In
this situation, with refs-src and refs->dst both equal to refs/*, we should
ignore refs/remotes/* when looking for stale refs to prevent this from
happening.
I do not think it is a right solution to single out refs/remotes/*.
Going back to your original example:
[remote "origin"]
url = git://github.com/git/git.git
fetch = +refs/*:refs/*
mirror = true
[remote "peff"]
url = git://github.com/peff/git.git
fetch = +refs/heads/*:refs/remotes/peff/*
Wouldn't you obtain "refs/remotes/github/html" from your "origin"
via "git pull origin"? What happens to your local copy of that ref,
when it goes away from the origin and then you try to "fetch --prune
origin" the next time with this patch (and without this patch)?
What should happen?
What if you had this instead of the above version of remote.peff.*?
[remote "peff"]
url = git://github.com/peff/git.git
fetch = +refs/heads/*:refs/remotes/github/*
I think this is an unsolvable problem, and I _think_ the root cause
of the issue is the configuration above that allows the RHS of
different fetch refspecs to overlap. refs/* is more generic and
covers refs/remotes/peff/* and refs/remotes/github/*. You cannot
even know, just by looking at "origin" and your local repository,
if refs/remotes/github/html you have should go away or it might have
come from somewhere else.
The best we _could_ do, without contacting all the defined remotes,
is probably to check each ref that we did not see from "origin" (for
example, you find "refs/remotes/peff/frotz" that your origin does
not have) and see if it could match RHS of fetch refspec of somebody
else (e.g. RHS of "refs/heads/*:refs/remotes/peff/*" matches that
ref). Then we can conclude that refs/remotes/peff/frotz _might_
have come from Peff's repository and not from "origin", and then we
can optionally issue a warning and refrain from removing it.
This inevitably will have false positives and leave something that
did originally came from "origin", because peff may no longer have
'frotz' branch in his repository. I do not think we can do better
than that, because we are trying to see if we can improve things
without having to contact all the remotes.
But if you go that route, the logic needs to go the same way when
you are pruning against 'peff', and anything that you do not see in
his repository right now but you have in refs/remotes/peff/ cannot
be pruned, because it might have come from your origin via more
generic refs/*:refs/* mapping. It follows that you could never
prune anything under refs/remotes/peff/* hierarchy.
You could introduce a "assume that more specific mapping never
overlaps with a more generic mapping" rule (i.e. refs/* from RHS of
remote.origin.fetch is more generic than refs/remotes/peff/* from
RHS of remote.peff.fetch, and assume everything that you see in your
local refs/remotes/peff/* came from peff and not from origin, I
think, but at that point, is it worth the possible complexity to
code that rule in the prune codepath and brittleness of that
assumption that your origin will never add a new ref under that
hierarchy, e.g. refs/remotes/peff/xyzzy?
So, I dunno.
quoted hunk
Signed-off-by: Dennis Kaarsemaker <redacted>
---
remote.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/remote.c b/remote.c
index e71f66d..863c183 100644
--- a/remote.c
+++ b/remote.c
@@ -1884,6 +1884,7 @@ struct stale_heads_info {
struct ref **stale_refs_tail;
struct refspec *refs;
int ref_count;
+ int ignore_remotes;
};
static int get_stale_heads_cb(const char *refname,@@ -1903,7 +1904,8 @@ static int get_stale_heads_cb(const char *refname,
* remote we consider it to be stale.
*/
if (!((flags & REF_ISSYMREF) ||
- string_list_has_string(info->ref_names, query.src))) {
+ string_list_has_string(info->ref_names, query.src) ||
+ (info->ignore_remotes && !prefixcmp(refname, "refs/remotes/")))) {
struct ref *ref = make_linked_ref(refname, &info->stale_refs_tail);
hashcpy(ref->new_sha1, sha1);
}@@ -1917,6 +1919,8 @@ struct ref *get_stale_heads(struct refspec *refs, int ref_count, struct ref *fet
struct ref *ref, *stale_refs = NULL;
struct string_list ref_names = STRING_LIST_INIT_NODUP;
struct stale_heads_info info;
+ if(!strcmp(refs->src, "refs/*") && !strcmp(refs->dst, "refs/*"))
+ info.ignore_remotes = 1;
info.ref_names = &ref_names;
info.stale_refs_tail = &stale_refs;
info.refs = refs;
(Sorry, I sent v2 before seeing this mail)
On do, 2013-06-20 at 15:46 -0700, Junio C Hamano wrote:
Dennis Kaarsemaker [off-list ref] writes:
quoted
When cloning a repo with --mirror, and adding more remotes later,
get_stale_heads for origin would mark all refs from other repos as stale. In
this situation, with refs-src and refs->dst both equal to refs/*, we should
ignore refs/remotes/* when looking for stale refs to prevent this from
happening.
I do not think it is a right solution to single out refs/remotes/*.
Going back to your original example:
[remote "origin"]
url = git://github.com/git/git.git
fetch = +refs/*:refs/*
mirror = true
[remote "peff"]
url = git://github.com/peff/git.git
fetch = +refs/heads/*:refs/remotes/peff/*
Wouldn't you obtain "refs/remotes/github/html" from your "origin"
via "git pull origin"? What happens to your local copy of that ref,
when it goes away from the origin and then you try to "fetch --prune
origin" the next time with this patch (and without this patch)?
git pull origin gives me refs/html in this case. I did not try fetch
--prune, but prune origin DTRT: if the html branch goes away at the
origin, it goes away locally. Both with and without this patch.
It's refs/remotes/peff/somebranch that in this case *also* goes away
without this patch, but is untouched with this patch
What should happen?
Exactly that.
What if you had this instead of the above version of remote.peff.*?
[remote "peff"]
url = git://github.com/peff/git.git
fetch = +refs/heads/*:refs/remotes/github/*
That doesn't change anything.
I think this is an unsolvable problem, and I _think_ the root cause
of the issue is the configuration above that allows the RHS of
different fetch refspecs to overlap. refs/* is more generic and
covers refs/remotes/peff/* and refs/remotes/github/*. You cannot
even know, just by looking at "origin" and your local repository,
if refs/remotes/github/html you have should go away or it might have
come from somewhere else.
The best we _could_ do, without contacting all the defined remotes,
is probably to check each ref that we did not see from "origin" (for
example, you find "refs/remotes/peff/frotz" that your origin does
not have) and see if it could match RHS of fetch refspec of somebody
else (e.g. RHS of "refs/heads/*:refs/remotes/peff/*" matches that
ref). Then we can conclude that refs/remotes/peff/frotz _might_
have come from Peff's repository and not from "origin", and then we
can optionally issue a warning and refrain from removing it.
I like that idea, though I also like the simplicity of simply singling
out "remotes" as that's where normal remotes usually sit. And don't
forget about tags (see patch v2).
This inevitably will have false positives and leave something that
did originally came from "origin", because peff may no longer have
'frotz' branch in his repository. I do not think we can do better
than that, because we are trying to see if we can improve things
without having to contact all the remotes.
But then the ref would have to be called "refs/remotes/peff/frotz"
upstream. Hmm, that is of course completely possible: cloning something
that's already a clone.
But if you go that route, the logic needs to go the same way when
you are pruning against 'peff', and anything that you do not see in
his repository right now but you have in refs/remotes/peff/ cannot
be pruned, because it might have come from your origin via more
generic refs/*:refs/* mapping. It follows that you could never
prune anything under refs/remotes/peff/* hierarchy.
You could introduce a "assume that more specific mapping never
overlaps with a more generic mapping" rule (i.e. refs/* from RHS of
remote.origin.fetch is more generic than refs/remotes/peff/* from
RHS of remote.peff.fetch, and assume everything that you see in your
local refs/remotes/peff/* came from peff and not from origin, I
think, but at that point, is it worth the possible complexity to
code that rule in the prune codepath and brittleness of that
assumption that your origin will never add a new ref under that
hierarchy, e.g. refs/remotes/peff/xyzzy?
So, I dunno.
Yeah, I'm starting to think this is not such a good idea. How about plan
B: issuing a warning when adding a remote with a refspec that also
matches another remote's refspec?
Or plan C: add a per-remote pruneIgnore setting that in this case I
could set to refs/tags/* refs/remotes/* as I know it's correct? Could
even be combined with plan B.
--
Dennis Kaarsemaker
www.kaarsemaker.net
On Thu, Jun 20, 2013 at 03:46:20PM -0700, Junio C Hamano wrote:
Dennis Kaarsemaker [off-list ref] writes:
quoted
When cloning a repo with --mirror, and adding more remotes later,
get_stale_heads for origin would mark all refs from other repos as stale. In
this situation, with refs-src and refs->dst both equal to refs/*, we should
ignore refs/remotes/* when looking for stale refs to prevent this from
happening.
I do not think it is a right solution to single out refs/remotes/*.
Yeah, I agree.
Going back to your original example:
[remote "origin"]
url = git://github.com/git/git.git
fetch = +refs/*:refs/*
mirror = true
[remote "peff"]
url = git://github.com/peff/git.git
fetch = +refs/heads/*:refs/remotes/peff/*
There is a fundamental namespace conflict here: one remote is claiming
the whole refs/* namespace, and another remote is claiming some subset.
"fetch --prune" is only one type of problem you can have; you might also
overwrite stuff from the "peff" remote with stuff from the "origin"
remote (if it happens to have "refs/remotes/peff/foo" itself).
I think this is an unsolvable problem, and I _think_ the root cause
of the issue is the configuration above that allows the RHS of
different fetch refspecs to overlap. refs/* is more generic and
covers refs/remotes/peff/* and refs/remotes/github/*. You cannot
even know, just by looking at "origin" and your local repository,
if refs/remotes/github/html you have should go away or it might have
come from somewhere else.
Exactly.
The best we _could_ do, without contacting all the defined remotes,
is probably to check each ref that we did not see from "origin" (for
example, you find "refs/remotes/peff/frotz" that your origin does
not have) and see if it could match RHS of fetch refspec of somebody
else (e.g. RHS of "refs/heads/*:refs/remotes/peff/*" matches that
ref). Then we can conclude that refs/remotes/peff/frotz _might_
have come from Peff's repository and not from "origin", and then we
can optionally issue a warning and refrain from removing it.
I think this is just papering over the problem in one instance. What
happens when you _do_ have overlapping refs in the "origin" remote, and
you have a true conflict.
I wonder why Dennis wants to "refs/*:refs/*" in the first place. It
is not usually a useful thing to have in a non-bare repository, because
fetches will overwrite local work on branches. If he just wanted the
automatic "git push --mirror" setting, that does not depend on the fetch
refspec.
We made this distinction in "git remote --mirror={fetch,remote}", but I
don't think "git clone --mirror" ever learned about it.
-Peff
On do, 2013-06-20 at 19:08 -0400, Jeff King wrote:
I wonder why Dennis wants to "refs/*:refs/*" in the first place. It
is not usually a useful thing to have in a non-bare repository,
because fetches will overwrite local work on branches. If he just
wanted the automatic "git push --mirror" setting, that does not depend
on the fetch refspec.
I'm not doing that in non-bare repositories, neither do I use it for
pushing. It's for a continuous integration system, which never has any
locally created branches or commits, but does integrate things from
different remotes in some cases. The example with git.git is used
roughly as follows:
* git fetch all remotes (for most projects that will be 1 remote)
* rebuild reflogs from github events (or fetch via http/ssh)
* per push to next, check out to a separate $GIT_WORK_TREE and run make
test
* for the last push, also build and publish daily tarball+deb+rpm
Then, for further testing of local requirements:
* cherry-pick your jk/blame_tree branch
* test, build and install package
Given that this system works with clones of what should be considered
canonical copies of repositories, those remotes shouldn't have any
remotes defined themselves, so at least being able to configure prune to
ignore refs/remotes/* and refs/tags/* would help me a lot.
--
Dennis Kaarsemaker
www.kaarsemaker.net