git fetch slow as molasses due to tag downloading

5 messages, 4 authors, 2016-08-11 · open the first message on its own page

git fetch slow as molasses due to tag downloading

From: Han-Wen Nienhuys <hidden>
Date: 2016-08-11 20:35:22

Hello,

just upgraded our autobuilder from 1.4.3 to 1.4.4.2.

Now, our standard download command comes to a complete halt.  Judging
from the "ps -ef" apparently, it does 

   git-show-ref --verify --quiet -- [TAG]

This is done for every one of the 1500 tags that are in my repository. 
At approx 20 tags per second this takes an awful lot of time. 

1. Is this necessary? 

2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref
invocation be potentially quicker?


-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen

Re: git fetch slow as molasses due to tag downloading

From: Jakub Narebski <hidden>
Date: 2016-08-11 19:20:01

Junio C Hamano wrote:
Johannes Schindelin [off-list ref] writes:
quoted
IMHO this should be solved as a filter: "git-show-ref --stdin 
--show-invalid". Thus, git does not have to traverse _every_ ref for 
_every_ incoming tag.
That sounds like a too specialized hack to me.  We should first
speed up the general "--verify $ref" case; as you corrected me
earlier it still has a useless loop.

If it is still too slow (which I suspect it could be the case,
with fork+exec overhead), we should _also_ pursue the filter
approach, but even then I think "filter out the valid ones" is a
specialized hack, if you mean "show only the invalid one's
names" by --show-invalid.

It would make sense to do

        $ git show-ref --show-invalid v1.0.0 v2.6.18
        f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
        - refs/tags/v2.6.18

(and its equivalent to take refs from --stdin) though.
Nice idea. And having '-' is probably better than using
00000000000000000000000000000000000000000 as for non-existing objects.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: git fetch slow as molasses due to tag downloading

From: Jakub Narebski <hidden>
Date: 2016-08-11 19:23:43

Johannes Schindelin wrote:
IMHO this should be solved as a filter: "git-show-ref --stdin 
--show-invalid". Thus, git does not have to traverse _every_ ref for 
_every_ incoming tag.
Can you make it so? TIA.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: git fetch slow as molasses due to tag downloading

From: Junio C Hamano <hidden>
Date: 2016-08-11 19:48:21

Johannes Schindelin [off-list ref] writes:
IMHO this should be solved as a filter: "git-show-ref --stdin 
--show-invalid". Thus, git does not have to traverse _every_ ref for 
_every_ incoming tag.
That sounds like a too specialized hack to me.  We should first
speed up the general "--verify $ref" case; as you corrected me
earlier it still has a useless loop.

If it is still too slow (which I suspect it could be the case,
with fork+exec overhead), we should _also_ pursue the filter
approach, but even then I think "filter out the valid ones" is a
specialized hack, if you mean "show only the invalid one's
names" by --show-invalid.

It would make sense to do

        $ git show-ref --show-invalid v1.0.0 v2.6.18
        f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
        - refs/tags/v2.6.18

(and its equivalent to take refs from --stdin) though.

Re: git fetch slow as molasses due to tag downloading

From: Johannes Schindelin <hidden>
Date: 2016-08-11 20:06:30

Hi,

On Thu, 14 Dec 2006, Han-Wen Nienhuys wrote:
just upgraded our autobuilder from 1.4.3 to 1.4.4.2.

Now, our standard download command comes to a complete halt.  Judging
quoted
from the "ps -ef" apparently, it does 
   git-show-ref --verify --quiet -- [TAG]

This is done for every one of the 1500 tags that are in my repository. 
At approx 20 tags per second this takes an awful lot of time. 

1. Is this necessary? 
Yes. The purpose is to check which tags have not yet been fetched.
2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref 
invocation be potentially quicker?
It is not efficient. But it cannot be solved like you propose, since it is 
inside a loop, and a "continue" is executed when the tag exists already.

IMHO this should be solved as a filter: "git-show-ref --stdin 
--show-invalid". Thus, git does not have to traverse _every_ ref for 
_every_ incoming tag.

Ciao,
Dscho
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help