The --prune option to fetch added in v1.6.5-8-gf360d84 seems to be
around 20-30x slower than the equivalent operation with git remote
prune. I'm wondering if I'm missing something and fetch does something
more, but it doesn't seem so.
To test this clone git.git, create 1000 branches it in, create two
local clones of that clone and then delete the 1000 branches in the
original. I have a script to do this at
https://gist.github.com/avar/497c8c8fbd641fb756ef
Then in each of the clones:
$ git branch -a|wc -l; time (~/g/git/git-fetch --prune origin
/dev/null 2>&1); git branch -a | wc -l
1003
real 0m3.337s
user 0m2.996s
sys 0m0.336s
3
$ git branch -a|wc -l; time (~/g/git/git-remote prune origin
/dev/null 2>&1); git branch -a | wc -l
1003
real 0m0.067s
user 0m0.020s
sys 0m0.040s
3
Both of these ends up doing a "git fetch", so it's not that. I'm quite
rusty in C profiling but here's a gprof of the git-fetch command:
$ gprof ~/g/git/git-fetch|head -n 20
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
26.42 0.33 0.33 1584583 0.00 0.00 strbuf_getwholeline
14.63 0.51 0.18 90601347 0.00 0.00 strbuf_grow
13.82 0.68 0.17 1045676 0.00 0.00 find_pack_entry_one
8.13 0.78 0.10 1050062 0.00 0.00 check_refname_format
6.50 0.86 0.08 1584675 0.00 0.00 get_sha1_hex
5.69 0.93 0.07 2100529 0.00 0.00 starts_with
3.25 0.97 0.04 1044043 0.00 0.00 refname_is_safe
3.25 1.01 0.04 8007 0.00 0.00 get_packed_ref_cache
2.44 1.04 0.03 2605595 0.00 0.00 search_ref_dir
2.44 1.07 0.03 1040500 0.00 0.00 peel_entry
1.63 1.09 0.02 2632661 0.00 0.00 get_ref_dir
1.63 1.11 0.02 1044043 0.00 0.00 create_ref_entry
1.63 1.13 0.02 8024 0.00 0.00 do_for_each_entry_in_dir
0.81 1.14 0.01 2155105 0.00 0.00 memory_limit_check
0.81 1.15 0.01 1580503 0.00 0.00 sha1_to_hex
And of the git-remote command:
$ gprof ~/g/git/git-remote|head -n 20
Flat profile:
Each sample counts as 0.01 seconds.
no time accumulated
% cumulative self self total
time seconds seconds calls Ts/call Ts/call name
0.00 0.00 0.00 197475 0.00 0.00 strbuf_grow
0.00 0.00 0.00 24214 0.00 0.00 sort_ref_dir
0.00 0.00 0.00 24190 0.00 0.00 search_ref_dir
0.00 0.00 0.00 21661 0.00 0.00 memory_limit_check
0.00 0.00 0.00 20236 0.00 0.00 get_ref_dir
0.00 0.00 0.00 9187 0.00 0.00 xrealloc
0.00 0.00 0.00 7048 0.00 0.00 strbuf_add
0.00 0.00 0.00 6348 0.00 0.00 do_xmalloc
0.00 0.00 0.00 6126 0.00 0.00 xcalloc
0.00 0.00 0.00 6056 0.00 0.00 cleanup_path
0.00 0.00 0.00 6050 0.00 0.00 get_git_dir
0.00 0.00 0.00 6050 0.00 0.00 vsnpath
0.00 0.00 0.00 5554 0.00 0.00 config_file_fgetc
Aside from the slowness of git-fetch it seems git-remote can be sped
up quite a bit by more aggressively allocating a larger string buffer
from the get-go.
From: Jeff King <hidden> Date: 2016-06-15 23:04:00
On Fri, Mar 06, 2015 at 05:48:39PM +0100, Ævar Arnfjörð Bjarmason wrote:
The --prune option to fetch added in v1.6.5-8-gf360d84 seems to be
around 20-30x slower than the equivalent operation with git remote
prune. I'm wondering if I'm missing something and fetch does something
more, but it doesn't seem so.
"git fetch --prune" is "do a normal fetch, and also prune anything
necessary". "git remote prune" is "ls-remote the other side and see if
there is anything we can prune; do not touch anything else".
If your fetch is a noop (i.e., the other side has not advanced any
branches), the outcome is the same. But perhaps fetch is doing more
work to find out that it is a noop.
One way to measure that would be to see how expensive a noop "git fetch"
is (if it's expensive, then there is room to improve there. If not, then
it is the pruning itself that is expensive).
But just guessing (I do not have time to dig in deeper right now), and
seeing this:
We spend a lot of time checking refs here. Probably this comes from
writing the `packed-refs` file out 1000 times in your example, because
fetch handles each ref individually. Whereas since c9e768b (remote:
repack packed-refs once when deleting multiple refs, 2014-05-23),
git-remote does it in one pass.
Now that we have ref_transaction_*, I think if git-fetch fed all of the
deletes (along with the updates) into a single transaction, we would get
the same optimization for free. Maybe that is even part of some of the
pending ref_transaction work from Stefan or Michael (both cc'd). I
haven't kept up very well with what is cooking in pu.
-Peff
From: Michael Haggerty <hidden> Date: 2016-06-15 23:04:12
On 03/06/2015 11:59 PM, Jeff King wrote:
On Fri, Mar 06, 2015 at 05:48:39PM +0100, Ævar Arnfjörð Bjarmason wrote:
quoted
The --prune option to fetch added in v1.6.5-8-gf360d84 seems to be
around 20-30x slower than the equivalent operation with git remote
prune. I'm wondering if I'm missing something and fetch does something
more, but it doesn't seem so.
[...]
We spend a lot of time checking refs here. Probably this comes from
writing the `packed-refs` file out 1000 times in your example, because
fetch handles each ref individually. Whereas since c9e768b (remote:
repack packed-refs once when deleting multiple refs, 2014-05-23),
git-remote does it in one pass.
Now that we have ref_transaction_*, I think if git-fetch fed all of the
deletes (along with the updates) into a single transaction, we would get
the same optimization for free. Maybe that is even part of some of the
pending ref_transaction work from Stefan or Michael (both cc'd). I
haven't kept up very well with what is cooking in pu.
I am looking into this now.
For pruning, we can't use a ref_transaction as it is currently
implemented because it would fail if any of the reference deletions
failed. But in this case I think if any deletions fail, we would prefer
to emit a warning but keep going.
I'm trying to decide whether to have a separate function in the refs API
to "delete as many of the following refs as possible", or whether to add
a flag to ref_transaction_update() that says "try this update, but don't
fail the transaction if it fails". The latter would probably be more
work because we would need to invent a way to return multiple error
messages from a single transaction.
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
From: Jeff King <hidden> Date: 2016-06-15 23:04:12
On Thu, Mar 19, 2015 at 03:49:08PM +0100, Michael Haggerty wrote:
For pruning, we can't use a ref_transaction as it is currently
implemented because it would fail if any of the reference deletions
failed. But in this case I think if any deletions fail, we would prefer
to emit a warning but keep going.
I'm trying to decide whether to have a separate function in the refs API
to "delete as many of the following refs as possible", or whether to add
a flag to ref_transaction_update() that says "try this update, but don't
fail the transaction if it fails". The latter would probably be more
work because we would need to invent a way to return multiple error
messages from a single transaction.
Hmm, yeah. That is sort of antithetical to the original purpose of ref
transactions (which was atomicity). Given the way it is implemented now,
I don't think it would be too big a deal to keep a per-ref status (one
per "struct ref_upate"). But I would worry in the long run that it
makes things much harder on proposed ref backends, which otherwise can
consider a transaction an atomic unit.
OTOH, I do not think there is an abstracted way to do this _without_
the ref backend knowing about it. You cannot just do a series of
transactions; that defeats the purpose. It is a bit of a layering
violation that builtin/remote.c (which does this optimization already)
calls repack_without_refs directly.
-Peff