From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:59
Here are the topics that have been cooking. Commits prefixed
with '-' are only in 'pu' while commits prefixed with '+' are
in 'next'. The topics list the commits in reverse chronological
order.
* sp/run-command (Mon Mar 12 19:00:29 2007 -0400) 9 commits
+ Use run_command within send-pack
+ Use run_command within receive-pack to invoke index-pack
+ Use run_command within merge-index
+ Use run_command for proxy connections
+ Use RUN_GIT_CMD to run push backends
+ Correct new compiler warnings in builtin-revert
+ Replace fork_with_pipe in bundle with run_command
+ Teach run-command to redirect stdout to /dev/null
+ Teach run-command about stdout redirection
This is really internal clean-up without behaviour change (but I
suspect the error messages from failure cases might be
different). Good to flush these to 'master' before 1.5.1-rc1.
* dz/mailinfo (Mon Mar 12 15:52:07 2007 -0400) 3 commits
+ Add a couple more test cases to the suite.
+ restrict the patch filtering
+ builtin-mailinfo.c infrastrcture changes
The mailinfo implementation in 'master' I punted to do
complicated multi-part and Don Zickus rewrote much of the hacky
parts. The less hacky code of mine remains in the tree, the
happier I am. Should be in 'master' before 1.5.1-rc1.
* jc/repack (Fri Mar 9 03:52:12 2007 -0800) 1 commit
+ prepare_packed_git(): sort packs by age and localness.
This is to improve the access pattern when repository has many
small packfiles, as recent push/fetch tend to keep packs
unexploded. The idea is to check younger packs and local packs
before others when we iterate over .idx files to look for packed
objects from find_pack_entry(). I've repacked linux-2.6 kernel
repository so that it has one pack per one public tag (which is
a bit excessive -- it results in 70 or so small packs), and saw
"git log -r --raw v2.6.20.." got some speed-up in hot cache case
(4.4 seconds vs 5.3 seconds on average).
* jc/fetch (Sun Mar 4 15:36:08 2007 -0800) 15 commits
+ .gitignore: add git-fetch--tool
+ builtin-fetch--tool: fix reflog notes.
+ git-fetch: retire update-local-ref which is not used anymore.
+ builtin-fetch--tool: make sure not to overstep ls-remote-result
buffer.
+ fetch--tool: fix uninitialized buffer when reading from stdin
+ builtin-fetch--tool: adjust to updated sha1_object_info().
+ git-fetch--tool takes flags before the subcommand.
+ Use stdin reflist passing in git-fetch.sh
+ Use stdin reflist passing in parse-remote
+ Allow fetch--tool to read from stdin
+ git-fetch: rewrite expand_ref_wildcard in C
+ git-fetch: rewrite another shell loop in C
+ git-fetch: move more code into C.
+ git-fetch--tool: start rewriting parts of git-fetch in C.
+ git-fetch: split fetch_main into fetch_dumb and fetch_native
This is a partial C rewrite of heaviest part of git-fetch to
help fetching between repositories with hundreds of refs. I do
not like the way it is split, but it may be a good idea to throw
it in 'master' as it does not seem to regress anything and see
if there are other interested people who want to finish the
rewriting.
* pb/branch-track (Thu Mar 8 13:59:54 2007 -0800) 2 commits
+ Fix broken create_branch() in builtin-branch.
+ git-branch, git-checkout: autosetup for remote branch tracking
As I personally do not use "git branch --track", all I can say
is that this, with the fix-up patch already in, does not seem to
regress anything. Positive feedbacks requested before advancing
to 'master'.
* jb/per-user-exclude (Tue Feb 27 22:31:10 2007 -0500) 1 commit
+ add: Support specifying an excludes file with a configuration
variable
Same as above.
* ml/workdir (Sun Mar 11 22:29:06 2007 +0100) 3 commits
- use $GIT_DIR/workdir as working directory with $GIT_DIR
- introduce GIT_WORK_DIR environment variable
- rev-parse: --is-bare-repository option
Not in 'next' yet, but I think this one is ready to be tested.
We need testsuite for it before that happens, though.
* js/fetch-progress (Sun Feb 25 13:13:17 2007 -0800) 1 commit
+ git-fetch: add --quiet
This does not break anything, but I am not sure how useful it
would be.
* sb/fetch (Mon Mar 12 19:01:11 2007 -0700) 19 commits
+ git-fetch.sh:append_fetch_head() no longer has a remote_nick
argument
+ git-fetch: Split fetch and merge logic
I have a soft spot to anything that claims to be a clean-up, but
I suspect that the shell loop this series introduces may defeat
the git-fetch--tool optimization. Also I think having to base
the patch on this made Paolo's "dot is special token to mean
'git pull' merges from a local branch" needlessly complex (but I
haven't tried rewriting it myself without these two). Although
I merged these to 'next', I am considering to revert them.
* jc/pathattr (Thu Mar 1 01:20:21 2007 -0800) 5 commits
- pathattr: allow piping to external program.
- pathattr: read from git_config().
- git-show: use pathattr to run "display"
- pathattr: path based configuration of various attributes.
+ convert: add scaffolding for path based selection of conversion
routines.
Stalled.
* jc/merge-subtree (Thu Feb 15 16:32:45 2007 -0800) 1 commit
- A new merge stragety 'subtree'.
Stalled.
* jc/diff (Mon Dec 25 01:08:50 2006 -0800) 2 commits
- test-para: combined diff between HEAD, index and working tree.
- para-walk: walk n trees, index and working tree in parallel
Just a reference code.
* ml/workdir (Sun Mar 11 22:29:06 2007 +0100) 3 commits
- use $GIT_DIR/workdir as working directory with $GIT_DIR
- introduce GIT_WORK_DIR environment variable
- rev-parse: --is-bare-repository option
Not in 'next' yet, but I think this one is ready to be tested.
We need testsuite for it before that happens, though.
Will you apply the git-init patch too?
I did not write any tests yet, but I can try.
Here is what I thought about:
check that --work-dir overrides $GIT_WORK_DIR and both override
$GIT_DIR/workdir.
use a correct and an invalid path for:
$GIT_DIR/workdir:
file containing a relative and an absolute path
symlink pointing to an invalid path, a directory, a file
test a symlink to something else (e.g. device, fifo, ..) too?
directory
$GIT_WORK_DIR: relative and absolute path
and test what git does with git-rev-parse
--is-bare-repository
--show-prefix
--show-cdup
test git rev-parse --is-inside-git-dir
A symlink pointing to an invalid path is currently handled as if there
is no $GIT_DIR/workdir at all because stat returns ENOENT. Is this ok
or should git complain like it does for an invalid path when
$GIT_DIR/workdir is a file? We could also decide to ignore all
invalid workdir settings and handle this the same as being outside the
workdir.
* jc/fetch (Sun Mar 4 15:36:08 2007 -0800) 15 commits
+ .gitignore: add git-fetch--tool
+ builtin-fetch--tool: fix reflog notes.
+ git-fetch: retire update-local-ref which is not used anymore.
+ builtin-fetch--tool: make sure not to overstep ls-remote-result
buffer.
+ fetch--tool: fix uninitialized buffer when reading from stdin
+ builtin-fetch--tool: adjust to updated sha1_object_info().
+ git-fetch--tool takes flags before the subcommand.
+ Use stdin reflist passing in git-fetch.sh
+ Use stdin reflist passing in parse-remote
+ Allow fetch--tool to read from stdin
+ git-fetch: rewrite expand_ref_wildcard in C
+ git-fetch: rewrite another shell loop in C
+ git-fetch: move more code into C.
+ git-fetch--tool: start rewriting parts of git-fetch in C.
+ git-fetch: split fetch_main into fetch_dumb and fetch_native
This is a partial C rewrite of heaviest part of git-fetch to
help fetching between repositories with hundreds of refs. I do
not like the way it is split, but it may be a good idea to throw
it in 'master' as it does not seem to regress anything and see
if there are other interested people who want to finish the
rewriting.
As it happens I was planning to start looking at writing a builtin fetch
when I got home this evening ... the fetch--tool improvements have whetted
my appetite for speed ... ;)
* sb/fetch (Mon Mar 12 19:01:11 2007 -0700) 19 commits
+ git-fetch.sh:append_fetch_head() no longer has a remote_nick
argument
+ git-fetch: Split fetch and merge logic
I have a soft spot to anything that claims to be a clean-up, but
I suspect that the shell loop this series introduces may defeat
the git-fetch--tool optimization. Also I think having to base
the patch on this made Paolo's "dot is special token to mean
'git pull' merges from a local branch" needlessly complex (but I
haven't tried rewriting it myself without these two). Although
I merged these to 'next', I am considering to revert them.
I found that this series did introduce a regression, but not a serious
one. A null fetch went from ~30s to ~40s IIRC. I moved
canon_refs_list_for_fetch from git-parse-remote.sh to git-fetch--tool in
response, and was pretty much able to get back to where I was before - I
can send the patch if you want, I didn't think it that important.
--
Julian
---
The new Congressmen say they're going to turn the government around. I
hope I don't get run over again.
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:59
Junio C Hamano [off-list ref] writes:
Here are the topics that have been cooking.
* sb/fetch (Mon Mar 12 19:01:11 2007 -0700) 19 commits
+ git-fetch.sh:append_fetch_head() no longer has a remote_nick
argument
+ git-fetch: Split fetch and merge logic
I have a soft spot to anything that claims to be a clean-up, but
I suspect that the shell loop this series introduces may defeat
the git-fetch--tool optimization. Also I think having to base
the patch on this made Paolo's "dot is special token to mean
'git pull' merges from a local branch" needlessly complex (but I
haven't tried rewriting it myself without these two). Although
I merged these to 'next', I am considering to revert them.
I tried the "NULL fetch between 1000-refs repositories" test,
which prompted the git-fetch--tool work that was done on
jc/fetch topic in 'next', with the following versions:
(1) 1.5.0 (without any git-fetch--tool optimization)
(2) master (ditto)
(3) master with jc/fetch (but not sb/fetch topic)
(4) next ((3) plus sb/fetch and others)
The test scripts are at the end of this message. Both (1) and
(2) take 3 minutes 7 seconds wallclock time. (3) improves it
down to 15 seconds. (4) makes the operation spend 24 seconds
(the times are all on my primary machine x86-64 with 1GB, hot
cache and average of three runs each).
So the "Split fetch and merge" series hurts the performance
quite a bit. If it had enough "code clean-up" merit to warrant
this, I would say it probably is a cost we should bear, but I
personally do not see it.
Paolo recently worked on top of next to base the fake '.' remote
patch. This wants to allow:
[branch "foo"]
remote = .
merge = refs/heads/master
with an implicit (meaning, you do not have to have this in your
configuration):
[remote "."]
url = .
fetch = refs/*
so that you can say:
$ git checkout foo
$ git pull
to merge from the local 'master' branch.
I haven't reimplemented Paolo's patch on top of (3) above for
comparison, but I have a feeling that it would not have been
helped by the alleged clean-up value of "Split fetch and merge"
patch (iow, I do not think it would be the case that the code
got clearer to understand thanks to the clean-up).
What Paolo's patch needs to do is to bypass the actual fetch and
generate the following line in .git/FETCH_HEAD:
sha1-of-our-master <TAB> <TAB> branch 'master' of .
I even suspect that "Split fetch and merge", by introducing
FETCH_FETCHED and making FETCH_HEAD generated from it, made
Paolo's patch more difficult to do and the end result less
efficient.
So unless there is a convincing counterexample otherwise, I'd
like to revert the "Split fetch and merge" series.
-- >8 -- setting up test repositories -- >8 --
#!/bin/sh
rm -fr origin clone
mkdir origin
cd origin
git init
: >hello
git add hello
git commit -a -m 'initial'
i=0
while test $i -lt 500
do
git tag t$i
git branch b$i
i=$(($i+1))
done
: >bye
git add bye
git commit -a -m 'second'
while test $i -lt 1000
do
git tag t$i
git branch b$i
i=$(($i+1))
done
cd ..
-- >8 -- NULL fetch test -- >8 --
#!/bin/sh
cd clone
echo '* fetching'
time git fetch origin
* sb/fetch (Mon Mar 12 19:01:11 2007 -0700) 19 commits
+ git-fetch.sh:append_fetch_head() no longer has a remote_nick
argument
+ git-fetch: Split fetch and merge logic
I have a soft spot to anything that claims to be a clean-up, but
I suspect that the shell loop this series introduces may defeat
the git-fetch--tool optimization. Also I think having to base
the patch on this made Paolo's "dot is special token to mean
'git pull' merges from a local branch" needlessly complex (but I
haven't tried rewriting it myself without these two). Although
I merged these to 'next', I am considering to revert them.
I tried the "NULL fetch between 1000-refs repositories" test,
which prompted the git-fetch--tool work that was done on
jc/fetch topic in 'next', with the following versions:
(1) 1.5.0 (without any git-fetch--tool optimization)
(2) master (ditto)
(3) master with jc/fetch (but not sb/fetch topic)
(4) next ((3) plus sb/fetch and others)
The test scripts are at the end of this message. Both (1) and
(2) take 3 minutes 7 seconds wallclock time. (3) improves it
down to 15 seconds. (4) makes the operation spend 24 seconds
(the times are all on my primary machine x86-64 with 1GB, hot
cache and average of three runs each).
I think it is not fair, I wonder what would be the time with the merge
logic in sb/fetch in C. I'll try to make the git-fetch--tool
optimization.
So the "Split fetch and merge" series hurts the performance
quite a bit. If it had enough "code clean-up" merit to warrant
this, I would say it probably is a cost we should bear, but I
personally do not see it.
Paolo recently worked on top of next to base the fake '.' remote
patch. This wants to allow:
[branch "foo"]
remote = .
merge = refs/heads/master
with an implicit (meaning, you do not have to have this in your
configuration):
[remote "."]
url = .
fetch = refs/*
so that you can say:
$ git checkout foo
$ git pull
to merge from the local 'master' branch.
I haven't reimplemented Paolo's patch on top of (3) above for
comparison, but I have a feeling that it would not have been
helped by the alleged clean-up value of "Split fetch and merge"
patch (iow, I do not think it would be the case that the code
got clearer to understand thanks to the clean-up).
What Paolo's patch needs to do is to bypass the actual fetch and
generate the following line in .git/FETCH_HEAD:
sha1-of-our-master <TAB> <TAB> branch 'master' of .
I even suspect that "Split fetch and merge", by introducing
FETCH_FETCHED and making FETCH_HEAD generated from it, made
Paolo's patch more difficult to do and the end result less
efficient.
I think my patch to support this is independent of the "Split fetch and merge".
So unless there is a convincing counterexample otherwise, I'd
like to revert the "Split fetch and merge" series.
From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:59
"Santi Béjar" [off-list ref] writes:
quoted
I tried the "NULL fetch between 1000-refs repositories" test,
which prompted the git-fetch--tool work that was done on
jc/fetch topic in 'next', with the following versions:
(1) 1.5.0 (without any git-fetch--tool optimization)
(2) master (ditto)
(3) master with jc/fetch (but not sb/fetch topic)
(4) next ((3) plus sb/fetch and others)
The test scripts are at the end of this message. Both (1) and
(2) take 3 minutes 7 seconds wallclock time. (3) improves it
down to 15 seconds. (4) makes the operation spend 24 seconds
(the times are all on my primary machine x86-64 with 1GB, hot
cache and average of three runs each).
I think it is not fair,...
That's a very unexpected response. I personally do not think
the separation of FETCH_FETCHED made improvements to the code,
but the above numbers do not have anything to do with such
perhaps subjective ascetic judgement.
The comparison showed that the "Split" patch is a step backward
from the existing optimization hack that was specifically made
to solve an issue raised on the list, and you may not like the
numbers, but if you call that is "not fair", I do not know what
could be considered fair.
Yes, life is unfair, but I do not think that word applies to
this particular case.
I tried the "NULL fetch between 1000-refs repositories" test,
which prompted the git-fetch--tool work that was done on
jc/fetch topic in 'next', with the following versions:
(1) 1.5.0 (without any git-fetch--tool optimization)
(2) master (ditto)
(3) master with jc/fetch (but not sb/fetch topic)
(4) next ((3) plus sb/fetch and others)
The test scripts are at the end of this message. Both (1) and
(2) take 3 minutes 7 seconds wallclock time. (3) improves it
down to 15 seconds. (4) makes the operation spend 24 seconds
(the times are all on my primary machine x86-64 with 1GB, hot
cache and average of three runs each).
I think it is not fair,...
[...]
, and you may not like the
numbers, but if you call that is "not fair", I do not know what
could be considered fair.
I would consider fair the comparison you did not quote, a comparison
with the merge logic written in C. I know that (4) is a step backwards
in performance as it is now, and I understand that with those numbers
the "Split" patch must be reverted.
Santi