Re: Efficient way to import snapshots?

Linus Torvalds [off-list ref] writes:

On Mon, 30 Jul 2007, Craig Boston wrote:

quoted

# On branch cvs_RELENG_4
nothing to commit (working directory clean)
git: 67.65 seconds

So I _seriously_ hope that about 65 of those 67 seconds was the "cvs 
update -d" or something like that. 

Anything that takes a minute in git is way way *way* too slow. Any 
half-way normal git operations should take less than a second.

I tried a git-add . on a TeX live tree (lots of itsy files).  About
80% of the processor time was wait.

I think that SHA1 is costly enough that the processor(s) could get
saturated when enough is done in parallel.

Potential for multithreading is here enough to make any Summer of Code
student weep with joy:

a) one thread for opendir/readdir at every directory level
b) one thread for stating the files in readdir order (more likely to
   correspond to the disk layout than sorted order)
c) one thread on each directory level doing a mergesort on a different
   link field (some merge passes could be parallelized even, but let's
   not get overexcited)
d) asynch I/O requesting the data for all files to be submitted to
   SHA1
e) one thread (but no more threads than there are CPUs) per
   independent file/tree for doing SHA1
f) asynch I/O reading the index sequentially
g) one thread doing a merge pass of already sorted stuff (this can
   start once the top level directory has been read and sorted
   completely, possibly having to stop until a complete subdirectory
   comes in).
h) asynch I/O writing out the results of the merge sequentially

In fact, git-ls-files|git-add --stdin is already exploiting a bit of
parallelism (and will probably profit from CFS by making much more use
of the buffering capacity of the pipe).  It is counterintuitive that
hand-built chains work more efficiently than explicit git commands.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum