Re: cygwin, 44k files: how to commit only index?

fork0@t-online.de (Alex Riesen) writes:

But aside from me trying ignoreState, can anyone help me with that
question regarding checking if the index is any different from HEAD?

Comparing index and HEAD should be cheap on a system with slow
lstat(), I think, as "git-diff-index --cached HEAD" should just
ignore the working tree altogether.  Is that what you want?

On 12/8/06, Alex Riesen [off-list ref] wrote:

On 12/8/06, Junio C Hamano [off-list ref] wrote:

quoted

In 'pu' (jc/diff topic), I have a very generic code to walk the
index, working tree and zero or more trees in parallel, taking
advantage of cache-tree.  If somebody is interested to learn the
internals of git, some of the code could be lifted from there
and simplified to walk just the index and a single tree, and I
think that would optimize "diff-index --cached" quite a bit.

Will try to look at it.

And now I'm playing with that (against test-para.c from pu).
I expect it to be broken by that webGmail, so it may not
apply to anything, but you'll get the idea. More clearly than
from me trying to explain.

commit 83642cdaca6dc1a2f94aa41923bc9e8f02d0e12f
Author: Alex Riesen [off-list ref]
Date:   Fri Dec 8 09:38:18 2006 +0100

    add --quiet to test-para: stop at the first difference

diff --git a/test-para.c b/test-para.c
index bce5f0c..99d3792 100644
--- a/test-para.c
+++ b/test-para.c

@@ -21,6 +21,7 @@ int main(int ac, const char **av)
 	unsigned char trees[64][20];
 	int num_tree = 0, i, using_head, show_all = 0;
 	int index_wanted = 1, work_wanted = 1, tree_wanted = 0;
+	int quiet = 0;
 	const char *prefix;
 	const char **pathspec;

@@ -43,6 +44,8 @@ int main(int ac, const char **av)
 			work_wanted = 0;
 		else if (!strcmp(av[1] + 2, "no-index"))
 			index_wanted = 0;
+		else if (!strcmp(av[1] + 2, "quiet"))
+			quiet = 1;
 		else if (!av[1][2])
 			break;
 		else

@@ -118,7 +121,9 @@ int main(int ac, const char **av)
 				show_one(z, e->name, e->namelen,
 					 e->hash, e->mode);
 			}
-			else
+			else {
+				if (quiet)
+					exit(1);
 				for (i = 0; i < w.num_trees + 2; i++) {
 					char numbuf[10];

@@ -149,6 +154,7 @@ int main(int ac, const char **av)
 					show_one(z, e->name, e->namelen,
 						 e->hash, e->mode);
 				}
+			}
 		}

fork0@t-online.de (Alex Riesen) writes:

yes, except that it'll compare the whole trees. Could I make it stop
at first mismatch? "-q|--quiet" for git-diff-index perhaps?
It's just not only stat, but also, open, read, mmap (yes, I try to use
it for packs) and close are really slow here as well.

That sounds like optimizing for a wrong case -- you expect the
index to match HEAD and trying to catch mistakes by detecting
a mismatch, right?

Having said that, I should point out that it is a low hanging
fruit to optimize "diff-index --cached" for cases where index
is expected to mostly match HEAD.

The current code for "diff-index --cached" reads the whole tree
into the index as stage #1 entries (diff-lib.c::run_diff_index),
and then compares stage #0 (from the original index contents)
and stage #1 (the tree parameter from the command line).  Even
if you stop at the first mismatch, you would already have paid
the overhead to open and read all tree objects before even
starting the comparison.

However, this code is from the ancient time before cache-tree
was introduced in the index.  If the index is expected to mostly
match HEAD, most of the cache-tree nodes are up-to-date, and
whole subtree can be skipped with a single comparison between
two tree SHA-1s at a shallower level of the directory tree.

In 'pu' (jc/diff topic), I have a very generic code to walk the
index, working tree and zero or more trees in parallel, taking
advantage of cache-tree.  If somebody is interested to learn the
internals of git, some of the code could be lifted from there
and simplified to walk just the index and a single tree, and I
think that would optimize "diff-index --cached" quite a bit.

A very unscientific test of running in the kernel repository I
just pulled (hot cache) on my box is:

$ /usr/bin/time git diff-index -r --cached --abbrev v2.6.19 >/tmp/1
0.91user 0.20system 0:01.12elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+10949minor)pagefaults 0swaps

while the para-walk to produce the moral equivalent is:

$ /usr/bin/time test-para --no-work v2.6.19 >/tmp/2
0.11user 0.02system 0:00.13elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4524minor)pagefaults 0swaps

On 12/8/06, Junio C Hamano [off-list ref] wrote:

quoted

yes, except that it'll compare the whole trees. Could I make it stop
at first mismatch? "-q|--quiet" for git-diff-index perhaps?
It's just not only stat, but also, open, read, mmap (yes, I try to use
it for packs) and close are really slow here as well.

That sounds like optimizing for a wrong case -- you expect the
index to match HEAD and trying to catch mistakes by detecting
a mismatch, right?

I expect the index to differ from HEAD. The test is to avoid the mistake
of doing an empty commit.

Having said that, I should point out that it is a low hanging
fruit to optimize "diff-index --cached" for cases where index
is expected to mostly match HEAD.

The current code for "diff-index --cached" reads the whole tree
into the index as stage #1 entries (diff-lib.c::run_diff_index),
and then compares stage #0 (from the original index contents)
and stage #1 (the tree parameter from the command line).  Even
if you stop at the first mismatch, you would already have paid
the overhead to open and read all tree objects before even
starting the comparison.

But I don't have to pay for the overhead of comparing all
entries, if I can stop at first mismatch and exit with non-0.
I think it'd make a difference (at least some difference).
But, if we could avoid loading of the entries which
will be never compared anyway, the speedup will be
of course more substantial...

In 'pu' (jc/diff topic), I have a very generic code to walk the
index, working tree and zero or more trees in parallel, taking
advantage of cache-tree.  If somebody is interested to learn the
internals of git, some of the code could be lifted from there
and simplified to walk just the index and a single tree, and I
think that would optimize "diff-index --cached" quite a bit.

Junio C Hamano, Thu, Dec 07, 2006 23:29:54 +0100:

quoted

But aside from me trying ignoreState, can anyone help me with that
question regarding checking if the index is any different from HEAD?

Comparing index and HEAD should be cheap on a system with slow
lstat(), I think, as "git-diff-index --cached HEAD" should just
ignore the working tree altogether.  Is that what you want?

yes, except that it'll compare the whole trees. Could I make it stop
at first mismatch? "-q|--quiet" for git-diff-index perhaps?
It's just not only stat, but also, open, read, mmap (yes, I try to use
it for packs) and close are really slow here as well.