Re-organize "git-rev-list --objects" logic

The logic to calculate the full object list used to be very inter-twined 
with the logic that looked up the commits.

For no good reason - it's actually a lot simpler to just do that logic 
as a separate pass.

This improves performance a bit, and uses slightly less memory in my 
tests, but more importantly it makes the code simpler to work with and 
follow what it does.

The performance win is less than I had hoped for, but I get:

Before:

	[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
	13.64user 0.42system 0:14.13elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+47947minor)pagefaults 0swaps
	58945

After:

	[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
	11.80user 0.36system 0:12.16elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+42684minor)pagefaults 0swaps
	58945

ie it improved by 2 seconds, and took a 5000+ fewer pages (hey, that's 
20MB out of 174MB to go). And got the same number of objects (in theory, 
the more expensive one might find some more shared objects to avoid. In 
practice it obviously doesn't).

I know how to make it use _lots_ less memory, which will probably speed it 
up. But that's for another time, and I'd prefer to see this go in first.

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
----

diff --git a/rev-list.c b/rev-list.c
--- a/rev-list.c
+++ b/rev-list.c

@@ -231,8 +231,6 @@ static void mark_parents_uninteresting(s
 {
 	struct commit_list *parents = commit->parents;
 
-	if (tree_objects)
-		mark_tree_uninteresting(commit->tree);
 	while (parents) {
 		struct commit *commit = parents->item;
 		commit->object.flags |= UNINTERESTING;

@@ -272,29 +270,6 @@ static int everybody_uninteresting(struc
 			continue;
 		return 0;
 	}
-
-	/*
-	 * Ok, go back and mark all the edge trees uninteresting,
-	 * since otherwise we can have situations where a parent
-	 * that was marked uninteresting (and we never even had
-	 * to look at) had lots of objects that we don't want to
-	 * include.
-	 *
-	 * NOTE! This still doesn't mean that the object list is
-	 * "correct", since we may end up listing objects that
-	 * even older commits (that we don't list) do actually
-	 * reference, but it gets us to a minimal list (or very
-	 * close) in practice.
-	 */
-	if (!tree_objects)
-		return 1;
-
-	while (orig) {
-		struct commit *commit = orig->item;
-		if (!parse_commit(commit) && commit->tree)
-			mark_tree_uninteresting(commit->tree);
-		orig = orig->next;
-	}
 	return 1;
 }
 

@@ -370,6 +345,19 @@ static struct commit_list *find_bisectio
 	return best;
 }
 
+static void mark_edges_uninteresting(struct commit_list *list)
+{
+	for ( ; list; list = list->next) {
+		struct commit_list *parents = list->item->parents;
+
+		for ( ; parents; parents = parents->next) {
+			struct commit *commit = parents->item;
+			if (commit->object.flags & UNINTERESTING)
+				mark_tree_uninteresting(commit->tree);
+		}
+	}
+}
+
 static struct commit_list *limit_list(struct commit_list *list)
 {
 	struct commit_list *newlist = NULL;

@@ -388,6 +376,8 @@ static struct commit_list *limit_list(st
 		}
 		p = &commit_list_insert(commit, p)->next;
 	}
+	if (tree_objects)
+		mark_edges_uninteresting(newlist);
 	if (bisect_list)
 		newlist = find_bisection(newlist);
 	return newlist;

Hi,

On Thu, 15 Sep 2005, Linus Torvalds wrote:

I know how to make it use _lots_ less memory, which will probably speed it 
up. But that's for another time, and I'd prefer to see this go in first.

Mind to elaborate just a bit?

On Fri, 16 Sep 2005, Johannes Schindelin wrote:

On Thu, 15 Sep 2005, Linus Torvalds wrote:

quoted

I know how to make it use _lots_ less memory, which will probably speed it 
up. But that's for another time, and I'd prefer to see this go in first.

Mind to elaborate just a bit?

Right now we keep track of every single object we parse, and never free 
it.

Some of it is necessary to keep track of whether a SHA has been seen or
not, but: we also keep every single relation list around. They're small,
but they are _many_. We can trivially free those after following them.
That should get us back another few tens of megabytes easily. It's the
parent pointers, and the "struct tree_entry" lists.

Now, some memory usage is very fundamental, and we can't get rid of the
"struct object" for each thing we've listed (or each thing we've
determined needs to be pruned). We can perhaps shrink it some, but we'll
always need to have at a minimum the 20-byte sha1 of every object. So if
we list/blacklist a hundred thousand objects, there's no way we can avoid
using a couple of megs just to keep track of that fact in order to avoid
duplicates.

However, I _think_ the majority of the memory use is in the refs. Even a 
full kernel tree only has about 95,000 objects, but we have many _many_ 
more of these relationship links. We've got ~9,000 commits, and pretty 
much each of them will have a different root tree with 20+ entries in it, 
and even when most of them end up pointing to the same blobs/trees (which 
is why we only have 95,000 objects - there's _tons_ of sharing), they'll 
all end up generating a "tree_entry" thing. So just there, you'll have 
something like 180,000 small allocations.

And yes, a "tree_entry" is smaller than a "struct object" or "struct
tree", but with twice as many "struct tree_entries" as there are objects
in _total_, that sure adds up pretty quickly.

And the refs. git-fsck-cache needs to be able to look through the 
dependency chains for all objects. git-rev-list doesn't. Again, they're 
small, but they're all over.

So I suspect half the memory allocations is just for these things. Maybe 
I'm overly optimistic. But I think that even without shrinking the object 
structures themselves, we should be able to make git-rev-list use a lot 
less memory.

			Linus