Re: People unaware of the importance of "git gc"?

Johannes Schindelin [off-list ref] writes:

On Wed, 5 Sep 2007, Junio C Hamano wrote:

quoted

@@ -20,6 +20,7 @@ static const char builtin_gc_usage[] = "git-gc [--prune] [--aggressive]";
 
 static int pack_refs = 1;
 static int aggressive_window = -1;
+static int gc_auto_threshold = 6700;

Please don't do that.

When you share objects with another git directory, git-gc --auto can get 
rid of the objects when some objects go away in the referenced repository.  

I thought the whole point of "gc --auto" was to have something
that does not lose/prune any objects, even the ones that do not
seem to be referenced from anywhere.  That is why invocations of
"git gc --auto" do not say --prune as you saw the second patch,
and the repack command "gc --auto" runs is "repack -d -l"
instead of "repack -a -d -l", which means that it does run
git-prune-packed after repacking but not git-prune.

Maybe I am missing something...

On Thu, 6 Sep 2007, Junio C Hamano wrote:

I thought the whole point of "gc --auto" was to have something
that does not lose/prune any objects, even the ones that do not
seem to be referenced from anywhere.  That is why invocations of
"git gc --auto" do not say --prune as you saw the second patch,
and the repack command "gc --auto" runs is "repack -d -l"
instead of "repack -a -d -l", which means that it does run
git-prune-packed after repacking but not git-prune.

I think "repack -d -l" should be ok from a safety perspective, but I'd 
also like to say that always running it incrementally is going to largely 
suck after a time.

IOW, if you get lots of small incrmental packs, after a while you really 
*do* need to do "git gc" to get the real pack generated.

In the case I saw, James really had hundreds of pack-files. That makes all 
our object lookups suck. Yes, not having loose objects at all is a big 
deal too, and yes, we try to start from the last pack-file we found (for 
the locality that we hope is there), but it's still pretty bad from a 
cache usage standpoint, and when we create a new object, we'll first 
search (in vain) in all the hundreds of pack-files.

So would "git gc --auto" have helped James? I'm sure it would have. But he 
already had lots of pack-files from doing "git fetch/pull", and while 
doing the "git gc --auto" will likely *delay* the point where you need to 
do a full repack, it doesn't make it go away.

We still need to tell people to do a full git gc at some point, or do it 
for them. And the longer you delay doing it, the more expensive it's going 
to get to do and/or the worse the final packing is going to be (especially 
if it ends up reusing non-optimal packing decisions from the smaller 
packs).

So I think the --auto stuff is still worth it, but it's really just 
pushing the pain somewhat further out.

(In the kernel community, if you fetch my tree daily, you really *are* 
going to have hundreds and hundreds of packfiles just from doing that).

So I'd really like us to also remind people to do a *real* and full "git 
gc", not just the incremental ones.

		Linus

Linus Torvalds wrote:

IOW, if you get lots of small incrmental packs, after a while you really 
*do* need to do "git gc" to get the real pack generated.
  

I wonder if it makes sense to repack just the small incremental packs 
into a large (but still incremental) pack, rather than repacking the 
entire repository. Presumably that would be a lot faster than a full 
"git gc", while still giving you reasonably good packing (at least, if 
the threshold is set to a hugh enough number of small packs) and keeping 
things fast. That could run as a second phase of "git gc --auto" -- it 
should be quick enough to not be too terribly annoying since we're not 
running it in the background.

Yeah, if you use the same repo for a long time, you'll accumulate a ton 
of medium-sized packs this way, but (a) that's much better than the 
situation we have today, and (b) it puts off the performance degradation 
for long enough that it becomes more reasonable to expect people to find 
out about running the full "git gc" in the meantime, or for git to 
further evolve to not need it.

-Steve

Junio C Hamano [off-list ref] wrote:

Johannes Schindelin [off-list ref] writes:

quoted

On Wed, 5 Sep 2007, Junio C Hamano wrote:

quoted

 static int aggressive_window = -1;
+static int gc_auto_threshold = 6700;

Please don't do that.

When you share objects with another git directory, git-gc --auto can get 
rid of the objects when some objects go away in the referenced repository.  

I thought the whole point of "gc --auto" was to have something
that does not lose/prune any objects, even the ones that do not
seem to be referenced from anywhere.  That is why invocations of
"git gc --auto" do not say --prune as you saw the second patch,
and the repack command "gc --auto" runs is "repack -d -l"
instead of "repack -a -d -l", which means that it does run
git-prune-packed after repacking but not git-prune.

Maybe I am missing something...

No, you aren't Junio.  `gc --auto` as you defined it is safe.
It won't delete objects from the database.  So it won't impact shared
repositories, or readers that are actively running in parallel with
the gc.  Both of which are important.

-- 
Shawn.

Hi,

On Thu, 6 Sep 2007, Junio C Hamano wrote:

Johannes Schindelin [off-list ref] writes:

quoted

On Wed, 5 Sep 2007, Junio C Hamano wrote:

quoted

@@ -20,6 +20,7 @@ static const char builtin_gc_usage[] = "git-gc [--prune] [--aggressive]";
 
 static int pack_refs = 1;
 static int aggressive_window = -1;
+static int gc_auto_threshold = 6700;

Please don't do that.

When you share objects with another git directory, git-gc --auto can 
get rid of the objects when some objects go away in the referenced 
repository.

I thought the whole point of "gc --auto" was to have something
that does not lose/prune any objects, even the ones that do not
seem to be referenced from anywhere.  That is why invocations of
"git gc --auto" do not say --prune as you saw the second patch,
and the repack command "gc --auto" runs is "repack -d -l"
instead of "repack -a -d -l", which means that it does run
git-prune-packed after repacking but not git-prune.

Maybe I am missing something...

No, _I_ missed the fact that no pack is rewritten...

Sorry for the line noise,
Dscho