Re: Why repository grows after "git gc"? / Purpose of *.keep files?

"Shawn O. Pearce" [off-list ref] writes:

David Tweed [off-list ref] wrote:

quoted

On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen [off-list ref] wrote:

quoted

Teemu Likonen wrote (2008-05-12 15:29 +0300):
Probably a crazy idea: What if "gc --aggressive" first removed *.keep
files and after packing and garbage-collecting and whatever it does it
would add a .keep file for the newly created pack?

My understanding is that the repacking with -a redoes the computation
to repack ALL the objects in every pack and loose objects,

No.  -a means repack all objects in all packs which do not have a
.keep on them.  Without -a we only repack loose objects.

quoted

whereas
what would be preferred is to try to delta new objects (loose and
packed) against the existing .keep pack (extending it with the new
objects) but not trying to re-deltify objects in the .keep pack.

We cannot do that.  Deltas in pack A may not reference base objects
in pack B.  This is a simplification rule that prevents us from
needing to worry about damaging a pack when we repack and delete
another pack.

quoted

This
is because .keep files are primarily for those who are cloning onto a
machine that isn't powerful (maybe even a laptop/palmtop) but who are
cloning from a powerful server, so that you wouldn't necessarily want
to apply your strategy unconditionally.

Yes, sort of.  We use .keep for two reasons:

  - As a "lock file" to prevent a pack that was just created by a
    git-fetch or git-recieve-pack from being deleted by a concurrent
    git-repack before the objects it contains are linked into the
    refs space and thus considered reachable;

  - As a way to avoid _huge_ packs (say >1G) that would take a lot
    of disk IO just to copy with 100% delta reuse from an old pack
    to a new pack each time the user runs git-gc.

I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;
most users working with the linux-2.6 sources have sufficient
hardware to deal with the disk IO required to copy that with 100%
delta reuse.  But I have a repository at day-job with a 600M pack,
that's starting to head into the realm where git-gc while running
on battery on a laptop would prefer to have that .keep.

Perhaps clone can decide to keep the .keep file depending on the size of
the pack then?

Junio C Hamano [off-list ref] wrote:

"Shawn O. Pearce" [off-list ref] writes:

quoted

I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;
most users working with the linux-2.6 sources have sufficient
hardware to deal with the disk IO required to copy that with 100%
delta reuse.  But I have a repository at day-job with a 600M pack,
that's starting to head into the realm where git-gc while running
on battery on a laptop would prefer to have that .keep.

Perhaps clone can decide to keep the .keep file depending on the size of
the pack then?

Yea, I think that's the better thing to do here.  I'm not sure where
the cut-off is, maybe its <512M delete the .keep once the refs are
inplace and the objects are ensured to be reachable.

Of course this does not fix the issue Nico was looking at.
We shouldn't be seeing a 98M explosion with objects duplicated
from the .keep pack into the new pack.

-- 
Shawn.

Shawn O. Pearce wrote:

Junio C Hamano [off-list ref] wrote:

quoted

"Shawn O. Pearce" [off-list ref] writes:

quoted

I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;
most users working with the linux-2.6 sources have sufficient
hardware to deal with the disk IO required to copy that with 100%
delta reuse.  But I have a repository at day-job with a 600M pack,
that's starting to head into the realm where git-gc while running
on battery on a laptop would prefer to have that .keep.

Perhaps clone can decide to keep the .keep file depending on the size of
the pack then?

Yea, I think that's the better thing to do here.  I'm not sure where
the cut-off is, maybe its <512M delete the .keep once the refs are
inplace and the objects are ensured to be reachable.

I think separate cutoffs should be in place for file size and number of 
objects.  Very tight packs probably require hours to repack as efficiently.

By the way, another scenario where I used pack files is when I can only 
distribute via http because of firewalls.  I make a clone of the 
original repository and mark the pack as keep; then I push to the 
distribution site, gc, and mark the pack as keep; then I have every day 
a cron job that does git-gc.  This way I know that the user will only 
have to download the third pack.  I think I'll modify the cron job to 
mark as keep the packs that exceed 2 megabytes or something like that.

Thinking about both use cases, the best would be to have options (common 
to git-clone, git-remote add, git-gc at least; and available via config 
keys too) like

   --keep-packs[=THRES1,THRES2,...]

where:

- one threshold would be enough to mark a pack as keep
- thresholds could be in the form "\d+[kmg]?b" for file size, 
"\d+[kmg]?" for number of objects.
- if no threshold is given, the default could be --keep-packs=100k,512MB 
or whatever is in the config.
- to mark all packs, use --keep-packs=0


Paolo

Paolo Bonzini [off-list ref] wrote:

Shawn O. Pearce wrote:

quoted

Junio C Hamano [off-list ref] wrote:

quoted

Perhaps clone can decide to keep the .keep file depending on the size of
the pack then?

Yea, I think that's the better thing to do here.  I'm not sure where
the cut-off is, maybe its <512M delete the .keep once the refs are
inplace and the objects are ensured to be reachable.

I think separate cutoffs should be in place for file size and number of 
objects.  Very tight packs probably require hours to repack as efficiently.

So long as you don't use `gc --aggressive` or `repack -f` the
tightness of a pack doesn't matter; delta reuse means we copy the
tight delta from the source pack to the new destination pack.

However, you are correct that the more objects in the source pack
the longer it will take to compute what is reachable, which does
extend the time needed for even a simple git-gc.
 
-- 
Shawn.

Paolo Bonzini wrote (2008-05-13 07:08 +0200):

I think separate cutoffs should be in place for file size and number
of  objects.  Very tight packs probably require hours to repack as
efficiently.

[...]

Thinking about both use cases, the best would be to have options
(common  to git-clone, git-remote add, git-gc at least; and available
via config  keys too) like

  --keep-packs[=THRES1,THRES2,...]

Some thoughts from user interface's point of view. Two assumptions:

  - gc is daily or weekly operation
  - gc --aggressive is more like weekly or monthly operation.

In big repositories gc can feel pretty slow if there are not any .keep
packs and user runs the command daily. So I think there's a point in
having a .keep pack in repositories the size of linux-2.6 for example.
But at the same time I think it would be nice to have an easy UI-way to
repack with better disk space optimization.

This started as a crazy idea but maybe it's not so crazy so I'll
rephrase my previous suggestion. At final stage the command gc
--aggressive would add new .keep file which contains an identifier like

  This .keep file was added by "gc --aggressive" and
  will be automatically deleted at next run.

(Or something like that, you get the idea.)

At first gc --aggressive looks for .keep files with such identifier and
deletes them if found. Then it proceeds normally and finally adds new
.keep file with the same identifier.

This way the "daily" gc would operate very fast (as it leaves .keep
packs alone), and with gc --aggressive user could easily decide when to
create new landmark .keep packs (and also prune possible dangling
objects inside previous .keep packs). Normal user don't need to know the
details. Just run gc occasionally and maybe gc --aggressive when better
optimization is needed.

How does this sound?

Teemu Likonen wrote:

This way the "daily" gc would operate very fast (as it leaves .keep
packs alone), and with gc --aggressive user could easily decide when to
create new landmark .keep packs (and also prune possible dangling
objects inside previous .keep packs). Normal user don't need to know the
details. Just run gc occasionally and maybe gc --aggressive when better
optimization is needed.

How does this sound?

It sounds sound :-).
I like the simplicity.
-- 
Sincerely,                                                          srb@cuci.nl
           Stephen R. van den Berg.

Stephen R. van den Berg wrote (2008-05-14 00:46 +0300):

Teemu Likonen wrote:

quoted

This way the "daily" gc would operate very fast (as it leaves .keep
packs alone), and with gc --aggressive user could easily decide when to
create new landmark .keep packs (and also prune possible dangling
objects inside previous .keep packs). Normal user don't need to know the
details. Just run gc occasionally and maybe gc --aggressive when better
optimization is needed.

quoted

How does this sound?

It sounds sound :-).
I like the simplicity.

It turned out that gc --aggressive is not what I thought it was, i.e.
"pack aggressively and efficiently". So my suggestion implies the
semantics that --aggressive would do effective compressing.