Why repository grows after "git gc"? / Purpose of *.keep files?

I have noticed that after cloning a repository (via git protocol) the
repo is packed pretty tightly and takes relatively small amount of disk
space. After using it a while and running "git gc" the repo sometimes
grows 25% or something like that.

For testing purposes I deleted objects/pack/*.keep file(s) and ran "git
gc" again. The repo resulted in small again, just like after the initial
clone. I don't have disk space problems but a repo growing about 25%
after manual "git gc" seems weird. What's the purpose of these *.keep
files? They just contain text like "fetch-pack <number> on <my
hostname>".


PS. I have merged Brandon Casey's new git-gc/repack patches. In case it
    has some effect. See the "pu" branch or "git log 9e7d5019".

Teemu Likonen wrote (2008-05-12 15:29 +0300):

For testing purposes I deleted objects/pack/*.keep file(s) and ran
"git gc" again. The repo resulted in small again, just like after the
initial clone.

After playing with test repo a while it seems that "git gc" never
touches pack files which have accompanying .keep file around. (And it's
common to have a .keep file after "git clone".) This makes gc perform
faster. A side effect seems to be that objects which later become
unreferenced in those pack-files-with-.keep are never pruned. *.keep
files also seem to prevent from really aggressively optimizing the
repository's size.

Probably a crazy idea: What if "gc --aggressive" first removed *.keep
files and after packing and garbage-collecting and whatever it does it
would add a .keep file for the newly created pack?

Hi,

On Mon, 12 May 2008, Teemu Likonen wrote:

Probably a crazy idea: What if "gc --aggressive" first removed *.keep 
files and after packing and garbage-collecting and whatever it does it 
would add a .keep file for the newly created pack?

Most .keep files are not meant to be removed by git-gc.  Usually, .keep 
files are only created interactively (if you _want_ to keep a pack, e.g. 
when it has been optimally packed and is big), or by git-index-pack while 
it is writing a pack (IIRC).

So I think it would be wrong for "gc --aggressive" to remove the .keep 
files.

Ciao,
Dscho

On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen [off-list ref] wrote:

Teemu Likonen wrote (2008-05-12 15:29 +0300):
Probably a crazy idea: What if "gc --aggressive" first removed *.keep
files and after packing and garbage-collecting and whatever it does it
would add a .keep file for the newly created pack?

My understanding is that the repacking with -a redoes the computation
to repack ALL the objects in every pack and loose objects, whereas
what would be preferred is to try to delta new objects (loose and
packed) against the existing .keep pack (extending it with the new
objects) but not trying to re-deltify objects in the .keep pack. This
is because .keep files are primarily for those who are cloning onto a
machine that isn't powerful (maybe even a laptop/palmtop) but who are
cloning from a powerful server, so that you wouldn't necessarily want
to apply your strategy unconditionally.

-- 
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"while having code so boring anyone can maintain it, use Python." --
attempted insult seen on slashdot

Johannes Schindelin wrote (2008-05-12 18:13 +0100):

On Mon, 12 May 2008, Teemu Likonen wrote:

quoted

Probably a crazy idea: What if "gc --aggressive" first removed
*.keep files and after packing and garbage-collecting and whatever
it does it would add a .keep file for the newly created pack?

Most .keep files are not meant to be removed by git-gc.  Usually,
.keep files are only created interactively (if you _want_ to keep
a pack, e.g. when it has been optimally packed and is big), or by
git-index-pack while it is writing a pack (IIRC).

So I think it would be wrong for "gc --aggressive" to remove the .keep
files.

I guess you're right. Maybe "gc --aggressive" could delete only certain
machine-generated .keep files which have an identifier inside?

Well, I don't really have any problems with the current behaviour; it
just feels a bit strange that, for example, Linus's kernel repository
grew about 90MB after just one update pull and gc. Also, dangling
objects are kept forever in .keep packs (which are created with "git
clone", for example).

On Mon, 12 May 2008, Teemu Likonen wrote:

Well, I don't really have any problems with the current behaviour; it
just feels a bit strange that, for example, Linus's kernel repository
grew about 90MB after just one update pull and gc.

That looks really odd.  Sure the repo might grow a bit, but 90MB seems 
really excessive.  How many time did pass between the initial clone and 
that subsequent pull?

Also, dangling
objects are kept forever in .keep packs (which are created with "git
clone", for example).

A pack obtained via 'git clone' will never contain any dangling objects.


Nicolas

Nicolas Pitre wrote (2008-05-12 14:56 -0400):

On Mon, 12 May 2008, Teemu Likonen wrote:

quoted

Well, I don't really have any problems with the current behaviour;
it just feels a bit strange that, for example, Linus's kernel
repository grew about 90MB after just one update pull and gc.

That looks really odd.  Sure the repo might grow a bit, but 90MB seems
really excessive.  How many time did pass between the initial clone
and that subsequent pull?

As I used the kernel repo just for testing this behaviour in question
I did both things today. Timestamps tell that there were six hours
between the initial .keep pack and the new pack created by manual "git
gc".

quoted

Also, dangling objects are kept forever in .keep packs (which are
created with "git clone", for example).

A pack obtained via 'git clone' will never contain any dangling
objects.

I think it can contain at some later point. For example, if a user first
fetches all the branches but later decides to track only one branch.
After deleting unneeded tracking branches and expiring the reflog
there'll be dangling objects in the original .keep pack created with
"git clone".

On Mon, 12 May 2008, Teemu Likonen wrote:

Nicolas Pitre wrote (2008-05-12 14:56 -0400):

quoted

On Mon, 12 May 2008, Teemu Likonen wrote:

quoted

Well, I don't really have any problems with the current behaviour;
it just feels a bit strange that, for example, Linus's kernel
repository grew about 90MB after just one update pull and gc.

That looks really odd.  Sure the repo might grow a bit, but 90MB seems
really excessive.  How many time did pass between the initial clone
and that subsequent pull?

As I used the kernel repo just for testing this behaviour in question
I did both things today. Timestamps tell that there were six hours
between the initial .keep pack and the new pack created by manual "git
gc".

This is way too big a difference.  Something is going on.

What git version is this? And can you send me the content of your 
.git/logs directory?

quoted

quoted

Also, dangling objects are kept forever in .keep packs (which are
created with "git clone", for example).

A pack obtained via 'git clone' will never contain any dangling
objects.

I think it can contain at some later point. For example, if a user first
fetches all the branches but later decides to track only one branch.
After deleting unneeded tracking branches and expiring the reflog
there'll be dangling objects in the original .keep pack created with
"git clone".

Sure.  But to decide to track only one branch and exclude the others 
require some higher level of git knowledge already.  At that point if 
you really care about top packing performances you certainly can deal 
with the .keep file as well.


Nicolas

On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre [off-list ref] wrote:

On Mon, 12 May 2008, Teemu Likonen wrote:

 > Nicolas Pitre wrote (2008-05-12 14:56 -0400):
 >
 > > On Mon, 12 May 2008, Teemu Likonen wrote:
 > >
 > > > Well, I don't really have any problems with the current behaviour;
 > > > it just feels a bit strange that, for example, Linus's kernel
 > > > repository grew about 90MB after just one update pull and gc.
 > >
 > > That looks really odd.  Sure the repo might grow a bit, but 90MB seems
 > > really excessive.  How many time did pass between the initial clone
 > > and that subsequent pull?
 >
 > As I used the kernel repo just for testing this behaviour in question
 > I did both things today. Timestamps tell that there were six hours
 > between the initial .keep pack and the new pack created by manual "git
 > gc".

 This is way too big a difference.  Something is going on.

 What git version is this? And can you send me the content of your
 .git/logs directory?


 > > > Also, dangling objects are kept forever in .keep packs (which are
 > > > created with "git clone", for example).
 > >
 > > A pack obtained via 'git clone' will never contain any dangling
 > > objects.
 >
 > I think it can contain at some later point. For example, if a user first
 > fetches all the branches but later decides to track only one branch.
 > After deleting unneeded tracking branches and expiring the reflog
 > there'll be dangling objects in the original .keep pack created with
 > "git clone".

 Sure.  But to decide to track only one branch and exclude the others
 require some higher level of git knowledge already.  At that point if
 you really care about top packing performances you certainly can deal
 with the .keep file as well.

I have had some similar problems with .keep files.  I cloned a repo I
created that had a branch that I wasn't interested in.  I deleted the
branch and then I could never get rid of the (large) number of objects
in that pack until I deleted the .keep and repacked.  I think there
should be some way of forcing git to fix this sort of thing.

It gets even worse, I had pushed up the branch I wanted to get rid of
to my hosted server and there was no way to get git to release that
disk space.  I had to have the hosting admin send me a tarball
of the repo, extract it, delete the .keep file and repack it then send
it back to him.  I was fortunate enough to have a service that would
let me do that.

Thanks,
Govind.

Nicolas Pitre wrote (2008-05-12 15:36 -0400):

On Mon, 12 May 2008, Teemu Likonen wrote:

quoted

quoted

On Mon, 12 May 2008, Teemu Likonen wrote:

quoted

Well, I don't really have any problems with the current
behaviour; it just feels a bit strange that, for example,
Linus's kernel repository grew about 90MB after just one update
pull and gc.

quoted

As I used the kernel repo just for testing this behaviour in
question I did both things today. Timestamps tell that there were
six hours between the initial .keep pack and the new pack created by
manual "git gc".

This is way too big a difference.  Something is going on.

What git version is this? And can you send me the content of your
.git/logs directory?

I'm using Git from the "master" branch; compiled it today. I have the
following gc/repack-related patches applied from the "pu" branch:

  builtin-gc.c: deprecate --prune, it now really has no effect
  git-gc: always use -A when manually repacking
  repack: modify behavior of -A option to leave unreferenced objects unpacked

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I'll send you the .git/logs directory but I'm afraid it doesn't tell
much. There are just three files:

  .git/logs/HEAD
  .git/logs/refs/heads/master
  .git/logs/refs/remotes/origin/master

They containt one line for the initial clone and one line for
the fast-forward pull.

quoted

I think it can contain at some later point. For example, if a user
first fetches all the branches but later decides to track only one
branch. After deleting unneeded tracking branches and expiring the
reflog there'll be dangling objects in the original .keep pack
created with "git clone".

Sure.  But to decide to track only one branch and exclude the others
require some higher level of git knowledge already.  At that point if
you really care about top packing performances you certainly can deal
with the .keep file as well.

Perhaps so. Although I don't consider this very high level Git
knowledge:

  $ git remote rm origin
  $ git remote add -t wanted_branch origin git://...

The first command removes all the tracking branches. The latter starts
to track only one branch.

On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I noticed that a while ago: when repacking multiple packs when one has a
.keep file, the resulting additional pack contains too many blobs and
trees, contrary to when only packing loose objects:

$ git init
$ echo a > a; git add a; git commit -m a
$ git gc
Counting objects: 3, done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 0 (delta 0)
$ git verify-pack -v .git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.pack
4bba7c0583de30efff4097299f89b199ab4a6dff commit 160 116 12
78981922613b2afb6025042ff6bd878ac1994e85 blob   2 11 167
aaff74984cccd156a469afa7d9ab10e4777beb24 tree   29 39 128
.git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.pack: ok

$ touch .git/objects/pack/pack-b87e61e2dc18ff37624d7f996f1270f923411530.keep
$ echo b > b; git add b; git commit -m b
$ git gc
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 0 (delta 0)
$ git verify-pack -v
.git/objects/pack/pack-aa817046e43f278d67c6b85962676246f57bb855.pack
3683f870be446c7cc05ffaef9fa06415276e1828 tree   58 65 158
61780798228d17af2d34fce4cfbdf35556832472 blob   2 11 223
647aed0360e964adc5cedb12e0719fb8bfc05867 commit 208 146 12
.git/objects/pack/pack-aa817046e43f278d67c6b85962676246f57bb855.pack: ok

$ git gc
Counting objects: 4, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), done.
Total 4 (delta 0), reused 4 (delta 0)
$ git verify-pack -v
.git/objects/pack/pack-5f692a665e062dedad7b4baf692517adec37899d.pack
3683f870be446c7cc05ffaef9fa06415276e1828 tree   58 65 158
61780798228d17af2d34fce4cfbdf35556832472 blob   2 11 234
647aed0360e964adc5cedb12e0719fb8bfc05867 commit 208 146 12
78981922613b2afb6025042ff6bd878ac1994e85 blob   2 11 223
.git/objects/pack/pack-5f692a665e062dedad7b4baf692517adec37899d.pack: ok

Mike

On Mon, 12 May 2008, Govind Salinas wrote:

On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre [off-list ref] wrote:

quoted

 Sure.  But to decide to track only one branch and exclude the others
 require some higher level of git knowledge already.  At that point if
 you really care about top packing performances you certainly can deal
 with the .keep file as well.

I have had some similar problems with .keep files.  I cloned a repo I
created that had a branch that I wasn't interested in.  I deleted the
branch and then I could never get rid of the (large) number of objects
in that pack until I deleted the .keep and repacked.

But as soon as you just "git pull" you'll get the deleted branch back.


Nicolas

On Mon, 12 May 2008, Teemu Likonen wrote:

I'll send you the .git/logs directory but I'm afraid it doesn't tell
much. There are just three files:

  .git/logs/HEAD
  .git/logs/refs/heads/master
  .git/logs/refs/remotes/origin/master

They containt one line for the initial clone and one line for
the fast-forward pull.

That's what I want.  This way I should be able to reproduce your exact 
case.


Nicolas

On Mon, May 12, 2008 at 4:06 PM, Nicolas Pitre [off-list ref] wrote:

On Mon, 12 May 2008, Govind Salinas wrote:

 > On Mon, May 12, 2008 at 2:36 PM, Nicolas Pitre [off-list ref] wrote:

quoted

quoted

 Sure.  But to decide to track only one branch and exclude the others

 > >  require some higher level of git knowledge already.  At that point if
 > >  you really care about top packing performances you certainly can deal
 > >  with the .keep file as well.
 >
 > I have had some similar problems with .keep files.  I cloned a repo I
 > created that had a branch that I wasn't interested in.  I deleted the
 > branch and then I could never get rid of the (large) number of objects
 > in that pack until I deleted the .keep and repacked.

 But as soon as you just "git pull" you'll get the deleted branch back.

If you read the rest of my mail, you will see where I removed it from the
hosted server as well.  But with difficulty.

Thanks,
Govind.

On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:

On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:

quoted

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I noticed that a while ago: when repacking multiple packs when one has a
.keep file, the resulting additional pack contains too many blobs and
trees, contrary to when only packing loose objects:

(...)

That is, it seems to also contain all the blobs and subtrees for all the
commits the pack contains, even when they already are in the pack having
a .keep file.

Mike

David Tweed [off-list ref] wrote:

On Mon, May 12, 2008 at 4:52 PM, Teemu Likonen [off-list ref] wrote:

quoted

Teemu Likonen wrote (2008-05-12 15:29 +0300):
Probably a crazy idea: What if "gc --aggressive" first removed *.keep
files and after packing and garbage-collecting and whatever it does it
would add a .keep file for the newly created pack?

My understanding is that the repacking with -a redoes the computation
to repack ALL the objects in every pack and loose objects,

No.  -a means repack all objects in all packs which do not have a
.keep on them.  Without -a we only repack loose objects.

whereas
what would be preferred is to try to delta new objects (loose and
packed) against the existing .keep pack (extending it with the new
objects) but not trying to re-deltify objects in the .keep pack.

We cannot do that.  Deltas in pack A may not reference base objects
in pack B.  This is a simplification rule that prevents us from
needing to worry about damaging a pack when we repack and delete
another pack.

This
is because .keep files are primarily for those who are cloning onto a
machine that isn't powerful (maybe even a laptop/palmtop) but who are
cloning from a powerful server, so that you wouldn't necessarily want
to apply your strategy unconditionally.

Yes, sort of.  We use .keep for two reasons:

  - As a "lock file" to prevent a pack that was just created by a
    git-fetch or git-recieve-pack from being deleted by a concurrent
    git-repack before the objects it contains are linked into the
    refs space and thus considered reachable;

  - As a way to avoid _huge_ packs (say >1G) that would take a lot
    of disk IO just to copy with 100% delta reuse from an old pack
    to a new pack each time the user runs git-gc.

I think git-clone marking a 150M linux-2.6 pack with .keep is wrong;
most users working with the linux-2.6 sources have sufficient
hardware to deal with the disk IO required to copy that with 100%
delta reuse.  But I have a repository at day-job with a 600M pack,
that's starting to head into the realm where git-gc while running
on battery on a laptop would prefer to have that .keep.

-- 
Shawn.

Mike Hommey [off-list ref] wrote:

On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:

quoted

On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:

quoted

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I noticed that a while ago: when repacking multiple packs when one has a
.keep file, the resulting additional pack contains too many blobs and
trees, contrary to when only packing loose objects:

(...)

That is, it seems to also contain all the blobs and subtrees for all the
commits the pack contains, even when they already are in the pack having
a .keep file.

I've noticed this too.  Like since day 1 when we added .keep.
But uh, nobody else complained and I forgot about it.

My theory (totally unproven) is that the new pack has objects we
copied from the .keep pack, because those objects were the best
delta-bases for the loose objects we have deltafied and want to
store in the new pack.  Except they aren't yet packed in the new
pack, so we pack them too.  Tada, duplicates.  :-\

Suddenly your repository nearly doubles in size if we have most
files/trees change, as those delta bases are copied whole into the
new pack.

-- 
Shawn.

On Mon, May 12, 2008 at 08:12:52PM -0400, Shawn O. Pearce wrote:

Mike Hommey [off-list ref] wrote:

quoted

On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:

quoted

On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:

quoted

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I noticed that a while ago: when repacking multiple packs when one has a
.keep file, the resulting additional pack contains too many blobs and
trees, contrary to when only packing loose objects:

(...)

That is, it seems to also contain all the blobs and subtrees for all the
commits the pack contains, even when they already are in the pack having
a .keep file.

I've noticed this too.  Like since day 1 when we added .keep.
But uh, nobody else complained and I forgot about it.

My theory (totally unproven) is that the new pack has objects we
copied from the .keep pack, because those objects were the best
delta-bases for the loose objects we have deltafied and want to
store in the new pack.  Except they aren't yet packed in the new
pack, so we pack them too.  Tada, duplicates.  :-\

Well, that does not seem delta related, since my testcase doesn't show
deltas in the second pack.

Mike

On Mon, 12 May 2008, Shawn O. Pearce wrote:

Mike Hommey [off-list ref] wrote:

quoted

On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote:

quoted

On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote:

quoted

But I have experienced the same earlier with some other post-1.5.5
version so I believe you can reproduce this yourself. After cloning
Linus's linux-2.6 repo its .git directory weights 209MB. After single
"git pull" and "git gc" it was 298MB in my test.

I noticed that a while ago: when repacking multiple packs when one has a
.keep file, the resulting additional pack contains too many blobs and
trees, contrary to when only packing loose objects:

(...)

That is, it seems to also contain all the blobs and subtrees for all the
commits the pack contains, even when they already are in the pack having
a .keep file.

I've noticed this too.  Like since day 1 when we added .keep.
But uh, nobody else complained and I forgot about it.

Well, now that I've reproduced Teemu Likonen's test case, I can confirm 
this is actually a problem.  Here I get:

|remote: Counting objects: 523, done.
|remote: Compressing objects: 100% (57/57), done.
|remote: Total 362 (delta 305), reused 362 (delta 305)
|Receiving objects: 100% (362/362), 65.37 KiB, done.
|Resolving deltas: 100% (305/305), completed with 105 local objects.
|From ../test1
|   492c2e4..9404ef0  master     -> master

The received pack is 449135 bytes large.  This is much larger than the 
actually received data which is 65.37 KiB, but we're completing a thin 
pack with 105 undeltified objects accounting for the size increase which 
is expected.  So far so good.

Now, in theory, running 'git gc' should only repack those 362 + 105 
objects, since the remaining ones are all found in the .keep flagged 
pack.  But that's not what's happening at all:

|Counting objects: 26559, done.
|Compressing objects: 100% (24708/24708), done.
|Writing objects: 100% (26559/26559), done.
|Total 26559 (delta 3054), reused 14011 (delta 1613)

So... there is something definitively wrong here.  The expectation was 
to get a pack in the same size range as the one received during the 
pack, or somewhat smaller due to a better delta compression of the added 
objects.  But instead we get a pack containing  26559 objects!!!  And in 
that lot, only 3054 (11%) are deltas.  That makes for a pack that 
started from 449135 bytes and grew to 72395940 bytes.

My theory (totally unproven) is that the new pack has objects we
copied from the .keep pack, because those objects were the best
delta-bases for the loose objects we have deltafied and want to
store in the new pack.  Except they aren't yet packed in the new
pack, so we pack them too.  Tada, duplicates.  :-\

Well, not exactly.

Let's see what happens here even before any packing is attempted

|$ git rev-list --objects 492c2e4..9404ef0
|362
|
|$ git rev-list --objects --all \
|   --unpacked=pack-6a3438b2702be06697023d80b77e67a73a0b0b5c.pack |
|	wc -l
|26559

So this --unpacked= argument (which undocumented semantics I still have 
issues with) is certainly not doing what is expected.


Nicolas