git gc & deleted branches

Hello,

I'm trying to reclaim space from an abandoned branch (never involved in any merge) using 'git gc', but it doesn't appear to work:

  mkdir testrepo
  cd testrepo
  git init
  dd if=/dev/urandom bs=1024k count=10 of=file
  git add file
  git commit -a -m 'initial checkin'
  git checkout -b test
  dd if=/dev/urandom bs=1024k count=10 of=file
  git commit -a -m 'branch checkin'
  git checkout master
  du -s .    # returns 30960
  git branch -D test
  git gc
  du -s .    # returns 30916

Here I had expected ~20000 since the branch uses ~10000.

My config is

[gc]
        reflogExpire = 0
        reflogExpireUnreachable = 0
        rerereresolved = 0
        rerereunresolved = 0
        packrefs = 1

I also tried 'git-pack-refs --all' or 'git-pack-refs --prune' but to no avail.

What am I doing wrong?

Thanks for any hints.

Regards

Guido

On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:

[gc]
        reflogExpire = 0
        reflogExpireUnreachable = 0
        rerereresolved = 0
        rerereunresolved = 0
        packrefs = 1

git-gc uses a "safe" pruning mode, where it only prunes unreferenced
objects that are older than a certain period (this makes it safe to run
git-gc, even if other processes are creating objects at the same time).

So try

[gc]
        pruneExpire = now

Alternatively, you can just run 'git prune' manually instead of 'git
gc'.

I also tried 'git-pack-refs --all' or 'git-pack-refs --prune' but to no  
avail.

Those won't help at all; they are purely about moving refs from
individual files into the 'packed-refs' file.

-Peff

On Thu, 8 May 2008, Jeff King wrote:

On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:

quoted

[gc]
        reflogExpire = 0
        reflogExpireUnreachable = 0
        rerereresolved = 0
        rerereunresolved = 0
        packrefs = 1

git-gc uses a "safe" pruning mode, where it only prunes unreferenced
objects that are older than a certain period (this makes it safe to run
git-gc, even if other processes are creating objects at the same time).

So try

[gc]
       pruneExpire = now

Alternatively, you can just run 'git prune' manually instead of 'git
gc'.

Jeff, I tried it, but it has no effect (see below). There is only the master branch left, and only one commit therein, still it uses the space former occupied by the branch. I'm using git version 1.5.5.1.147.g867f.

Any further ideas?


$ git config -l
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
gc.reflogexpire=0
gc.reflogexpireunreachable=0
gc.rerereresolved=0
gc.rerereunresolved=0
gc.packrefs=1
gc.pruneexpire=now

$ git gc
Counting objects: 6, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), done.
Total 6 (delta 0), reused 6 (delta 0)

$ git prune

$ git branch
* master

$ du -s .
30820   .

$ git log
commit 9717437cdcb2a4457f28f41db5f6fad9ca55b54e
Author: Testuser [off-list ref]
Date:   Thu May 8 19:40:06 2008 +0200

    initial checkin

$ ls -l
total 10240
-rw-r--r-- 1 testuser users 10485760 May  8 19:40 file

Regards

Guido

Guido Ostkamp wrote:

On Thu, 8 May 2008, Jeff King wrote:

quoted

On Thu, May 08, 2008 at 07:45:31PM +0200, Guido Ostkamp wrote:

quoted

[gc]
        reflogExpire = 0
        reflogExpireUnreachable = 0
        rerereresolved = 0
        rerereunresolved = 0
        packrefs = 1

git-gc uses a "safe" pruning mode, where it only prunes unreferenced
objects that are older than a certain period (this makes it safe to run
git-gc, even if other processes are creating objects at the same time).

So try

[gc]
       pruneExpire = now

Alternatively, you can just run 'git prune' manually instead of 'git
gc'.

Jeff, I tried it, but it has no effect (see below). There is only the
master branch left, and only one commit therein, still it uses the space
former occupied by the branch. I'm using git version 1.5.5.1.147.g867f.

Any further ideas?

Possibly that object got packed? git-prune only removes loose objects.
Try 'git gc --prune' which will call git-repack with the -a option.

btw, this is _really_ a non-issue. It seems to keep coming up on the list.

Just know that each one of the config options that you set to zero, including
the one Jeff suggested setting to "now", is a safety mechanism that is there
to ensure that you never ever lose data and that mistakes are recoverable.

And be assured that the objects referenced by a deleted branch will be removed
from the repository eventually as long as 'git gc --prune' is run periodically.

-brandon

On Thu, May 08, 2008 at 08:55:50PM +0200, Guido Ostkamp wrote:

Jeff, I tried it, but it has no effect (see below). There is only the  
master branch left, and only one commit therein, still it uses the space  
former occupied by the branch. I'm using git version 1.5.5.1.147.g867f.

It worked fine for me; it's possible, as Brandon mentioned, that it is
in a pack already, and only a "repack -a" would get rid of it. FWIW, my
steps were:

  # ...same as you for repo and branch creation
  git branch -D test
  git config gc.reflogexpire 0
  git config gc.reflogexpireunreachable 0
  git config gc.pruneexpire now
  git gc
  du -s .git ;# shows 10396

-Peff

On Thu, 8 May 2008, Brandon Casey wrote:

Possibly that object got packed? git-prune only removes loose objects. Try 'git gc --prune' which will call git-repack with the -a option.

Thanks, Brandon - that did the trick :-)

Just know that each one of the config options that you set to zero, including the one Jeff suggested setting to "now", is a safety mechanism that is there to ensure that you never ever lose data and that mistakes are recoverable.

I am aware of this. However, at work I am unfortunately bound to a very restrictive filesystem quota on central development servers, so every single byte counts in (our official versioning control system is ClearCase where less space is required due to working tree and history being supplied through virtual filesystems).

And be assured that the objects referenced by a deleted branch will be removed from the repository eventually as long as 'git gc --prune' is run periodically.

Ok. I did not know about the 'prune' option yet as it neither mentioned in the "Git Tutorial" nor "Everyday Git", there only 'git gc' is used with no options.

Regards

Guido

On Thu, May 08, 2008 at 03:07:53PM -0500, Brandon Casey wrote:

btw, this is _really_ a non-issue. It seems to keep coming up on the list.

Just know that each one of the config options that you set to zero, including
the one Jeff suggested setting to "now", is a safety mechanism that is there
to ensure that you never ever lose data and that mistakes are recoverable.

Yes, I want to chime in since I have been giving advice in such threads:
Please don't construe my help as any sort of endorsement of this
behavior. Git tries hard not to lose your data, and it is almost always
a bad idea to try to override these safety checks unless you really know
what you are doing.

And even then, try to consider balancing a bit of freed disk space (and
generally _no_ performance gain, because git is very good about not
looking at objects that aren't necessary to the current operation)
versus thinking "oops, I wish I still had that data" in a few days.

I can think offhand of only one time when it was truly useful for me to
prune aggressively, and it was a very special case: a pathologically
large repo for which I was doing a one-shot conversion from another
format (and I wanted to prune failed attempts).

-Peff

On Thu, May 08, 2008 at 10:52:19PM +0200, Guido Ostkamp wrote:

quoted

And be assured that the objects referenced by a deleted branch will be  
removed from the repository eventually as long as 'git gc --prune' is  
run periodically.

Ok. I did not know about the 'prune' option yet as it neither mentioned in 
the "Git Tutorial" nor "Everyday Git", there only 'git gc' is used with no 
options.

It is deprecated; see 25ee9731.

According to that commit message, prune is now a no-op. However, it
looks like it is still used for trigger a "repack -a" rather than
"repack -A". I don't know if it is worth making that behavior available
through some more sane command line option (I would think people who
really know that they want "repack -a" would just call it).

-Peff

On Thu, 8 May 2008, Jeff King wrote:

On Thu, May 08, 2008 at 10:52:19PM +0200, Guido Ostkamp wrote:

quoted

quoted

And be assured that the objects referenced by a deleted branch will be  
removed from the repository eventually as long as 'git gc --prune' is  
run periodically.

Ok. I did not know about the 'prune' option yet as it neither mentioned in 
the "Git Tutorial" nor "Everyday Git", there only 'git gc' is used with no 
options.

It is deprecated; see 25ee9731.

According to that commit message, prune is now a no-op. However, it
looks like it is still used for trigger a "repack -a" rather than
"repack -A". I don't know if it is worth making that behavior available
through some more sane command line option (I would think people who
really know that they want "repack -a" would just call it).

Well, actually this is a problem.

I think it is a good thing to deprecate gc --prune.  but if that means 
that repack -a is never used then unreferenced and expired objects will 
never be pruned if they're packed if one is always using 'git gc' as we 
are advocating.


Nicolas

On Thu, May 08, 2008 at 05:15:34PM -0400, Nicolas Pitre wrote:

quoted

According to that commit message, prune is now a no-op. However, it
looks like it is still used for trigger a "repack -a" rather than
"repack -A". I don't know if it is worth making that behavior available

Well, actually this is a problem.

I think it is a good thing to deprecate gc --prune.  but if that means 
that repack -a is never used then unreferenced and expired objects will 
never be pruned if they're packed if one is always using 'git gc' as we 
are advocating.

I thought that -A would eventually put them all into a single pack,
killing off the old packs.

-Peff

Jeff King wrote:

On Thu, May 08, 2008 at 05:15:34PM -0400, Nicolas Pitre wrote:

quoted

quoted

According to that commit message, prune is now a no-op. However, it
looks like it is still used for trigger a "repack -a" rather than
"repack -A". I don't know if it is worth making that behavior available

Well, actually this is a problem.

I think it is a good thing to deprecate gc --prune.  but if that means 
that repack -a is never used then unreferenced and expired objects will 
never be pruned if they're packed if one is always using 'git gc' as we 
are advocating.

I thought that -A would eventually put them all into a single pack,
killing off the old packs.

'-a' puts everything in a single pack and kills off old packs. Anything that
was unreachable is not repacked in the new pack.

'-A' does the same thing but it also repacks the unreachable objects that were
previously packed.

So if something gets packed that subsequently becomes unreachable it will never
be removed unless 'repack -a' is used.

Possibly --keep-unreachable should instead unpack the unreachable items which would
allow them to eventually be pruned based on pruneExpire. Then we could indeed
get rid of the --prune option to git-gc.

-brandon

On Thu, May 08, 2008 at 04:23:53PM -0500, Brandon Casey wrote:

quoted

I thought that -A would eventually put them all into a single pack,
killing off the old packs.

'-a' puts everything in a single pack and kills off old packs. Anything that
was unreachable is not repacked in the new pack.

'-A' does the same thing but it also repacks the unreachable objects that were
previously packed.

Ah, indeed. I hadn't looked closely at the -A behavior before. So yes,
we are never killing off prunable packed objects. Probably we could use
the same solution as "git prune --expire"; perhaps a
"--keep-unreachable=2.weeks.ago"?

-Peff

On Thu, 8 May 2008, Jeff King wrote:

According to that commit message, prune is now a no-op. However, it looks like it is still used for trigger a "repack -a" rather than "repack -A".

I tried to look at this but found this option '-A' to be undocumented in the manpage (git/Documentation/git-repack.txt and what is generated from it).

Regards

Guido

Jeff King wrote:

On Thu, May 08, 2008 at 04:23:53PM -0500, Brandon Casey wrote:

quoted

quoted

I thought that -A would eventually put them all into a single pack,
killing off the old packs.

'-a' puts everything in a single pack and kills off old packs. Anything that
was unreachable is not repacked in the new pack.

'-A' does the same thing but it also repacks the unreachable objects that were
previously packed.

Ah, indeed. I hadn't looked closely at the -A behavior before. So yes,
we are never killing off prunable packed objects. Probably we could use
the same solution as "git prune --expire"; perhaps a
"--keep-unreachable=2.weeks.ago"?

The 'prune --expire' behavior is based on object mtime (i.e. file modification time).
That is lost once something is packed right?

I was thinking that either repack or pack-objects could be modified to unpack those
unreachable objects and leave them loose, and also give them the timestamp of the
pack file they came from. Then the --expire behavior of git-prune could work normally
and remove them. This seems like it would work nicely since prune follows repack in
git-gc.

-brandon

On Thu, May 08, 2008 at 04:40:20PM -0500, Brandon Casey wrote:

The 'prune --expire' behavior is based on object mtime (i.e. file
modification time).  That is lost once something is packed right?

Yes. You would have to use the pack mtime. But of course you would have
to actually _leave_ them in a pack, or they would just keep getting
added to the new pack.

I was thinking that either repack or pack-objects could be modified to
unpack those unreachable objects and leave them loose, and also give
them the timestamp of the pack file they came from. Then the --expire
behavior of git-prune could work normally and remove them. This seems
like it would work nicely since prune follows repack in git-gc.

That is sensible, I think.

-Peff

Jeff King wrote:

On Thu, May 08, 2008 at 04:40:20PM -0500, Brandon Casey wrote:

quoted

The 'prune --expire' behavior is based on object mtime (i.e. file
modification time).  That is lost once something is packed right?

Yes. You would have to use the pack mtime. But of course you would have
to actually _leave_ them in a pack, or they would just keep getting
added to the new pack.

I had the impression that unreachable objects would not be packed. Maybe it
was more of an assumption.

-brandon

On Thu, May 08, 2008 at 04:53:20PM -0500, Brandon Casey wrote:

quoted

Yes. You would have to use the pack mtime. But of course you would have
to actually _leave_ them in a pack, or they would just keep getting
added to the new pack.

I had the impression that unreachable objects would not be packed. Maybe it
was more of an assumption.

Look in builtin-pack-objects.c:1981-1982. We basically just say "if it's
in a pack now, then it should go into the new pack."

-Peff

Jeff King <peff <at> peff.net> writes:

On Thu, May 08, 2008 at 04:53:20PM -0500, Brandon Casey wrote:

quoted

quoted

Yes. You would have to use the pack mtime. But of course you would have
to actually _leave_ them in a pack, or they would just keep getting
added to the new pack.

I had the impression that unreachable objects would not be packed. Maybe it
was more of an assumption.

Look in builtin-pack-objects.c:1981-1982. We basically just say "if it's
in a pack now, then it should go into the new pack."

-Peff

Here's what I was thinking (posted using gmane):

diff --git a/git-repack.sh b/git-repack.sh
index e18eb3f..064c331 100755
--- a/git-repack.sh
+++ b/git-repack.sh

@@ -30,7 +30,7 @@ do
        -n)     no_update_info=t ;;
        -a)     all_into_one=t ;;
        -A)     all_into_one=t
-               keep_unreachable=--keep-unreachable ;;
+               keep_unreachable=t ;;
        -d)     remove_redundant=t ;;
        -q)     quiet=-q ;;
        -f)     no_reuse=--no-reuse-object ;;

@@ -78,9 +78,6 @@ case ",$all_into_one," in
        if test -z "$args"
        then
                args='--unpacked --incremental'
-       elif test -n "$keep_unreachable"
-       then
-               args="$args $keep_unreachable"
        fi
        ;;
 esac

@@ -116,7 +113,15 @@ for name in $names ; do
                echo >&2 "old-pack-$name.{pack,idx} in $PACKDIR."
                exit 1
        }
-       rm -f "$PACKDIR/old-pack-$name.pack" "$PACKDIR/old-pack-$name.idx"
+       rm -f "$PACKDIR/old-pack-$name.idx"
+       test -z "$keep_unreachable" ||
+         ! test -f "$PACKDIR/old-pack-$name.pack" ||
+         git unpack-objects < "$PACKDIR/old-pack-$name.pack" || {
+               echo >&2 "Failed unpacking unreachable objects from old pack"
+               echo >&2 "saved as old-pack-$name.pack in $PACKDIR."
+               exit 1
+       }
+       rm -f "$PACKDIR/old-pack-$name.pack"
 done
 
 if test "$remove_redundant" = t

@@ -130,7 +135,18 @@ then
                  do
                        case " $fullbases " in
                        *" $e "*) ;;
-                       *)      rm -f "$e.pack" "$e.idx" "$e.keep" ;;
+                       *)
+                               rm -f "$e.idx" "$e.keep"
+                               if test -n "$keep_unreachable" &&
+                                  test -f "$e.pack"
+                               then
+                                       git unpack-objects < "$e.pack" || {
+                                               echo >&2 "Fail AVOID GMANE WRAP"
+                                               exit 1
+                                       }
+                               fi
+                               rm -f "$e.pack"
+                       ;;
                        esac
                  done
                )

Is the first invocation of unpack-objects necessary? pack-objects has created
a pack which hashes to the same name of a pack we already have, and we replace
the original with the new one. Is that what is happening? They will be identical
right?

Of course this won't set the timestamp on the created objects based on the
timestamp of the pack file, but this was easy. Setting the timestamp would be
proper, but what's another two weeks. Besides, for those users not manually
running git-gc, this code path won't even be executed until there are enough
pack files for git-gc to add -A to the repack options.

Then, for git-gc it should be enough to just always use -A with repack when
manually running it. Then --prune can be deprecated.

-brandon

On Fri, May 09, 2008 at 01:41:30AM +0000, Brandon Casey wrote:

I like it. It makes an easy rule to say "packed objects _never_ get
pruned, they only get demoted to loose objects." And then of course
we have sane rules for pruning loose objects.

-       rm -f "$PACKDIR/old-pack-$name.pack" "$PACKDIR/old-pack-$name.idx"
+       rm -f "$PACKDIR/old-pack-$name.idx"
+       test -z "$keep_unreachable" ||
+         ! test -f "$PACKDIR/old-pack-$name.pack" ||
+         git unpack-objects < "$PACKDIR/old-pack-$name.pack" || {
+               echo >&2 "Failed unpacking unreachable objects from old pack"
+               echo >&2 "saved as old-pack-$name.pack in $PACKDIR."
+               exit 1
+       }
+       rm -f "$PACKDIR/old-pack-$name.pack"
[...]

Is the first invocation of unpack-objects necessary? pack-objects has created
a pack which hashes to the same name of a pack we already have, and we replace
the original with the new one. Is that what is happening? They will be
identical right?

Yeah, that's what it looks like to me (that the first unpack is
unnecessary, because we will just be putting the new pack into place
that has all the same objects). AIUI, two packs with identical hashes
must contain the exact same objects.

Of course this won't set the timestamp on the created objects based on the
timestamp of the pack file, but this was easy. Setting the timestamp would be
proper, but what's another two weeks. Besides, for those users not manually
running git-gc, this code path won't even be executed until there are enough
pack files for git-gc to add -A to the repack options.

I think the extra two weeks is fine.

-Peff

On May 9, 2008, at 00:19, Jeff King wrote:

I like it. It makes an easy rule to say "packed objects _never_ get
pruned, they only get demoted to loose objects." And then of course
we have sane rules for pruning loose objects.

Isn't there an issue with the "git gc" triggering because there
may be too many loose unreferenced objects?
Still, I do like the approach.

Maybe unreferenced objects and old refs should go to a .git/lost+found
directory and be expired from there. This has a couple of benefits:

  -  Easy to manually inspect or blow away any crud
  -  One git-gc run can make one pack in lost+found,
     avoiding huge numbers of loose objects (and massive disk use)
     when trying to do a large cleanup (to possibly reclaim disk space)
  -  Objects will not be accessible by ordinary git commands for a while,
     before they are really removed, avoiding surprises

Only some tools would look in the lost+found to restore stuff.

  -Geert

Geert Bosch wrote:

On May 9, 2008, at 00:19, Jeff King wrote:

quoted

I like it. It makes an easy rule to say "packed objects _never_ get
pruned, they only get demoted to loose objects." And then of course
we have sane rules for pruning loose objects.

Isn't there an issue with the "git gc" triggering because there
may be too many loose unreferenced objects?
Still, I do like the approach.

This would be an argument for going the extra mile and having the loose
objects adopt the timestamp of their pack file. In the normal case they
would probably be pruned immediately during the same git-gc run.

Maybe unreferenced objects and old refs should go to a .git/lost+found
directory and be expired from there. This has a couple of benefits:

  -  Objects will not be accessible by ordinary git commands for a while,
     before they are really removed, avoiding surprises

Unreferenced objects are sometimes used by other repositories which have
this repository listed as an alternate. So it may not be a good idea to
make the unreferenced objects inaccessible.

-brandon

On Fri, May 09, 2008 at 10:14:12AM -0500, Brandon Casey wrote:

quoted

  -  Objects will not be accessible by ordinary git commands for a while,
     before they are really removed, avoiding surprises

Unreferenced objects are sometimes used by other repositories which have
this repository listed as an alternate. So it may not be a good idea to
make the unreferenced objects inaccessible.

But that is precisely what we're going to do, but in two weeks. Isn't it
better to have the dependent repo fail while the change is recoverable?

-Peff

Jeff King wrote:

On Fri, May 09, 2008 at 10:14:12AM -0500, Brandon Casey wrote:

quoted

quoted

  -  Objects will not be accessible by ordinary git commands for a while,
     before they are really removed, avoiding surprises

Unreferenced objects are sometimes used by other repositories which have
this repository listed as an alternate. So it may not be a good idea to
make the unreferenced objects inaccessible.

But that is precisely what we're going to do, but in two weeks. Isn't it
better to have the dependent repo fail while the change is recoverable?

good point.

-b

On Fri, 9 May 2008, Brandon Casey wrote:

Geert Bosch wrote:

quoted

On May 9, 2008, at 00:19, Jeff King wrote:

quoted

I like it. It makes an easy rule to say "packed objects _never_ get
pruned, they only get demoted to loose objects." And then of course
we have sane rules for pruning loose objects.

Isn't there an issue with the "git gc" triggering because there
may be too many loose unreferenced objects?
Still, I do like the approach.

This would be an argument for going the extra mile and having the loose
objects adopt the timestamp of their pack file. In the normal case they
would probably be pruned immediately during the same git-gc run.

Well, not necessarily.  If you created a large branch yesterday and you 
are deleting it today, then if you repacked in between means that those 
loose objects won't be more than one day old.  Yet there could be enough 
of them to trigger auto gc.  But that auto gc won't pack those objects 
since they are unreferenced.  Hence auto gc will trigger all the time 
without making any progress.

quoted

Maybe unreferenced objects and old refs should go to a .git/lost+found
directory and be expired from there. This has a couple of benefits:

quoted

  -  Objects will not be accessible by ordinary git commands for a while,
     before they are really removed, avoiding surprises

Unreferenced objects are sometimes used by other repositories which have
this repository listed as an alternate. So it may not be a good idea to
make the unreferenced objects inaccessible.

Nah.  If this is really the case then you shouldn't be running gc at all 
in the first place.


Nicolas

Nicolas Pitre wrote:

On Fri, 9 May 2008, Brandon Casey wrote:

quoted

Geert Bosch wrote:

quoted

On May 9, 2008, at 00:19, Jeff King wrote:

quoted

I like it. It makes an easy rule to say "packed objects _never_ get
pruned, they only get demoted to loose objects." And then of course
we have sane rules for pruning loose objects.

Isn't there an issue with the "git gc" triggering because there
may be too many loose unreferenced objects?
Still, I do like the approach.

This would be an argument for going the extra mile and having the loose
objects adopt the timestamp of their pack file. In the normal case they
would probably be pruned immediately during the same git-gc run.

Well, not necessarily.  If you created a large branch yesterday and you 
are deleting it today, then if you repacked in between means that those 
loose objects won't be more than one day old.  Yet there could be enough 
of them to trigger auto gc.  But that auto gc won't pack those objects 
since they are unreferenced.  Hence auto gc will trigger all the time 
without making any progress.

That's true, but the intermediate repack is not the cause here. You'd be
in the same situation if a large branch was created yesterday and then
deleted today even if packing had never occurred.

I do see your point, but you should have said a large branch created a month
ago, deleted today, but repacked yesterday. :)

-brandon