From: Thomas Rast <hidden> Date: 2016-06-15 22:46:10
Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.
This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.
Thanks to doener (Björn Steinbrink) for comments and corrections.
Signed-off-by: Thomas Rast <redacted>
---
Or we could just add an option --prune-everything-I-have-a-backup-I-promise
to git-filter-branch, so that users can get the same result with less effort.
Documentation/git-filter-branch.txt | 36 +++++++++++++++++++++++++++++++++++
1 files changed, 36 insertions(+), 0 deletions(-)
@@ -339,6 +339,42 @@ git filter-branch --index-filter \ ---------------------------------------------------------------++Checklist for Shrinking a Repository+------------------------------------++git-filter-branch is often used to get rid of a subset of files,+usually with some combination of `\--index-filter` and+`\--subdirectory-filter`. If you want to physically shrink the+repository afterwards, you have some choices:++* Clone it with `git clone file:///path/to/repo`. The clone will not+ have the removed objects. See linkgit:git-clone[1]. (Note that+ cloning with a plain path just hardlinks everything!)++If you really don't want to clone it, for whatever reasons, check the+following points (in this order). This is a very destructive+approach, so *make a backup* or go back to cloning it. You have been+warned.++* Make sure you really removed all variants of a filename, if a blob+ was moved over its lifetime. `git log \--follow \--all \-- foo` can+ help you find renames.++* Make sure you really filtered all refs: use `\--tag-name-filter cat+ \-- \--all` when calling git-filter-branch.++* Make sure you remove the original refs backed up by+ git-filter-branch: say `git for-each-ref \--format="%(refname)"+ refs/original/ | xargs -n 1 git update-ref -d`.++* Expire all reflogs with `git reflog expire \--expire=now \--all`.++* Repack to remove packed objects with `git repack -ad`.++* Prune unpacked unreferenced objects with `git prune \--expire=now`.++ Author ------ Written by Petr "Pasky" Baudis <pasky@suse.cz>,
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:46:10
Hi,
On Sat, 14 Feb 2009, Jan Krüger wrote:
On Fri, 13 Feb 2009 18:14:39 +0100, Thomas Rast [off-list ref]
wrote:
quoted
Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.
Good idea.
Indeed. It is funny when some people ask, but reading it several times a
week in #git makes that fun go away real quick.
quoted
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.++* Repack to remove packed objects with `git repack -ad`.++* Prune unpacked unreferenced objects with `git prune \--expire=now`.
Is the --expire=now actually needed for prune? The way I read it
git-prune(1), it defaults to pruning everything anyway.
I hope not. IIRC I set the default to 2 weeks.
*goes to read the source* Ah, that default is only set in "git gc", and
"git prune" still prunes everything. So you're absolutely correct.
But maybe we do not want to direct the user to use "prune" here, but
rather "gc"?
Ciao,
Dscho
From: Jan Krüger <hidden> Date: 2016-06-15 22:46:10
Hi,
On Sat, 14 Feb 2009 06:46:18 +0100 (CET), Johannes Schindelin
[off-list ref] wrote:
quoted
Is the --expire=now actually needed for prune? The way I read it
git-prune(1), it defaults to pruning everything anyway.
[...] Ah, that default is only set in "git gc", and "git prune" still
prunes everything. [...] But maybe we do not want to direct the user
to use "prune" here, but rather "gc"?
I just read the source code of gc and noticed that the only way to
influence its expiration limit for prune is to set it in the config file
(gc.pruneExpire). Thus there is no immediate way to tell gc to prune
*all* unreachable objects and we need to stick to the instructions that
are in the patch (minus the unnecessary --expire=now for prune). The
alternative would be to add a command line option to gc that
corresponds to gc.pruneExpire. I don't really care about that, but a
patch for it is below.
-Jan
-----8<-----
From 42226c6c542be4e9fff0817662115f180033e75a Mon Sep 17 00:00:00 2001
From: =?utf-8?q?Jan=20Kr=C3=BCger?= <redacted>
Date: Sat, 14 Feb 2009 07:39:23 +0100
Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
In semi-regular intervals, people tend to want to get rid of huge
objects they accidentally created in a repository. Of course it
shouldn't be too easy to do but also not too hard. Thus we allow them to
get rid of unreferenced objects more easily once they have managed to
make the objects unreferenced, by allowing them to override the prune
expiration limit.
Signed-off-by: Jan Krüger <redacted>
---
Documentation/git-gc.txt | 5 +++++
builtin-gc.c | 5 +++--
2 files changed, 8 insertions(+), 2 deletions(-)
@@ -59,6 +59,11 @@ are consolidated into a single pack by using the `-A` option of 'git-repack'. Setting `gc.autopacklimit` to 0 disables automatic consolidation of packs.+--prune-expire <time>::+ Expire unreachable objects older than <time>, rather than+ considering the configuration variable or the default explained in+ the next section.+ --quiet:: Suppress all progress reports.
From: Thomas Rast <hidden> Date: 2016-06-15 22:46:10
Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.
This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.
It may seem contradictory to document fallbacks for older Gits in
newer docs, but we want to point people at this as a FAQ answer, and
they will frequently not have the newest version installed.
Thanks to Björn Steinbrink and Junio C Hamano for comments and
corrections.
Signed-off-by: Thomas Rast <redacted>
---
Johannes Schindelin wrote:
On Sat, 14 Feb 2009, Jan Krüger wrote:
quoted
Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
I like it, and if nobody beats me to it, I will add a test for this
option tonight.
Cool, then let's use that in the docs.
Documentation/git-filter-branch.txt | 36 +++++++++++++++++++++++++++++++++++
1 files changed, 36 insertions(+), 0 deletions(-)
@@ -339,6 +339,42 @@ git filter-branch --index-filter \ ---------------------------------------------------------------++Checklist for Shrinking a Repository+------------------------------------++git-filter-branch is often used to get rid of a subset of files,+usually with some combination of `\--index-filter` and+`\--subdirectory-filter`. If you want to physically shrink the+repository afterwards, you have some choices:++* Make sure you really removed all variants of a filename, if a blob+ was moved over its lifetime. `git log \--name-only \--follow \--all+ \-- filename` can help you find renames.++* Make sure you really filtered all refs: use `\--tag-name-filter cat+ \-- \--all` when calling git-filter-branch.++* Clone it with `git clone file:///path/to/repo`. The clone will not+ have the removed objects. See linkgit:git-clone[1]. (Note that+ cloning with a plain path just hardlinks everything!)++If you really don't want to clone it, for whatever reasons, check the+following points instead (in this order). This is a very destructive+approach, so *make a backup* or go back to cloning it. You have been+warned.++* Remove the original refs backed up by git-filter-branch: say `git+ for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git+ update-ref -d`.++* Expire all reflogs with `git reflog expire \--expire=now \--all`.++* Garbage collect all unreferenced objects with `git gc+ \--prune-expire=now` (or if your git-gc is not new enough to have+ that option, `git repack -ad; git prune`).++ Author ------ Written by Petr "Pasky" Baudis <pasky@suse.cz>,
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:46:10
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
Signed-off-by: Johannes Schindelin <redacted>
---
Actually, when trying to add tests to Jan's patch, I realized that
I already have a patch in my personal tree, which makes --prune
accept an optional parameter...
Jan, what do you think? If you prefer --prune-expire, can you
please adapt my test?
Documentation/git-gc.txt | 9 ++++++++-
builtin-gc.c | 19 ++++++++++++-------
t/t5304-prune.sh | 20 ++++++++++++++++++++
3 files changed, 40 insertions(+), 8 deletions(-)
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository SYNOPSIS ---------'git gc' [--aggressive] [--auto] [--quiet]+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>] DESCRIPTION -----------
@@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A` option of 'git-repack'. Setting `gc.autopacklimit` to 0 disables automatic consolidation of packs.+--prune=<date>::+ Prune loose objects older than date (default is 2 weeks ago).+ This option is on by default.++--no-prune::+ Do not prune any loose objects.+ --quiet:: Suppress all progress reports.
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)"\"git help gc\" for more information.\n");}elseappend_option(argv_repack,-!strcmp(prune_expire,"now")?"-a":"-A",+prune_expire&&!strcmp(prune_expire,"now")+?"-a":"-A",MAX_ADD);if(pack_refs&&run_command_v_opt(argv_pack_refs,RUN_GIT_CMD))
@@ -371,7 +371,7 @@ warned. * Expire all reflogs with `git reflog expire \--expire=now \--all`. * Garbage collect all unreferenced objects with `git gc- \--prune-expire=now` (or if your git-gc is not new enough to have+ \--prune=now` (or if your git-gc is not new enough to have that option, `git repack -ad; git prune`).
From: Junio C Hamano <hidden> Date: 2016-06-15 22:46:10
Thomas Rast [off-list ref] writes:
quoted hunk
+git-filter-branch is often used to get rid of a subset of files,+usually with some combination of `\--index-filter` and+`\--subdirectory-filter`. If you want to physically shrink the+repository afterwards, you have some choices:++* Make sure you really removed all variants of a filename, if a blob+ was moved over its lifetime. `git log \--name-only \--follow \--all+ \-- filename` can help you find renames.++* Make sure you really filtered all refs: use `\--tag-name-filter cat+ \-- \--all` when calling git-filter-branch.++* Clone it with `git clone file:///path/to/repo`. The clone will not+ have the removed objects. See linkgit:git-clone[1]. (Note that+ cloning with a plain path just hardlinks everything!)++If you really don't want to clone it, for whatever reasons, check the+following points instead (in this order). This is a very destructive+approach, so *make a backup* or go back to cloning it. You have been+warned.++* Remove the original refs backed up by git-filter-branch: say `git+ for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git+ update-ref -d`.++* Expire all reflogs with `git reflog expire \--expire=now \--all`.++* Garbage collect all unreferenced objects with `git gc+ \--prune-expire=now` (or if your git-gc is not new enough to have+ that option, `git repack -ad; git prune`).
I am sorry, but the above does not flow well. The first two common
preparatory steps appear as if it is only for the "clone" method.
... often used to get rid of ... and people expect the resulting
repository to be smaller than the original, but you need a few
more steps to actually make it smaller, because git tries hard
not to lose your objects until you tell it to. First make sure
that:
* you really removed all...
* you reall filtered all...
Then, there are two ways to get a smaller repository. A safer way
is to clone, that keeps your original intact.
* Clone it with ...
If you do not want to ...
* Remove the original refs...
* Expire all...
From: Thomas Rast <hidden> Date: 2016-06-15 22:46:10
Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.
This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.
It may seem contradictory to document fallbacks for older Gits in
newer docs, but we want to point people at this as a FAQ answer, and
they will frequently not have the newest version installed.
Thanks to Björn Steinbrink and Junio C Hamano for comments and
corrections.
Signed-off-by: Thomas Rast <redacted>
---
Junio C Hamano wrote:
I am sorry, but the above does not flow well. The first two common
preparatory steps appear as if it is only for the "clone" method.
You're absolutely right. I should have re-read it more thoroughly
after the rearrangement...
I also changed the --prune-expire to --prune, and changed the
surrounding wording since it's now not a new option any more, just new
semantics for an old one.
Furthermore, I noticed that asciidoc tries to turn file:///... into a
link, mangling the entire paragraph's formatting in the process.
Apparently +++quoting+++ helps.
Documentation/git-filter-branch.txt | 41 +++++++++++++++++++++++++++++++++++
1 files changed, 41 insertions(+), 0 deletions(-)
@@ -339,6 +339,47 @@ git filter-branch --index-filter \ ---------------------------------------------------------------++Checklist for Shrinking a Repository+------------------------------------++git-filter-branch is often used to get rid of a subset of files,+usually with some combination of `\--index-filter` and+`\--subdirectory-filter`. People expect the resulting repository to+be smaller than the original, but you need a few more steps to+actually make it smaller, because git tries hard not to lose your+objects until you tell it to. First make sure that:++* You really removed all variants of a filename, if a blob was moved+ over its lifetime. `git log \--name-only \--follow \--all \--+ filename` can help you find renames.++* You really filtered all refs: use `\--tag-name-filter cat \--+ \--all` when calling git-filter-branch.++Then there are two ways to get a smaller repository. A safer way is+to clone, that keeps your original intact.++* Clone it with `git clone +++file:///path/to/repo+++`. The clone+ will not have the removed objects. See linkgit:git-clone[1]. (Note+ that cloning with a plain path just hardlinks everything!)++If you really don't want to clone it, for whatever reasons, check the+following points instead (in this order). This is a very destructive+approach, so *make a backup* or go back to cloning it. You have been+warned.++* Remove the original refs backed up by git-filter-branch: say `git+ for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git+ update-ref -d`.++* Expire all reflogs with `git reflog expire \--expire=now \--all`.++* Garbage collect all unreferenced objects with `git gc \--prune=now`+ (or if your git-gc is not new enough to support arguments to+ `\--prune`, use `git repack -ad; git prune` instead).++ Author ------ Written by Petr "Pasky" Baudis <pasky@suse.cz>,
From: Thomas Rast <hidden> Date: 2016-06-15 22:46:10
Johannes Schindelin wrote:
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
Prompted by discussion on IRC, I just spent some time staring at the
code, and for the code part and FWIW,
Acked-by: Thomas Rast [off-list ref]
Meaning that for all I can see, it handles every combination of
config, --prune and --no-prune correctly.
I have some non-code nits however:
quoted hunk
+--prune=<date>::+ Prune loose objects older than date (default is 2 weeks ago).+ This option is on by default.
This isn't really true; to the user, it defaults to the value of
'gc.pruneExpire'. That this is done by defaulting the state of
--prune to 2 weeks ago is an implementation detail.
+test_expect_success 'gc --no-prune && gc --prune=<date>' '++ before=$(git count-objects | sed "s/ .*//") &&+ BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&+ BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&+ test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&+ test -f $BLOB_FILE &&+ test-chmtime =-$((86400*5001)) $BLOB_FILE &&+ git gc --no-prune &&+ test 1 = $(git count-objects | sed "s/ .*//") &&+ test -f $BLOB_FILE &&+ git gc --prune=5002.days.ago &&+ test 1 = $(git count-objects | sed "s/ .*//") &&+ test -f $BLOB_FILE &&+ git gc --prune=5000.days.ago &&+ test 0 = $(git count-objects | sed "s/ .*//") &&+ test ! -f $BLOB_FILE++'+
I'm not a huge fan of such contiguous blocks, as a failure is harder
to pin to any of the several tests inside, especially with 'test' not
showing any output when it fails.
Also, it does test the interaction between having gc.pruneExpire set,
but passing --no-prune: the tests above leave gc.pruneExpire at
"2.days.ago". This wasn't immediately obvious however, and you should
probably set the config again just in case someone decides to insert a
test with a different value.
--
Thomas Rast
trast@{inf,student}.ethz.ch
From: Jan Krüger <hidden> Date: 2016-06-15 22:46:10
Hi,
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
@@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A`
option of
'git-repack'. Setting `gc.autopacklimit` to 0 disables
automatic consolidation of packs.
+--prune=<date>::
+ Prune loose objects older than date (default is 2 weeks ago).
+ This option is on by default.
Maybe refer to the config option at least briefly.
quoted hunk
++--no-prune::+ Do not prune any loose objects.+ --quiet:: Suppress all progress reports.
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:46:10
Thanks for both of your sets of comments.
This is the interdiff.
---
Documentation/git-gc.txt | 7 ++++---
builtin-gc.c | 2 +-
t/t5304-prune.sh | 24 +++++++++++++++++++++---
3 files changed, 26 insertions(+), 7 deletions(-)
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository SYNOPSIS ---------'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune] DESCRIPTION -----------
@@ -60,8 +60,9 @@ are consolidated into a single pack by using the `-A` option of automatic consolidation of packs. --prune=<date>::- Prune loose objects older than date (default is 2 weeks ago).- This option is on by default.+ Prune loose objects older than date (default is 2 weeks ago,+ overrideable by the config variable `gc.pruneExpire`). This+ option is on by default. --no-prune:: Do not prune any loose objects.
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:46:10
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
Signed-off-by: Johannes Schindelin <redacted>
---
Documentation/git-gc.txt | 10 +++++++++-
builtin-gc.c | 19 ++++++++++++-------
t/t5304-prune.sh | 38 ++++++++++++++++++++++++++++++++++++++
3 files changed, 59 insertions(+), 8 deletions(-)
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository SYNOPSIS ---------'git gc' [--aggressive] [--auto] [--quiet]+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune] DESCRIPTION -----------
@@ -59,6 +59,14 @@ are consolidated into a single pack by using the `-A` option of 'git-repack'. Setting `gc.autopacklimit` to 0 disables automatic consolidation of packs.+--prune=<date>::+ Prune loose objects older than date (default is 2 weeks ago,+ overrideable by the config variable `gc.pruneExpire`). This+ option is on by default.++--no-prune::+ Do not prune any loose objects.+ --quiet:: Suppress all progress reports.
@@ -161,7 +161,8 @@ static int need_to_gc(void)*/if(too_many_packs())append_option(argv_repack,-!strcmp(prune_expire,"now")?"-a":"-A",+prune_expire&&!strcmp(prune_expire,"now")?+"-a":"-A",MAX_ADD);elseif(!too_many_loose_objects())return0;
@@ -173,14 +174,15 @@ static int need_to_gc(void)intcmd_gc(intargc,constchar**argv,constchar*prefix){-intprune=0;intaggressive=0;intauto_gc=0;intquiet=0;charbuf[80];structoptionbuiltin_gc_options[]={-OPT_BOOLEAN(0,"prune",&prune,"prune unreferenced objects (deprecated)"),+{OPTION_STRING,0,"prune",&prune_expire,"date",+"prune unreferenced objects",+PARSE_OPT_OPTARG,NULL,(intptr_t)prune_expire},OPT_BOOLEAN(0,"aggressive",&aggressive,"be more thorough (increased runtime)"),OPT_BOOLEAN(0,"auto",&auto_gc,"enable auto-gc mode"),OPT_BOOLEAN('q',"quiet",&quiet,"suppress progress reports"),
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)"\"git help gc\" for more information.\n");}elseappend_option(argv_repack,-!strcmp(prune_expire,"now")?"-a":"-A",+prune_expire&&!strcmp(prune_expire,"now")+?"-a":"-A",MAX_ADD);if(pack_refs&&run_command_v_opt(argv_pack_refs,RUN_GIT_CMD))
From: Thomas Rast <hidden> Date: 2016-06-15 22:46:10
Johannes Schindelin wrote:
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
Signed-off-by: Johannes Schindelin <redacted>
---
Documentation/git-gc.txt | 10 +++++++++-
builtin-gc.c | 19 ++++++++++++-------
t/t5304-prune.sh | 38 ++++++++++++++++++++++++++++++++++++++
3 files changed, 59 insertions(+), 8 deletions(-)
From: Junio C Hamano <hidden> Date: 2016-06-15 22:46:12
Johannes Schindelin [off-list ref] writes:
With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.
This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
Signed-off-by: Johannes Schindelin <redacted>
Thanks.
Being able to say --no-$opt to an OPT_STRING to NULLify a variable that
has non-NULL default value is somewhat amusing ;-)