David Kastrup [off-list ref] writes:
Duy Nguyen [off-list ref] writes:
quoted
OK with git://git.savannah.gnu.org/emacs.git we have
- a 209MB pack with --aggressive
- 1.3GB with --depth=50
- 1.3GB with --window=4000 --depth=32
- 1.3GB with --depth=20
- 821MB with --depth=250 for commits --before=2.years.ago, --depth=50
for the rest
So I don't think we should go with your following patch because the
size explosion is just too much no matter how faster it could be. An
immediate action could be just make --depth=250 configurable and let
people deal with it. A better option is something like "3 repack
steps" you described where we pack deep depth first, mark .keep, pack
shallower depth and combine them all into one.
I'm not really happy with --depth=250 producing 209MB while
--depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe
I did something wrong)
That does look strange: Emacs has a history of more than 30 years. But
the Git mirror is quite younger. Maybe one needs to make sure to use
the author date rather than the commit date here?
Another thing: did you really use --depth=250 here or did you use
--aggressive? It may be that the latter also sets other options?
--
David Kastrup
On Sat, Feb 22, 2014 at 3:53 PM, David Kastrup [off-list ref] wrote:
David Kastrup [off-list ref] writes:
quoted
Duy Nguyen [off-list ref] writes:
quoted
OK with git://git.savannah.gnu.org/emacs.git we have
- a 209MB pack with --aggressive
- 1.3GB with --depth=50
- 1.3GB with --window=4000 --depth=32
- 1.3GB with --depth=20
- 821MB with --depth=250 for commits --before=2.years.ago, --depth=50
for the rest
So I don't think we should go with your following patch because the
size explosion is just too much no matter how faster it could be. An
immediate action could be just make --depth=250 configurable and let
people deal with it. A better option is something like "3 repack
steps" you described where we pack deep depth first, mark .keep, pack
shallower depth and combine them all into one.
I'm not really happy with --depth=250 producing 209MB while
--depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe
I did something wrong)
That does look strange: Emacs has a history of more than 30 years. But
the Git mirror is quite younger. Maybe one needs to make sure to use
the author date rather than the commit date here?
I think commit date is fine because it covers a large portion of
objects (649946 per total 739990) and it does not (or should not)
affect object ordering in pack-objects/rev-list.
Another thing: did you really use --depth=250 here or did you use
--aggressive? It may be that the latter also sets other options?
I can't use --aggressive because I need to feed revisions directly to
pack-objects. --aggressive also sets --window=250. Thanks for
checking. My machine will have another workout session.
--
Duy
On Sat, Feb 22, 2014 at 4:14 PM, Duy Nguyen [off-list ref] wrote:
On Sat, Feb 22, 2014 at 3:53 PM, David Kastrup [off-list ref] wrote:
quoted
David Kastrup [off-list ref] writes:
quoted
Duy Nguyen [off-list ref] writes:
quoted
OK with git://git.savannah.gnu.org/emacs.git we have
- a 209MB pack with --aggressive
- 1.3GB with --depth=50
- 1.3GB with --window=4000 --depth=32
- 1.3GB with --depth=20
- 821MB with --depth=250 for commits --before=2.years.ago, --depth=50
for the rest
...
quoted
quoted
quoted
I'm not really happy with --depth=250 producing 209MB while
--depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe
I did something wrong)
....
quoted
Another thing: did you really use --depth=250 here or did you use
--aggressive? It may be that the latter also sets other options?
I can't use --aggressive because I need to feed revisions directly to
pack-objects. --aggressive also sets --window=250. Thanks for
checking. My machine will have another workout session.
And 800MB is reduced to 177MB, containing history older than 2 years.
The final pack is 199MB, within the size range of current --aggressive
and should be reasonably fast on most operations. Again blame could
still hit long delta chains but I think we should just unpack some
trees/blobs when we hit long delta chains.
I think we should update --aggressive to do it this way. So
- gc.aggressiveDepth defaults to 50 (or 20?), this is used for recent history
- gc.aggressiveDeepDepth defaults to 250 (or smaller??), used for
ancient history
- gc.aggressiveDeepOption is rev-list a rev-list option to define
"ancient history", default to --before=2.years.ago. This option could
be specified multiple times.
Both packing phases use the same gc.aggressiveWindow. We could add
gc.aggressiveDeepWindow too.
GSoC project?
--
Duy