From: Junio C Hamano <hidden> Date: 2016-06-15 22:55:57
Jeff King [off-list ref] writes:
quoted
I also wonder if we would be helped by another "repack" mode that
coalesces small packs into a single one with minimum overhead, and
run that often from "gc --auto", so that we do not end up having to
have 50 packfiles.
...
I'm not sure. If I understand you correctly, it would basically just be
concatenating packs without trying to do delta compression between the
objects which are ending up in the same pack. So it would save us from
having to do (up to) 50 binary searches to find an object in a pack, but
would not actually save us much space.
The point is not about space. Disk is cheap, and it is not making
it any worse than what happens to your target audience, that is a
fetch-only repository with only "gc --auto" in it, where nobody
passes "-f" to "repack" to cause recomputation of delta.
What I was trying to seek was a way to reduce the runtime penalty we
pay every time we run git in such a repository.
- Object look-up cost will become log2(50*n) from 50*log2(n), which
is about 50/log2(50) improvement;
- System resource cost we incur by having to keep 50 file
descriptors open and maintaining 50 mmap windows will reduce by
50 fold.
- Anything else I missed?
I would be interested to see the timing on how quick it is compared to a
real repack,...
Yes, that is what I meant by "wonder if we would be helped by" ;-)
But how do these somewhat mediocre concatenated packs get turned into
real packs?
How do they get processed in a fetch-only repositories that
sometimes run "gc --auto" today? By runninng "repack -a -d -f"
occasionally, perhaps?
At some point, you would need to run a repack that involves a real
object-graph traversal that feeds you the paths for objects to
obtain a reasonably compressed pack. We can inspect existing packs
and guess a rough traversal order for commits, but for trees and
blobs, you cannot unify existing delta chains from multiple packs
effectively with data in the pack alone.
From: Jeff King <hidden> Date: 2016-06-15 22:55:57
On Tue, Jan 29, 2013 at 07:58:01AM -0800, Junio C Hamano wrote:
The point is not about space. Disk is cheap, and it is not making
it any worse than what happens to your target audience, that is a
fetch-only repository with only "gc --auto" in it, where nobody
passes "-f" to "repack" to cause recomputation of delta.
What I was trying to seek was a way to reduce the runtime penalty we
pay every time we run git in such a repository.
- Object look-up cost will become log2(50*n) from 50*log2(n), which
is about 50/log2(50) improvement;
Yes and no. Our heuristic is to look at the last-used pack for an
object. So assuming we have locality of requests, we should quite often
get "lucky" and find the object in the first log2 search. Even if we
don't assume locality, a situation with one large pack and a few small
packs will have the large one as "last used" more often than the others,
and it will also have the looked-for object more often than the others
So I can see how it is something we could potentially optimize, but I
could also see it being surprisingly not a big deal. I'd be very
interested to see real measurements, even of something as simple as a
"master index" which can reference multiple packfiles.
- System resource cost we incur by having to keep 50 file
descriptors open and maintaining 50 mmap windows will reduce by
50 fold.
I wonder how measurable that is (and if it matters on Linux versus less
efficient platforms).
quoted
I would be interested to see the timing on how quick it is compared to a
real repack,...
Yes, that is what I meant by "wonder if we would be helped by" ;-)
There is only one way to find out... :)
Maybe I am blessed with nice machines, but I have mostly found the
repack process not to be that big a deal these days (especially with
threaded delta compression).
quoted
But how do these somewhat mediocre concatenated packs get turned into
real packs?
How do they get processed in a fetch-only repositories that
sometimes run "gc --auto" today? By runninng "repack -a -d -f"
occasionally, perhaps?
Do we run "repack -adf" regularly? The usual "git gc" procedure will not
use "-f", and without that, we will not even consider making deltas
between objects that were formerly in different packs (but now are in
the same pack).
So you are avoiding doing medium-effort packs ("repack -ad") in favor of
doing potentially quick packs, but occasionally doing a big-effort pack
("repack -adf"). It may be reasonable advice to "repack -adf"
occasionally, but I suspect most people are not doing it regularly (if
only because "git gc" does not do it by default).
-Peff
On Tue, Jan 29, 2013 at 1:19 PM, Jeff King [off-list ref] wrote:
On Tue, Jan 29, 2013 at 07:58:01AM -0800, Junio C Hamano wrote:
quoted
The point is not about space. Disk is cheap, and it is not making
it any worse than what happens to your target audience, that is a
fetch-only repository with only "gc --auto" in it, where nobody
passes "-f" to "repack" to cause recomputation of delta.
What I was trying to seek was a way to reduce the runtime penalty we
pay every time we run git in such a repository.
- Object look-up cost will become log2(50*n) from 50*log2(n), which
is about 50/log2(50) improvement;
Yes and no. Our heuristic is to look at the last-used pack for an
object. So assuming we have locality of requests, we should quite often
get "lucky" and find the object in the first log2 search. Even if we
don't assume locality, a situation with one large pack and a few small
packs will have the large one as "last used" more often than the others,
and it will also have the looked-for object more often than the others
Opening all of those files does impact performance. It depends on how
slow your open(2) syscall is. I know on Mac OS X that its not the
fastest function we get from the C library. Performing ~40 opens to
look through the most recent pack files and finally find the "real"
pack that contains that tag you asked `git show` for isn't that quick.
Some of us also use Git on filesystems that are network based, and
slow compared to local disk Linux ext2/3/4 with gobs of free RAM.
So I can see how it is something we could potentially optimize, but I
could also see it being surprisingly not a big deal. I'd be very
interested to see real measurements, even of something as simple as a
"master index" which can reference multiple packfiles.
I actually tried this many many years ago. There are threads in the
archive about it. Its slower. We ruled it out.
quoted
- System resource cost we incur by having to keep 50 file
descriptors open and maintaining 50 mmap windows will reduce by
50 fold.
I wonder how measurable that is (and if it matters on Linux versus less
efficient platforms).
It does matter. We know it has a negative impact on JGit even on Linux
for example. You don't want 300 packs in a repository. 50 might be
tolerable. 300 is not.
From: Jeff King <hidden> Date: 2016-06-15 22:56:01
On Thu, Jan 31, 2013 at 08:47:37AM -0800, Shawn O. Pearce wrote:
quoted
quoted
- System resource cost we incur by having to keep 50 file
descriptors open and maintaining 50 mmap windows will reduce by
50 fold.
I wonder how measurable that is (and if it matters on Linux versus less
efficient platforms).
It does matter. We know it has a negative impact on JGit even on Linux
for example. You don't want 300 packs in a repository. 50 might be
tolerable. 300 is not.
I'd love to see numbers if you have them. It's not that I don't believe
it is slower, but knowing _how much_ is important when thinking about
what kind of performance increase we are looking to get (which in turn
impacts how much effort to put into the repacking).
-Peff
On Fri, Feb 1, 2013 at 1:14 AM, Jeff King [off-list ref] wrote:
On Thu, Jan 31, 2013 at 08:47:37AM -0800, Shawn O. Pearce wrote:
quoted
quoted
quoted
- System resource cost we incur by having to keep 50 file
descriptors open and maintaining 50 mmap windows will reduce by
50 fold.
I wonder how measurable that is (and if it matters on Linux versus less
efficient platforms).
It does matter. We know it has a negative impact on JGit even on Linux
for example. You don't want 300 packs in a repository. 50 might be
tolerable. 300 is not.
I'd love to see numbers if you have them. It's not that I don't believe
it is slower, but knowing _how much_ is important when thinking about
what kind of performance increase we are looking to get (which in turn
impacts how much effort to put into the repacking).
Never done a formal experiment. Just working from memory where 4 years
and 3 desks ago I got called because one of our Git servers was
struggling to keep up with user git fetch commands. Turns out the
repository had O(200) pack files. git gc that normally took only about
5 minutes took a hellva lot longer, like an hour or more.
The problem happened because the server was saving every push to a
pack and never exploding incoming packs to loose objects. This meant
the thin packs from the wire got delta bases appended to them.
pack-objects was looking at O(50) different alternatives for most
objects when trying to decide which one it should copy into the output
pack... for either a fetch/clone client, or the gc I was trying to run
to fix the repository.