Ramkumar Ramachandra [off-list ref] writes:
Junio C Hamano wrote:
quoted
it is
not a problem for the pack that consolidates young objects into a
single pack to contain some unreachable crufts.
So far, we have never considered putting unreachable objects in packs.
Let me ask the obvious question first: what happens when I push? Do I
pack up all the loose objects quickly (without bothering about
reachability) and send unreachable cruft to the server?
No.
I thought the discussion was about making the local gc cheaper, and
the "Imagine we have a cheap way" was to address it by assuming that
the daily "pack young objects into a single pack" can be sped up if
we did not have to traverse history. More permanent packs (the
older ones in "set of packs staggered by age" Martin proposes) in
the repository should go through the normal history traversal route.
And of course we do not transfer objects that are not asked for from
or to a repository over pack tranfer.
Most importantly, it is not about butchering the pack machinery in
such a way that we can create _only_ such "non history traversal"
packs.
So I do not see how that question is "obvious". The question
obviously pointless and misses the mark by wide margin? The
question makes it obvious that whoever asks it does not understand
how Git works?
On Thursday, August 08, 2013 10:56:38 am Junio C Hamano
wrote:
I thought the discussion was about making the local gc
cheaper, and the "Imagine we have a cheap way" was to
address it by assuming that the daily "pack young
objects into a single pack" can be sped up if we did not
have to traverse history. More permanent packs (the
older ones in "set of packs staggered by age" Martin
proposes) in the repository should go through the normal
history traversal route.
Assuming I understand what you are suggesting, would these
"young object" likely still get "deduped" in an efficient
way without doing history traversal (it sounds like they
would)? In other words, if I understand correctly, it would
save time by not pruning unreferenced objects, but it would
still be deduping things and delta compressing also, so you
would still likely get a great benefit from creating these
young object packs? In other words, is there still a good
chance that my 317 new pack files which included a 33M pack
file will still get consolidated down to something near 8M?
If so, then yeah this might be nice, especially if the
history traversal is what would speed this up. Because
today, my solution mostly saves IO and not time. I think it
still saves time, I believe I have seen up to a 50% savings,
but that is nothing compared to massive, several orders of
magnitude IO savings. But if what you suggest could also
give massive time (orders of magnitude) savings along with
the IO improvements I am seeing, then suddenly repacking
regularly would become very cheap even on large repos.
The only time consuming piece would be pruning then? Could
bitmaps eventually help out there?
-Martin
--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation
Junio C Hamano wrote:
So I do not see how that question is "obvious". The question
obviously pointless and misses the mark by wide margin? The
question makes it obvious that whoever asks it does not understand
how Git works?
Shall we all sit and mourn over the fact that I don't understand how
Git works, or are you willing to explain it to me?
And of course we do not transfer objects that are not asked for from
or to a repository over pack tranfer.
Most importantly, it is not about butchering the pack machinery in
such a way that we can create _only_ such "non history traversal"
packs.
I asked you a very simple question: what happens when I do "git push"?
Instead of answering the question, you butchered the pack machinery to
"only" create packs with garbage in them (aka. stripped out the
reachability analysis code completely), and blamed me for doing it.
Explain it to me in plain English without getting agitated:
1. I'm on my terminal doing various repository operations: constantly
creating new objects and moving my refs around to create unreachable
objects. I have lots of loose objects.
2. I say "git push". What happens? A reachability analysis is
performed on my loose objects, and the ones reachable by the ref I'm
sending are packed up and sent over the network. Now, I no longer have
any loose objects.
3. After a few days of working, the gc heuristics figure out that I
have too much garbage and too many packs; a cleanup is required. The
gc --auto which doesn't tolerate fragmentation: it tries to put
everything into one large pack.
Loop.
We're talking about tackling the gc aggression problem in 3. And you
propose putting the young objects in a pack without performing
reachability analysis: I'm asking how this is going to benefit me;
when I say "git push" (or when gc decides to repack), won't I need to
explode the young pack into loose objects, do a reachability analysis,
and repack anyway?