I just pulled from Linus' Linux repository ... and my
wrapper script alerted me to the fact that he had just
re-packed.
So, I logged into kernel.org and linked the new packfiles
across to my repo there, and did a "git prune-packed" to
clean away all the unneeded unpacked objects.
Back on my local box I merged all the new stuff that
had been pulled into my "release" and "test" branches.
(just a fast-forward for the release branch).
Next I used "scp" to copy Linus' new pack files to my
local objects/pack ... and ran git prune packed here.
Finally, after making sure that release and test
branches still built with all the new stuff I did
a "git push" to update my kernel repo. I saw this:
updating 'refs/heads/test'
from 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f
to 2e488d3f8f299eb0ed826055bc30003e2c27d909
updating 'refs/heads/release'
from 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404
to 581c1b14394aee60aff46ea67d05483261ed6527
Packing 10785 objects
Unpacking 10785 objects
100% (10785/10785) done
refs/heads/test: 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f -> 2e488d3f8f299eb0ed826055bc30003e2c27d909
refs/heads/release: 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404 -> 581c1b14394aee60aff46ea67d05483261ed6527
Now the "unpack" on kernel.org did the right thing and noticed
that over 9000 of the objects were already in the packfile. But
I wonder if it couldn't have been smarter and not sent them?
Or am I just subverting the whole paradigm by hand-copying
packfiles around?
-Tony
On Mon, 31 Oct 2005, Luck, Tony wrote:
Finally, after making sure that release and test
branches still built with all the new stuff I did
a "git push" to update my kernel repo. I saw this:
...
Packing 10785 objects
Unpacking 10785 objects
Now the "unpack" on kernel.org did the right thing and noticed
that over 9000 of the objects were already in the packfile. But
I wonder if it couldn't have been smarter and not sent them?
It should have been smarter, but I suspect you got caught by the fact that
kernel.org by default has git-0.99.8f on it, which has the old
pre-multi_ack code to figure out what the common commit was.
The pack-file contains over ten times as many objects, so you definitely
didn't get all of them - but because there has been a lot of merges
lately, and the common commit finder algorithm wasn't all that careful,
you ended up getting many more objects than you really needed.
Remember: the pack generation is not "exact" - it will often generate a
few extra objects for any non-trivial case (for example, it fundamentally
happens if there has been reverts: it won't realize that you had the older
version of a file already). It just so happens that the old algorithm had
some cases where it would decide on totally the wrong common commit, and
re-send a _lot: more objects than it needs.
I'm not sure multi-ack fixes it entirely either, but I think it makes it a
lot less likely (but even with multi-ack, the "file revert" case still
happens, so you should always expect that can get a _couple_ of
unnecessary objects).
Now, even the old stupid algorithm got the _easy_ cases obviously right,
so people might have incorrectly gotten the idea that it was careful and
exact, just because quite often it ends up being that in practice.
Linus
On Mon, 31 Oct 2005, Linus Torvalds wrote:
quoted
Now the "unpack" on kernel.org did the right thing and noticed
that over 9000 of the objects were already in the packfile. But
I wonder if it couldn't have been smarter and not sent them?
It should have been smarter, but I suspect you got caught by the fact that
kernel.org by default has git-0.99.8f on it, which has the old
pre-multi_ack code to figure out what the common commit was.
Oh, actually, I take that back.
This is not the multi-ack code at all, I suspect.
The problem is totally different: you copied the new pack to your
master.kernel.org repository, but you never updated any branches there.
So you had the objects, but git had no way of knowing. The pack generation
doesn't look at what _objects_ you have, it looks at what _refs_ you have.
And your refs were all to the old state.
So git actually did everything right (you can never rely on objects: you
may have a partial object list due to some earlier incomplete pull/push).
This is actually easy enough to fix up in one of several ways:
a) The "don't do that then" approach:
Don't go behind git's back and add objects on your own, and expect git
to realize what you did. ;^p
b) The "live with it" approach:
You copied the pack by hand, and that will keep git-unpack-objects
from duplicating the objects, but you'll still waste time and network
when trying pushing the objects (just once, though).
Ie this is what happened this time: nothing really lost, and the end
result is fine. Now you know why it happened, and you're fine.
c) The "I'm smarter than git" approach:
When you copy my objects, copy my reference to the top-of-tree too
(and rename it). NOTE! Now you need to be really careful, and you need
to make sure you copy _all_ the objects, because if you screw this up,
your repo will be missing objects that you claim are there, and it
will be all your fault.
I really don't advocate this approach at all. It's certainly doable,
but it's also the only approach where you can really screw up.
d) Just let git do it for you.
Copy the pack-files, or add my object directory as an "alternates" for
your object directory, do the "git prune-objects", and then _locally_
on master.kernel.org just do something like
git fetch ..linus-directory.. master:linus
which will still create the unnecessary pack-file and unpack it into
nothingness (since you have the objects in the pack-file you copied by
hand), but at least it won't eat any network bandwidth, and it will do
the right thing if it turns out that I've pushed something after doing
the pack-file, and fetch those individual objects in _addition_ to the
pack-file you snarfed by hand. It will also obviously update a ref in
your tree (the "linus" branch), so now when you send stuff later, it
will know all about the objects you already have.
e) Re-create the tree entirely
Blow away your tree on master entirely, just re-create it locally with
"git clone -l -s" from my tree (which will do all the "alternates"
object files for you), and then populate the result with a simple push
from your home tree.
f) any number of variations on a theme. IOW, there are endless ways to do
this.
Hmm?
Linus
On Mon, Oct 31, 2005 at 11:36:09AM -0800, Linus Torvalds wrote:
a) The "don't do that then" approach:
Don't go behind git's back and add objects on your own, and expect git
to realize what you did. ;^p
To be a responsible kernel.org git user I need to use packfiles. The
cheapest and easiest way to do that is to steal them from you.
b) The "live with it" approach:
You copied the pack by hand, and that will keep git-unpack-objects
from duplicating the objects, but you'll still waste time and network
when trying pushing the objects (just once, though).
Ie this is what happened this time: nothing really lost, and the end
result is fine. Now you know why it happened, and you're fine.
I like this one best. There's a pretty fat pipe between here and
kernel.org ... more of the time wasted appeared to be on the
unpacking part than on the transfer.
d) Just let git do it for you.
Copy the pack-files, or add my object directory as an "alternates" for
your object directory, do the "git prune-objects", and then _locally_
on master.kernel.org just do something like
Longer term (when everyone is pulling using the "git" protocol)
adding your object directory to my info/alternates looks to be
the best thing.
-Tony