From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:00
quoted
quoted
quoted
quoted
"LT" == Linus Torvalds [off-list ref] writes:
LT> Now, there's still a misfeature there, which is that when you create a new
LT> object, it doesn't check whether that object already exists in the
LT> pack-file, so you'll end up with a few recent objects that you really
LT> don't need (notably tree objects), and we'll fix that eventually.
Patch will be sent separately.
LT> ... Also, please note that the pack-file _only_ packs the commits
LT> and the things reachable from them ...
Shouldn't feeding "git-rev-list --object" output plus
handcrafted list of objects in 2.6.11 tree object to
git-pack-objects just work???
LT> Maybe you might not want to switch over yet, and as mentioned, rsync then
LT> ends up not being a good way to sync (nor git-local-pull), but the
LT> "git-http/ssh-pull" family should hopefully just work.
No. The pull protocol Dan did expects to throw compressed
representation around on the wire (which is valid if you assume
uncompressed transfer) and does not use read-sha1-file --
write-sha1-file pair, so all three do not work.
LT> ... Also, please note that the pack-file _only_ packs the commits
LT> and the things reachable from them ...
Shouldn't feeding "git-rev-list --object" output plus
handcrafted list of objects in 2.6.11 tree object to
git-pack-objects just work???
You could do that. And yes, we can add support for "tag" objects too
(which the packing doesn't do at all right now. So this is not a
"fundamental" problem, it's just a practical one right now.
quoted
[.. git-ssh-pull hopefully working ..]
No. The pull protocol Dan did expects to throw compressed
representation around on the wire (which is valid if you assume
uncompressed transfer) and does not use read-sha1-file --
write-sha1-file pair, so all three do not work.
Fair enough. I'd prefer for the pull/push to push object packs around
anyway, so there's some more work there..
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:00
On Mon, 27 Jun 2005, Linus Torvalds wrote:
quoted
quoted
[.. git-ssh-pull hopefully working ..]
No. The pull protocol Dan did expects to throw compressed
representation around on the wire (which is valid if you assume
uncompressed transfer) and does not use read-sha1-file --
write-sha1-file pair, so all three do not work.
Fair enough. I'd prefer for the pull/push to push object packs around
anyway, so there's some more work there..
It shouldn't be hard to add; the main issue is determining when
transfering a pack file is a good idea, because it probably doesn't make
sense to transfer a pack file just because the source side has an object
that the target side wants in that pack. (If you pull from someone who
packed up the whole history of everything, which you already have, into a
file with one new commit, you'd be sad to get the huge thing; you really
want a little custom (or just limited) pack file.)
The ideal thing is probably to pick up some tricks from Mercurial in
figuring out what needs to be transferred, and have the source side write
a pack file directly to the connection, which the target side would then
save directly. I never worked out exactly what those tricks were, though.
The next trick would be to put something in place of cleverly-chosen
objects to specify what pack file they're in, so that the HTTP client
could find things from a packed repository. (Or we could just have an
option to unpack post-transfer.)
-Daniel
*This .sig left intentionally blank*
It shouldn't be hard to add; the main issue is determining when
transfering a pack file is a good idea, because it probably doesn't make
sense to transfer a pack file just because the source side has an object
that the target side wants in that pack.
Oh, you'd never just transfer the whole big pack-file at all: you'd just
create a new one. And creatign a new one is just a matter of finding the
common parent, and then doing
git-rev-list --objects common..HEAD | git-pack-file .git/tmp-pack
and then you send the result to the other side..
Linus
Oh, you'd never just transfer the whole big pack-file at all: you'd just
create a new one. And creatign a new one is just a matter of finding the
common parent, and then doing
git-rev-list --objects common..HEAD | git-pack-file .git/tmp-pack
and then you send the result to the other side..
To clarify: this also works with objects that are already in another
pack-file (now that Junio fixed the "get size of a deltified packed
entry"), so you can have any number of unpacked objects in your objects
directory, _and_ a pack-file (or several), and you can generate a new
temporary pack-file just for sending somewhere else that contains
arbistrary parts of that (ie a mix of objects that are in your "main"
packfiles and objects that are unpacked).
You don't have to use "git-rev-list" to generate the objects, btw,
git-pack-file takes an arbitrary list of object ID's (plus a "packing
hint" in the form of a filename that is not required, but that can help
the packing heuristics, and that git-rev-list does provide).
I'll also fix up git-pack-file to be able to pack tag objects (and the
unpacking to understand them), so that any valid object can be packed.
Right now it only handles the objects that git-rev-list knows about.
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:00
On Tue, 28 Jun 2005, Linus Torvalds wrote:
I'll also fix up git-pack-file to be able to pack tag objects (and the
unpacking to understand them), so that any valid object can be packed.
Right now it only handles the objects that git-rev-list knows about.
Actually, the ideal thing would be to move the packing code into an object
file that git-ssh-push can include; that way it can write directly to the
socket instead of going through disk, and it can also go from getting the
remote end's list of common ancestors to having a pack to send without
needing to exec a script.
-Daniel
*This .sig left intentionally blank*
Actually, the ideal thing would be to move the packing code into an object
file that git-ssh-push can include; that way it can write directly to the
socket instead of going through disk
It doesn't work very easily that way because the index file (which
contains the object list and the offsets into the pack file) cannot be
created until after the pack file has been created (and we don't want to
evaluate that one in memory, since it can be quite big).
Now, what we could do is to stream out the pack file first to stdout, and
write the index file afterwards. But since we don't know how big the pack
file will be when we start packing, and the pack-file can contain
basically arbitrary patterns, that requires that the receiver actually
parse the pack-file as it comes in.
The format of the pack-file is a fairly trivial data stream of
- rinse and repeat for each object:
- one character of type of file (C, T, B, G, D for "commit", "tree",
"blob", "tag" or "delta" respectively)
- four bytes of network-order unpacked data length
- [ if delta: 20 bytes of delta object ID ]
- zlib-packed data (length unknown, except we know how much we want
it to unpack to)
- Finally at the end: 20 bytes of SHA1 of the pack-file contents (up to
the SHA1)
so it's actually possible to pick up the objects as they come off the
stream, since the SHA1 name is defined by the contents and you don't need
the index file unless you want to look things up.
So the receiver side could try this algorithm:
- unpack each object in memory on the receiving side
If the unpack failed, it must have been the SHA1 at the end, so
verify it!
- if it's a delta object and you haven't seen the object it's a delta
against, keep it in memory.
- if it's a non-delta object, just write it to the object store, and try
to resolve any delta objects you have pending that this new object
satisfies. That in turn creates other objects that may have more deltas
they satisfy etc.
which looks quite doable. The delta objects are small, so keeping them in
memory shouldn't be a problem (especially since we _tend_ to write deltas
after the object they depend on).
I can certainly add an option to git-pack-file that disables writing of
the index file, and just writes the pack-file to stdout. I'm not sure I
want to write the "parse incoming pack-file" thing, but git-unpack-objects
comes _reasonably_ close (but right now it seeks around using the index
file to resolve deltas, instead of keeping them in memory and resolving
them when possible). But I can make the infrastructure ready for it.
Sounds like a plan.
Linus
I can certainly add an option to git-pack-file that disables writing of
the index file, and just writes the pack-file to stdout.
Done.
I'm not sure I
want to write the "parse incoming pack-file" thing, but git-unpack-objects
comes _reasonably_ close (but right now it seeks around using the index
file to resolve deltas, instead of keeping them in memory and resolving
them when possible).
I'm still thinking about this one. I think I'll just do it.
One problem here is that since we don't know how big the incoming
pack-file will be, in a streaming input environment the receiver needs to
either make the pack-file reception be the last thing it sees, or it will
have to live with the fact that "git-unpack-objects" will read some more
than it needs before it notices that it got it all...
We can handle the latter either by padding (make the rule be that
git-unpack-file will always read in chunks of 4kB max, and pad the output
with 4kB of zero bytes or something, and then you can execute
git-unpack-objects and continue reading stdin afterwards, removing any
zeroes that git-unpack-file didn't eat), or by having git-unpack-objects
flush anything after the final SHA1 to _its_ stdout, so that you can get
the following data/commands in the stream from the unpack-file thing.
Ugly, in any case.
Linus
Why not chunk the thing?
In other words, the stream shouldn't be
"here's a big-ass packfile of unknown size"
but an arbitrary number of
"here's a N-byte sized chunk of the current pack file"
snippets, followed by a
"here's the SHA1 of the whole thing"
packet.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
Be like a duck -- keep calm and unruffled on the surface but paddle like the
devil under water.
Having the number of files sent first would work too, I'd think.
I'm wary of trying to interpret something non-decompressible as a sha1
chunk, however -- the set of random bytes that, to zlib, look like a
sufficiently valid zip header that it wants to read more than 20 of them
before punting is certainly not zero.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
I was sure the old fellow would never make it
to the other side of the curb when I struck him.
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:00
On Tue, 28 Jun 2005, Linus Torvalds wrote:
On Tue, 28 Jun 2005, Linus Torvalds wrote:
quoted
I can certainly add an option to git-pack-file that disables writing of
the index file, and just writes the pack-file to stdout.
Done.
What I actually meant was that it would be useful for git-ssh-push to be
able to pack stuff as a function call rather than execing an external
program, because just sticking git-ssh-push at the end of a pipeline
doesn't work if you don't remember what the remote side has.
quoted
I'm not sure I
want to write the "parse incoming pack-file" thing, but git-unpack-objects
comes _reasonably_ close (but right now it seeks around using the index
file to resolve deltas, instead of keeping them in memory and resolving
them when possible).
I'm still thinking about this one. I think I'll just do it.
One possibility would be to put a special type tag (like '\0') before the
hash, so that the format is more deterministic.
One problem here is that since we don't know how big the incoming
pack-file will be, in a streaming input environment the receiver needs to
either make the pack-file reception be the last thing it sees, or it will
have to live with the fact that "git-unpack-objects" will read some more
than it needs before it notices that it got it all...
In a completely streaming environment, yes; but the receiving side is the
one sending commands, so you don't run into the next thing unless you're
overlapping requests. Failing that, we can just keep a 4k buffer of stuff
we've already read around; we don't have to worry about reading into
something we won't want to read at all.
-Daniel
*This .sig left intentionally blank*
I'm not sure I
want to write the "parse incoming pack-file" thing, but git-unpack-objects
comes _reasonably_ close (but right now it seeks around using the index
file to resolve deltas, instead of keeping them in memory and resolving
them when possible).
I'm still thinking about this one. I think I'll just do it.
Ok, done. I had to basically rewrite that unpacking logic, but the end
result is actually slightly smaller and cleaner, and it can now unpack
from a stream. That stream reading logic that uncompresses directly from
the stream buffer might be considered a bit too subtle (and somebody
should really double-check it), but hey, it works for me.
In fact, I just did this:
#
# Create empty git archive "~/unpack"
#
mkdir ~/unpack
cd ~/unpack
git-init-db
#
# Copy the git archive there over a pipe
#
cd ~/git
git-rev-list --objects HEAD | git-pack-objects --depth=50 --window=50 --stdout | (cd ~/unpack ; git-unpack-objects)
#
# Go to new archive, set up the head, and fsck to verify
#
cd ~/unpack
cat ~/git/.git/HEAD > .git/HEAD
git-fsck-cache --unreachable
Now, the above is a silly example, since I _could_ just have moved the
pack file into .git/objects/pack, but that was not the point of this whole
thing. The point was to do what a "git-ssh-push" would basically boil down
to.
I'd like somebody who knows zlib intimately to take a look at how I do the
streaming input thing (in particular, the "use(len - stream.avail_in);"
part in the inflate loop in the "get_data()" function).
Linus
Shouldn't feeding "git-rev-list --object" output plus
handcrafted list of objects in 2.6.11 tree object to
git-pack-objects just work???
You could do that. And yes, we can add support for "tag" objects too
(which the packing doesn't do at all right now. So this is not a
"fundamental" problem, it's just a practical one right now.
Ok, I've added the logic to "git-rev-list --object" to handle arbitrary
object dependencies.
So you can do things like this, if you want to:
git-rev-list --object HEAD ^v2.6.11-tree
which basically generates the complete list of every object reachable from
HEAD, but not reachable from the v2.6.11 tree. It also understands about
tags, so if you do
git-rev-list --object v2.6.12 ^v2.6.11-tree
the end result will have the "v2.6.12" tag in it (along with all the
objects reachable from it, but not reachable from v2.6.11-tree).
What does this mean? It means that you can do a "push" from repository "a"
to repository "b" by doing
- in "b", do
refs_in_b=($(find .git/refs -type f | xargs cat))
- in "a" do
refs_in_a=($(find .git/refs -type f | xargs cat))
- then, in "a", do
git-rev-list "${refs_in_a[@]}" --not "${refs_in_b[@]}" |
git-pack-objects --stdout > push.pack
to generate the objects pack in "push.pack"
- then, in "b", do
git-unpack-objects < push.pack
and you now have moved over _all_ the objects that were referenced in "a",
but not in "b". Including tags etc. So after that last stage, when you've
unpacked the objects, the only thing left to do is to make the refs in "b"
point to the new references from "a" (which basically boils down to a
"cp", except it would be good to verify that the refs in "b" still have
the same values as they did before we did the object push).
Daniel (or anybody else), interested? Please?
Of course, you can do this one branch at a time, too, if you want to, but
the above was meant as an example of how you can actually do all the
branches in one single pack-file, which is a lot more efficient (if you do
it one branch at a time, you'll quite possible end up transferring objects
that are reachable in other branches multiple times, while the "all in one
go" thing will pack each object just once).
Now, have I actually _tested_ the above? Hell no. But all the heavy
lifting should now be done for doing an efficient "git push" that pushes
all branches in one go (or one at a time, it's your choice on how you end
up using git-rev-list).
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:01
On Wed, 29 Jun 2005, Linus Torvalds wrote:
and you now have moved over _all_ the objects that were referenced in "a",
but not in "b". Including tags etc. So after that last stage, when you've
unpacked the objects, the only thing left to do is to make the refs in "b"
point to the new references from "a" (which basically boils down to a
"cp", except it would be good to verify that the refs in "b" still have
the same values as they did before we did the object push).
Daniel (or anybody else), interested? Please?
I'll probably get to this over the weekend.
Of course, you can do this one branch at a time, too, if you want to, but
the above was meant as an example of how you can actually do all the
branches in one single pack-file, which is a lot more efficient (if you do
it one branch at a time, you'll quite possible end up transferring objects
that are reachable in other branches multiple times, while the "all in one
go" thing will pack each object just once).
It should transfer each only once if you recalculate "refs_in_b" after
each push, right? Or is the marking for "--objects ^commit" still not
tight wrt object and tree files? I think branch-at-a-time is preferable
for the case where the source doesn't want to send quite everything, and
the target doesn't necessarily want everything named the same.
Now, have I actually _tested_ the above? Hell no. But all the heavy
lifting should now be done for doing an efficient "git push" that pushes
all branches in one go (or one at a time, it's your choice on how you end
up using git-rev-list).
The one thing I can think of is whether things will blow up if the target
repository has heads that aren't in the source, at which point the source
has no clue what to exclude. I.e.:
parent -- new-b
\
new-a
If I've moved the head on b forward to new-b, and a wants to push new-a
(as a new branch, perhaps), refs_in_b has only new-b, refs_in_a has parent
and new-a, and git-rev-list in a can't see that b has parent (and
everything upwards of that). You probably just don't want to do this, but
I bet that some people will (e.g. projects that synchronize through a
shared-owner repository).
-Daniel
*This .sig left intentionally blank*
Of course, you can do this one branch at a time, too, if you want to, but
the above was meant as an example of how you can actually do all the
branches in one single pack-file, which is a lot more efficient (if you do
it one branch at a time, you'll quite possible end up transferring objects
that are reachable in other branches multiple times, while the "all in one
go" thing will pack each object just once).
It should transfer each only once if you recalculate "refs_in_b" after
each push, right?
Yes, you can do it that way too. It will possibly not pack as well due to
giving you fewer opportunities for deltas, but that's likely not a huge
issue.
The one thing I can think of is whether things will blow up if the target
repository has heads that aren't in the source
Right. I think that's a "feature" of pushing: you cannot push to an
archive that has state that you don't know about. Ie you can only push to
something that is a proper subset of what you are (on a per-branch basis,
of course - not necessarily on a "global" stage - so you could push just
_one_ branch, even if another branch was ahead of where you are).
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:01
On Wed, 29 Jun 2005, Linus Torvalds wrote:
On Wed, 29 Jun 2005, Daniel Barkalow wrote:
quoted
The one thing I can think of is whether things will blow up if the target
repository has heads that aren't in the source
Right. I think that's a "feature" of pushing: you cannot push to an
archive that has state that you don't know about. Ie you can only push to
something that is a proper subset of what you are (on a per-branch basis,
of course - not necessarily on a "global" stage - so you could push just
_one_ branch, even if another branch was ahead of where you are).
The issue is really distinguishing the "other" branches I don't care about
from the one that I do care about. With -w, I almost certainly care about
the ref I'm writing, but that doesn't help for refs that are new (new
branches or tags), for which I care about some other thing. Also, the
failure is a bit hard to detect, I think, in that I could find I do
recognize some ancient thing that's barely useful for exclusion, and miss
something that should exclude almost everything but it's been updated. In
any case, when things go wrong we simply send stuff the recipient already
has, so it's not the end of the world. (And there's probably some clever
way of dealing with it)
-Daniel
*This .sig left intentionally blank*