Ok,
I'm happy to say that the first cut of my new packed-object-sending thing
seems to work. I have successfully sent updates both locally and over ssh,
and it seems to work fine, although it has some limitations.
The syntax is very simple indeed:
git-send-pack destination
will go to the destination (which can be either a local directory or a
remote ssh one, with the remote destination format currently being _only_
the "machine:path" format), and it will go through all the refs in the
remote destination, compare them with the local ones, and create a pack
that updates from one to the other.
If the pack/unpack sequence is successful, it then updates the refs at the
other end, and is done.
My quick tests were very successful, in the sense that it even performed
really well. But I only tested some small updates.
Anyway, what are the limitations? Here's a few obvious ones:
- the code actually contains support for limiting the refs to be updated
on the remote end, but I don't actually pass the arguments to the
remote git-receive-pack binary yet, so this is currently not
functional. Call me lazy.
- the thing currently refuses to create new refs. Again, this is mainly
just me being lazy: it should be easy to add support for creating a new
branch, it just requires some care to make sure that we take the old
branches into account when generating the pack-file so that we don't
send too many objects over.
- I really hate how "ssh" apparently cannot be told to have alternate
paths. For example, on master.kernel.org, I don't control the setup, so
I can't install my own git binaries anywhere except in my ~/bin
directory, but I also cannot get ssh to accept that that is a valid
path. This one really bums me out, and I think it's an ssh deficiency.
You apparently have to compile in the paths at compile-time into sshd,
and PermitUserEnvironment is disabled by default (not that it even
seems to work for the PATH environment, but that may have been my
testing that didn't re-start sshd).
That just sucks.
- It doesn't update the working directory at the other end. This is fine
for what it's intended for (pushing to a central "raw" git archives),
so this could be considered a feature, but it's worth pointing out.
Only a "pull" will update your working directory, and this pack sending
really is meant to be used in a kind of "push to central archive" way.
- this is also (at least once we've tested it a lot more and added the
code to allow it to create new refs on the remote side) meant to be a
good way to mirror things out, since clearly rsync isn't scaling.
However, I don't know what the rules for acceptable mirroring
approaches are, and it's entirely possible (nay, probable) that an ssh
connection from the "master" ain't it. It would be good to know what
(of any) would be acceptable solutions..
Anyway, please do give it a test. I think I'll use this to sync up to
kernel.org, except I _really_ would want to solve that ssh issue some
other way than hardcoding the /home/torvalds/bin/ path in my local
copies.. If somebody knows a good solution, pls holler.
Linus
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:42:01
Have you tried something like the following?
ssh torvalds@master.kernel.org \
'/bin/sh -c "export PATH=/tmp/foo:$PATH ; env"'
Linus Torvalds wrote:
...
>
Anyway, please do give it a test. I think I'll use this to sync up to
kernel.org, except I _really_ would want to solve that ssh issue some
other way than hardcoding the /home/torvalds/bin/ path in my local
copies.. If somebody knows a good solution, pls holler.
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:42:01
Damn! That should have been:
ssh torvalds@master.kernel.org \
'/bin/sh -c "export PATH=~/tmp/foo:$PATH ; env"'
A Large Angry SCM wrote:
Have you tried something like the following?
ssh torvalds@master.kernel.org \
'/bin/sh -c "export PATH=/tmp/foo:$PATH ; env"'
Linus Torvalds wrote:
quoted
...
>
quoted
Anyway, please do give it a test. I think I'll use this to sync up to
kernel.org, except I _really_ would want to solve that ssh issue some
other way than hardcoding the /home/torvalds/bin/ path in my local
copies.. If somebody knows a good solution, pls holler.
From: Jan Harkes <jaharkes@cs.cmu.edu> Date: 2016-06-15 22:42:01
On Thu, Jun 30, 2005 at 10:54:48AM -0700, Linus Torvalds wrote:
Anyway, please do give it a test. I think I'll use this to sync up to
kernel.org, except I _really_ would want to solve that ssh issue some
other way than hardcoding the /home/torvalds/bin/ path in my local
copies.. If somebody knows a good solution, pls holler.
I've got a couple of 'export FOO=bar' lines in ~/.bashrc on the
"remote-side" and it looks like they are set correctly when
I do something like "ssh remote.host env".
Jan
Have you tried something like the following?
ssh torvalds@master.kernel.org \
'/bin/sh -c "export PATH=/tmp/foo:$PATH ; env"'
The point is that the user does not call "ssh" itself, but git-send-pack
does it automatically.
And that means that git-send-pack will always do the same thing, for any
host it is given. If one host needs a special PATH, that's an effing pain.
However, Kees Cook points out that it's driver error: I set up my PATH in
.bash_profile, and if I just do it in .bashrc instead it all works.
Danke,
Linus
From: Mike Taht <hidden> Date: 2016-06-15 22:42:01
However, I don't know what the rules for acceptable mirroring
approaches are, and it's entirely possible (nay, probable) that an ssh
connection from the "master" ain't it. It would be good to know what
(of any) would be acceptable solutions..
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
However, I don't know what the rules for acceptable mirroring
approaches are, and it's entirely possible (nay, probable) that an ssh
connection from the "master" ain't it. It would be good to know what
(of any) would be acceptable solutions..
Well, I was hoping for something that has git knowledge, since there are
issues like updating objects in the right order.
So "git-send-pack" is nice in many ways: it allows you to update any
number of branches (in particular, it allows you to update just a _subset_
of the branches, which is nice if you have a shared central repository,
and some people have write permissions to some branches but not to
others), but it also allows for efficient unpacking on the receiver side
in a way no "general-purpose" mirror program can really match.
However, that requires the receiver to run a git-aware unpacker (in this
case git-receive-pack). I'm hoping that would be acceptable, I'm just
wondering what kind of safety concerns I'd need to make sure of in order
to make people comfortable running a special receiver program.
So the current approach is very flexible: if the pusher has ssh access, he
can do it. Safe, secure, and no new security issues. And since the only
programs the receiver has to be able to run is two git programs
(git-receive-pack will run git-unpack-objects), maybe it would be ok to
even have "git-receive-pack" as the shell for the receiver side, so that
you don't actually give the mirrorer any shell access at all. But it's
still "push-based" in the sense that it's kernel.org that is doing the
pushing, and that may simply not be acceptable.
Linus
Anyway, please do give it a test. I think I'll use this to sync up to
kernel.org
In fact, the most recent push was gone with a
git-send-pack master.kernel.org:/pub/scm/linux/kernel/git/torvalds/git.git
so if the new commit ("Do ref matching on the sender side rather than on
receiver") shows up after the mirrors have caught up, then this thing is
officially in production use..
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:01
On Thu, 30 Jun 2005, Linus Torvalds wrote:
Anyway, what are the limitations? Here's a few obvious ones:
- I really hate how "ssh" apparently cannot be told to have alternate
paths. For example, on master.kernel.org, I don't control the setup, so
I can't install my own git binaries anywhere except in my ~/bin
directory, but I also cannot get ssh to accept that that is a valid
path. This one really bums me out, and I think it's an ssh deficiency.
You apparently have to compile in the paths at compile-time into sshd,
and PermitUserEnvironment is disabled by default (not that it even
seems to work for the PATH environment, but that may have been my
testing that didn't re-start sshd).
That just sucks.
The easiest thing might be to have a centrally-installed wrapper script
that could run programs installed in your home directory. E.g., if
"git" had a "source ~/.git-env" at the beginning, and your ~/.git-env
fixed your PATH, then "git receive-pack ARGS" should work, for a generic
centrally installed git and special stuff in your home directory.
- It doesn't update the working directory at the other end. This is fine
for what it's intended for (pushing to a central "raw" git archives),
so this could be considered a feature, but it's worth pointing out.
Only a "pull" will update your working directory, and this pack sending
really is meant to be used in a kind of "push to central archive" way.
I thought only "resolve" (as part of "fetch") updated your working
directory, so this is completely consistant.
- this is also (at least once we've tested it a lot more and added the
code to allow it to create new refs on the remote side) meant to be a
good way to mirror things out, since clearly rsync isn't scaling.
However, I don't know what the rules for acceptable mirroring
approaches are, and it's entirely possible (nay, probable) that an ssh
connection from the "master" ain't it. It would be good to know what
(of any) would be acceptable solutions..
The right solution probably involves getting each pack file you push to
the mirrors as well as to the master. They'll probably update no less
frequently than you push, and they should go through a series of states
which matches the master, so it's not necessary to have anything smart on
master sending them, and they only have to unpack the files they get (and
update the refs afterward). That should make the cross-system trust
requirements relatively minimal; the mirror can fetch things from master,
and neither side has to allow the other to specify a command line.
-Daniel
*This .sig left intentionally blank*
The right solution probably involves getting each pack file you push to
the mirrors as well as to the master. They'll probably update no less
frequently than you push, and they should go through a series of states
which matches the master, so it's not necessary to have anything smart on
master sending them, and they only have to unpack the files they get (and
update the refs afterward).
Hmm, yes. That would work, together with just fetching the heads.
It won't _really_ solve the problem, since the pushed pack objects will
grow at a proportional rate to the current objects - it's just a constant
factor (admittedly a potentially fairly _big_ constant factor)
improvement both in size and in number of files.
So the mirroring ends up getting slowly slower and slower as the number of
pack files go up. In contrast, a git-aware thing can be basically
constant-time, and mirroring expense ends up being relative to the size of
the change rather than the size of the repository.
But mirroring just pack-files might solve the problem for the forseeable
future, so..
"git-receive-pack" would need to take a flag to tell it to instead of
unpacking just check the object instead (ie call "git-unpack-object" with
the "-n" flag - it will check that everything looks ok, including the
embedded protecting SHA1 hash), and write it out to the filesystem (as it
comes in) and then rename it to the right place.
Linus
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2016-06-15 22:42:01
Linus Torvalds wrote:
It won't _really_ solve the problem, since the pushed pack objects will
grow at a proportional rate to the current objects - it's just a constant
factor (admittedly a potentially fairly _big_ constant factor)
improvement both in size and in number of files.
If I've understood this correctly, it's not a constant factor
improvement in the number of files (in the size, yes); it's changing it
from O(t*c) to O(t) where t is number of trees and c is number of
changesets. That's key.
The problem we're having (on kernel.org) right now is that there isn't a
hierarchial time stamp in Unix, so we have to compare on a file-by-file
level. rsync is quite good at discovering an invariant beginning of a
file, but when it comes to a mass of files it has to compare the stamps
on each and every one, each time. It will only descend into a single
file, however, if that file has had its timestamp changed.
For the purposes of rsync, storing the objects in a single append-only
file would be a very efficient method, since the rsync algorithm will
quickly discover an invariant head and only transmit the tail. It's not
ideal, and having something git-aware would be better, but I think it's
really would be nice to have something which also plays well with rsync.
There is a *lot* of infrastructure in rsync which is actually hard to
replicate with another tool (including the server architecture); in many
ways it would be easier to convince the rsync developers to create a
plugin architecture and re-use all that code rather than developing an
equivalent tool from scratch.
-hpa
If I've understood this correctly, it's not a constant factor
improvement in the number of files (in the size, yes); it's changing it
from O(t*c) to O(t) where t is number of trees and c is number of
changesets. That's key.
No, it _is_ a constant factor even in number of files, if you just keep
the pack objects around without re-packing them.
Basically, you'd get one new pack-file every time I push. That's better
than getting <n> "raw object" files (where <n> can be anything from just a
couple to several thousand, depending on whether I had pulled things), but
it's still just a constant factor on both number of files and size of
files.
Now, you could re-pack the objects every once in a while: it would force a
whole new "epoch", of course and then the mirrorers would have to fetch
the whole repacked file, but that might be fine. Especially if you stop
re-packing after you've hit a certain size (say, a couple of megs), and
then start on the next pack.
For the purposes of rsync, storing the objects in a single append-only
file would be a very efficient method, since the rsync algorithm will
quickly discover an invariant head and only transmit the tail.
Actually, it won't be "quick" - it will have to read the whole file and do
it's hash window thing.
You _could_ append the pack-files into one single "superpack" file (since
you can figure out where the pack boundaries are), but it would be
extremely big after a while, and rsync would spend all its time doing over
the hash window. You'd definitely be better off with re-packing.
Linus
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:42:01
On Thu, 30 Jun 2005, Linus Torvalds wrote:
On Thu, 30 Jun 2005, Daniel Barkalow wrote:
quoted
The right solution probably involves getting each pack file you push to
the mirrors as well as to the master. They'll probably update no less
frequently than you push, and they should go through a series of states
which matches the master, so it's not necessary to have anything smart on
master sending them, and they only have to unpack the files they get (and
update the refs afterward).
Hmm, yes. That would work, together with just fetching the heads.
It won't _really_ solve the problem, since the pushed pack objects will
grow at a proportional rate to the current objects - it's just a constant
factor (admittedly a potentially fairly _big_ constant factor)
improvement both in size and in number of files.
So the mirroring ends up getting slowly slower and slower as the number of
pack files go up. In contrast, a git-aware thing can be basically
constant-time, and mirroring expense ends up being relative to the size of
the change rather than the size of the repository.
But mirroring just pack-files might solve the problem for the forseeable
future, so..
Whenever it gets slow, you could replace all the old packs with a single
new pack containing all the old objects; and master could repack whenever
it has a lot of pack files. That's pretty close to O(n) in change size.
Alternatively, having a reverse-ordered list of pack files would mean that
mirrors could just go through that list until they found one they already
had, and stop there, which would really be O(n).
-Daniel
*This .sig left intentionally blank*
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2016-06-15 22:42:01
H. Peter Anvin wrote:
Linus Torvalds wrote:
quoted
quoted
For the purposes of rsync, storing the objects in a single
append-only file would be a very efficient method, since the rsync
algorithm will quickly discover an invariant head and only transmit
the tail.
Actually, it won't be "quick" - it will have to read the whole file
and do it's hash window thing.
It does that, but it only have to do that when the actual file has
changed. That's acceptable, at least for the repository sizes we're
likely to deal with within the medium term.
I guess I should clarify a bit here. I'm concerned with two aspects:
the "keeping mirrors in sync" problem, where asking people to use a tool
other than rsync is a really tough sell, and the developer usage
scenario, in which case something git-aware is obviously the better thing.
-hpa
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2016-06-15 22:42:01
Linus Torvalds wrote:
quoted
For the purposes of rsync, storing the objects in a single append-only
file would be a very efficient method, since the rsync algorithm will
quickly discover an invariant head and only transmit the tail.
Actually, it won't be "quick" - it will have to read the whole file and do
it's hash window thing.
It does that, but it only have to do that when the actual file has
changed. That's acceptable, at least for the repository sizes we're
likely to deal with within the medium term.
-hpa
It does that, but it only have to do that when the actual file has
changed. That's acceptable, at least for the repository sizes we're
likely to deal with within the medium term.
Well, realize that "incremental packs" deltify a lot worse than a "big
pack", since pack-files don't do deltas to objects outside the pack-file.
So we'd get _some_ compression, but not as much as possible. The current
kernel compresses down to a single 63 MB pack-file (that's with the 2.6.11
tree too, not just the HEAD history), but without deltas it weights in at
about 177 MB.
So a "sum of incremental packs" should be somewhere in between those two
values, even today. For a single kernel archive.
So repository sizes aren't exactly trivial. I don't know how expensive
that rsync hash thing is, but one thing you lose is the ability to
hardlink objects, so if you have a few kernel repositories at some point
it doesn't fit in the cache any more, and then the rsync will have to read
that much pack object stuff from disk in addition to doing the hash. Ugh.
Linus
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2016-06-15 22:42:01
Linus Torvalds wrote:
On Thu, 30 Jun 2005, H. Peter Anvin wrote:
quoted
It does that, but it only have to do that when the actual file has
changed. That's acceptable, at least for the repository sizes we're
likely to deal with within the medium term.
Well, realize that "incremental packs" deltify a lot worse than a "big
pack", since pack-files don't do deltas to objects outside the pack-file.
So we'd get _some_ compression, but not as much as possible. The current
kernel compresses down to a single 63 MB pack-file (that's with the 2.6.11
tree too, not just the HEAD history), but without deltas it weights in at
about 177 MB.
So a "sum of incremental packs" should be somewhere in between those two
values, even today. For a single kernel archive.
So repository sizes aren't exactly trivial. I don't know how expensive
that rsync hash thing is, but one thing you lose is the ability to
hardlink objects, so if you have a few kernel repositories at some point
it doesn't fit in the cache any more, and then the rsync will have to read
that much pack object stuff from disk in addition to doing the hash. Ugh.
The bulk of the cost in doing the hashing comes from having to read the
file.
Well, if you grow a single pack file with appending, then you can have
delta references to earlier objects within the same pack file.
At least at this point, we'd handle a few very large files a lot better
than an enormous swarm of smaller ones.
In the end, it might be that the right thing to do for git on kernel.org
is to have a single, unified object store which isn't accessible by
anything other than git-specific protocols. There would have to be some
way of dealing with, for example, conflicting tags that apply to
different repositories, though.
-hpa
maybe it would be ok to
even have "git-receive-pack" as the shell for the receiver side, so that
you don't actually give the mirrorer any shell access at all.
You can probably just set the remote command (in ~/.ssh/authorized_keys)
to git-receive-pack. That also works around any $PATH issues.
Once this is stable, master.kernel.org should be updated with the
latest git.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
People are never so ready to believe you as when you say things in dispraise
of yourself; and you are never so much annoyed as when they take you at your
word.
-- Somerset Maugham
In the end, it might be that the right thing to do for git on kernel.org
is to have a single, unified object store which isn't accessible by
anything other than git-specific protocols.
Makes sense.
There would have to be some
way of dealing with, for example, conflicting tags that apply to
different repositories, though.
It seems that user-specific subdirectories in refs/heads (and, presumably,
../tags) mostly work already.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
Don't lock the barn after it is stolen.
From: Jan Harkes <jaharkes@cs.cmu.edu> Date: 2016-06-15 22:42:01
On Fri, Jul 01, 2005 at 12:31:53PM +0200, Matthias Urlichs wrote:
quoted
In the end, it might be that the right thing to do for git on kernel.org
is to have a single, unified object store which isn't accessible by
anything other than git-specific protocols.
Makes sense.
quoted
There would have to be some
way of dealing with, for example, conflicting tags that apply to
different repositories, though.
It seems that user-specific subdirectories in refs/heads (and, presumably,
../tags) mostly work already.
They work pretty well, the core git commands have no problem with them
and I just sent off some patches for gitweb and gitk.
All git/objects directories can be merged into a common repository. The
refs/heads and refs/tags be copied to user specific subdirectories.
Then a pull like,
git pull http://www.kernel.org/.../torvalds/linux-2.6.git
Would become,
git pull http://www.kernel.org/.../linux-2.6.git torvalds/linux-2.6/master
It would make rsync more expensive for people who are interested in only
a branch or two, but there is only one repository which should be easier
on the mirrors. The http, ssh, and some future 'pack' transfer methods
won't see a difference since they only pull the specific commits they
need to catch up with a branch.
Jan