From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
Any ideas on why git protocol clone is failing?
2007-08-24_20:51:33.85649 [9758] Connection from 72.74.92.181:19367
2007-08-24_20:51:33.85828 [9758] Extended attributes (33 bytes) exist
<host=git.jonsmirl.is-a-geek.net>
2007-08-24_20:51:33.96990 [9758] Request upload-pack for
'/home/git/mpc5200b.git'
2007-08-24_20:51:45.00789 fatal: Out of memory? mmap failed: Cannot
allocate memory
2007-08-24_20:51:45.08746 error: git-upload-pack: git-rev-list died with error.
2007-08-24_20:51:45.08771 fatal: git-upload-pack: aborting due to
possible repository corruption on the remote side.
NSLU2 ($70) is 266Mhz ARM with 32MB memory.
It's running Debian on a 250GB disk with 180MB swap.
Watching top the process runs up to about 60MB in virtual size and exits.
Setting the window down made no difference packedGitWindowSize = 4194304
--
Jon Smirl
jonsmirl@gmail.com
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:30
Jon Smirl [off-list ref] wrote:
Any ideas on why git protocol clone is failing?
2007-08-24_20:51:33.85649 [9758] Connection from 72.74.92.181:19367
2007-08-24_20:51:33.85828 [9758] Extended attributes (33 bytes) exist
<host=git.jonsmirl.is-a-geek.net>
2007-08-24_20:51:33.96990 [9758] Request upload-pack for
'/home/git/mpc5200b.git'
2007-08-24_20:51:45.00789 fatal: Out of memory? mmap failed: Cannot
allocate memory
2007-08-24_20:51:45.08746 error: git-upload-pack: git-rev-list died with error.
2007-08-24_20:51:45.08771 fatal: git-upload-pack: aborting due to
possible repository corruption on the remote side.
NSLU2 ($70) is 266Mhz ARM with 32MB memory.
It's running Debian on a 250GB disk with 180MB swap.
Watching top the process runs up to about 60MB in virtual size and exits.
Setting the window down made no difference packedGitWindowSize = 4194304
ulimits? packedGitLimit may also need to be decreased? Though we
always try to free unused windows before we declare we are out
of memory...
--
Shawn.
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
I'm still trying to debug git-daemon
I do find it surprising that git-index-pack can't be happy with in
20MB of RAM and it has to continuously swap it's 30MB of virtual. My
disk is chattering itself to death. It stayed that way for 40 minutes.
I'm practicing on the kernel tree.
--
Jon Smirl
jonsmirl@gmail.com
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:30
On Fri, 24 Aug 2007, Jon Smirl wrote:
I'm still trying to debug git-daemon
I do find it surprising that git-index-pack can't be happy with in
20MB of RAM and it has to continuously swap it's 30MB of virtual. My
disk is chattering itself to death. It stayed that way for 40 minutes.
I'm practicing on the kernel tree.
You hope for miracles, do you? ;-)
Please stop hammering that poor little NSLU2 with such a workset, or
hack some additional 224MB of RAM into it. There is no magical
solution.
Nicolas
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
On 8/24/07, Nicolas Pitre [off-list ref] wrote:
On Fri, 24 Aug 2007, Jon Smirl wrote:
quoted
I'm still trying to debug git-daemon
I do find it surprising that git-index-pack can't be happy with in
20MB of RAM and it has to continuously swap it's 30MB of virtual. My
disk is chattering itself to death. It stayed that way for 40 minutes.
I'm practicing on the kernel tree.
You hope for miracles, do you? ;-)
We're going something wrong in git-daemon. I can clone the tree in
five minutes using the http protocol. Using the git protocol would
take 24hrs if I let it finish.
Please stop hammering that poor little NSLU2 with such a workset, or
hack some additional 224MB of RAM into it. There is no magical
solution.
Nicolas
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:30
On Fri, 24 Aug 2007, Jon Smirl wrote:
On 8/24/07, Nicolas Pitre [off-list ref] wrote:
quoted
On Fri, 24 Aug 2007, Jon Smirl wrote:
quoted
I'm still trying to debug git-daemon
I do find it surprising that git-index-pack can't be happy with in
20MB of RAM and it has to continuously swap it's 30MB of virtual. My
disk is chattering itself to death. It stayed that way for 40 minutes.
I'm practicing on the kernel tree.
You hope for miracles, do you? ;-)
We're going something wrong in git-daemon. I can clone the tree in
five minutes using the http protocol. Using the git protocol would
take 24hrs if I let it finish.
The http protocol is merely only a dumb file copy with no packing
optimization what so ever.
The native protocol performs a whole more to provide clients with only
the minimum data needed.
Try running "git repack -a" directly on the NSLU2. You should have the
same performance problems as with a clone.
Nicolas
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
On 8/24/07, Jon Smirl [off-list ref] wrote:
We're going something wrong in git-daemon. I can clone the tree in
five minutes using the http protocol. Using the git protocol would
take 24hrs if I let it finish.
Try running "git repack -a" directly on the NSLU2. You should have the
same performance problems as with a clone.
This is true, it would take over 24hrs to finish.
Is their a reason why initial clone hasn't been special cased? Why
can't initial clone just blast over the pack file already sitting on
the disk?
I also wonder if a little application of some sorting to in-memory
data structures could help with the random IO patterns. I'm getting
the same data out of a stupid HTTP server and it doesn't go all IO
bound on me so a solution has to be possible.
--
Jon Smirl
jonsmirl@gmail.com
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:30
Jon Smirl wrote:
On 8/24/07, Nicolas Pitre [off-list ref] wrote:
quoted
Try running "git repack -a" directly on the NSLU2. You should have the
same performance problems as with a clone.
This is true, it would take over 24hrs to finish.
Is their a reason why initial clone hasn't been special cased? Why
can't initial clone just blast over the pack file already sitting on
the disk?
There was idea to special case clone (just concatenate the packs, the
receiving side as someone told there can detect pack boundaries; do not
forget to pack loose objects, first), instead of using generic fetch --all
for clone, bnut no code. Code speaks louder than words (although if someone
would provide details of pack boundary detection...)
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
I can clone the tree in five minutes using the http protocol. Using the
git protocol would take 24hrs if I let it finish.
The http side doesn't actually do any global verification, the way
git-daemon does. So to it, everything is just temporary buffers, and you
don't need any memory at all, really.
git-daemon will create a packfile. That means that it has to generate the
*global* object reachability, and will then optimize the object packing
etc etc. That's a minimum of something like 48 bytes per object for just
the object chains, and the kernel has a *lot* of objects (over half a
million).
In addition to the object chains yourself, the native protocol will also
obviously have to actually *look* at and parse all the tree and commit
objects while it does all this, so while it doesn't necessarily keep all
of those in memory all the time, it will need to access them, and if you
don't have enough memory to cache them, that will add its own set of IO.
So I haven't checked exactly how much memory you really want to have to
serve big projects, but with some handwavy guesstimate, if you actually
want to do a good job I'd guess that you really want to have at least as
much memory as the size of largest project you are serving, and probably
add at least 10-20% on top of that.
So for the kernel, at a guess, you'd probably want to have at least 256MB
of RAM to do a half-way good job. 512MB is likely nicer and allows you to
actually cache the stuff over multiple accesses.
But I haven't actually tested. Maybe it might be bearable at 128M.
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
On 8/24/07, Jakub Narebski [off-list ref] wrote:
There was idea to special case clone (just concatenate the packs, the
receiving side as someone told there can detect pack boundaries; do not
forget to pack loose objects, first), instead of using generic fetch --all
for clone, bnut no code. Code speaks louder than words (although if someone
would provide details of pack boundary detection...)
A related concept, initial clone of a repository does the equivalent
of repack -a on the repo before transmitting it. Why aren't we saving
those results by switching the repo onto the new pack file? Then the
next clone that comes along won't have to do anything but send the
file.
But this logic can be flipped around, if the remote needs any object
from the pack file, just send them the whole pack file and let the
remote sort it out. Using this logic you can still minimize the IO
statistically.
When a remote does a fetch you have to pack all of the loose objects.
When the loose object pile reaches 20MB or so, the fetch can trigger a
repack of the oldest half into a pack that is kept by the tree and
replaces those older loose objects. For future fetches simply apply
the rule of sending the whole pack if any object is needed.
The repack of the 10MB of older objects can be kicked out to another
process and copied into the tree when it is finished. At that point
the loose objects can be deleted. The git db can tolerate a process
copying in a new packfile and deleting the old objects while other
processes may be using the database, right?
This model shouldn't statistically change the amount of data very
much. If you haven't synced your tree in a month a few too many
objects may get sent to you. However, it should dramatically reduce
the IO load on the server cause by git protocol initial clones.
--
Jon Smirl
jonsmirl@gmail.com
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:30
On Fri, 24 Aug 2007, Jon Smirl wrote:
On 8/24/07, Nicolas Pitre [off-list ref] wrote:
quoted
Try running "git repack -a" directly on the NSLU2. You should have the
same performance problems as with a clone.
This is true, it would take over 24hrs to finish.
Is their a reason why initial clone hasn't been special cased? Why
can't initial clone just blast over the pack file already sitting on
the disk?
What is the gain? You'll get back to the same performance problem
eventually with some fetch operation, unless you intend to serve clients
with the whole pack everytime just like the http protocol does.
Also you don't want people cloning from you getting stuff that sits in
your reflog. The native protocol makes sure that only the needed
objects are sent over and no more.
I also wonder if a little application of some sorting to in-memory
data structures could help with the random IO patterns. I'm getting
the same data out of a stupid HTTP server and it doesn't go all IO
bound on me so a solution has to be possible.
The http application is, indeed, stupid. It performs no reachability
analysis, no repacking, no nothing except copying the bits over.
And yes I did add some sorting optimizations in this round, so if you
try 2.5.3-* you should have them. But there is a limit to what can be
done.
Point is, if you want serious Git serving, and not only _dumb_ protocols
(http is one of them) then you need more RAM. The NSLU2 is cool, but
maybe not appropriate for serving the Linux kernel natively with Git.
Nicolas
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:30
On 8/24/07, Linus Torvalds [off-list ref] wrote:
quoted
I can clone the tree in five minutes using the http protocol. Using the
git protocol would take 24hrs if I let it finish.
The http side doesn't actually do any global verification, the way
git-daemon does. So to it, everything is just temporary buffers, and you
don't need any memory at all, really.
git-daemon will create a packfile. That means that it has to generate the
*global* object reachability, and will then optimize the object packing
etc etc. That's a minimum of something like 48 bytes per object for just
the object chains, and the kernel has a *lot* of objects (over half a
million).
A large, repeating work load is created in this process when you take
a 200MB pack, repack it to add a few loose objects and then don't save
the results. This model makes the NSLU2 unusable, but I also see it at
my shared hosting provider. Initial clones of a repo that take 3min
from kernel.org take 25min on a shared host since the RAM is not
dedicated.
There are three categories of fetches:
1) initial clone, fetch all
2) fetch recent
3) I haven't fetched in three months
99% of fetches fall in the first two categories.
A very simple solution is to sendfile() existing packs if they contain
any objects that the client wants and let the client deal with the
unwanted objects. Yes this does send extra traffic over the net, but
the only group significantly impacted is #2 which is the most
infrequent group.
Loose objects are handled as they are currently. To optimize this
scheme you need to let the loose objects build up at the server and
then periodically sweep only the older ones into a pack. Packing the
entire repo into a single pack would cause recent fetches to retrieve
the entire pack.
Initial clone can be optimized further by recognizing that the
receiving repository is empty and sending them everything; no need to
compute which objects are missing at the server. This method will
speed up initial clone since the existing pack can be immediately sent
instead of waiting on a pack file to be built. Build the loose object
pack in parallel with sending the existing packs.
I recognize that in the case of cloning a single branch or --reference
too many objects will also be transmitted but I believe the benefits
of reducing the server load outweigh the overhead of transmitting
extra objects in this case. You can always remove the extra objects on
the client side.
On 8/24/07, Jakub Narebski [off-list ref] wrote:
There was idea to special case clone (just concatenate the packs, the
receiving side as someone told there can detect pack boundaries; do not
forget to pack loose objects, first), instead of using generic fetch --all
for clone, bnut no code. Code speaks louder than words (although if someone
would provide details of pack boundary detection...)
Write the file name and length into the socket before sending the
pack. Use sendfile() or it's current incarnation to actually send the
pack. Insert these header lines between packs.
In addition to the object chains yourself, the native protocol will also
obviously have to actually *look* at and parse all the tree and commit
objects while it does all this, so while it doesn't necessarily keep all
of those in memory all the time, it will need to access them, and if you
don't have enough memory to cache them, that will add its own set of IO.
So I haven't checked exactly how much memory you really want to have to
serve big projects, but with some handwavy guesstimate, if you actually
want to do a good job I'd guess that you really want to have at least as
much memory as the size of largest project you are serving, and probably
add at least 10-20% on top of that.
So for the kernel, at a guess, you'd probably want to have at least 256MB
of RAM to do a half-way good job. 512MB is likely nicer and allows you to
actually cache the stuff over multiple accesses.
But I haven't actually tested. Maybe it might be bearable at 128M.
Linus
From: Jeff King <hidden> Date: 2016-06-15 22:43:31
On Sat, Aug 25, 2007 at 11:44:07AM -0400, Jon Smirl wrote:
A very simple solution is to sendfile() existing packs if they contain
any objects that the client wants and let the client deal with the
unwanted objects. Yes this does send extra traffic over the net, but
the only group significantly impacted is #2 which is the most
infrequent group.
Loose objects are handled as they are currently. To optimize this
scheme you need to let the loose objects build up at the server and
then periodically sweep only the older ones into a pack. Packing the
entire repo into a single pack would cause recent fetches to retrieve
the entire pack.
I was about to write "but then 'fetch recent' clients will have to get
the entire repo after the upstream does a 'git-repack -a -d'" but you
seem to have figured that out already.
I'm unclear: are you proposing new behavior for git-daemon in general,
or a special mode for resource-constrained servers? If general behavior,
are you suggesting that we never use 'git-repack -a' on repos which
might be cloned?
-Peff
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:31
On 8/26/07, Jeff King [off-list ref] wrote:
On Sat, Aug 25, 2007 at 11:44:07AM -0400, Jon Smirl wrote:
quoted
A very simple solution is to sendfile() existing packs if they contain
any objects that the client wants and let the client deal with the
unwanted objects. Yes this does send extra traffic over the net, but
the only group significantly impacted is #2 which is the most
infrequent group.
Loose objects are handled as they are currently. To optimize this
scheme you need to let the loose objects build up at the server and
then periodically sweep only the older ones into a pack. Packing the
entire repo into a single pack would cause recent fetches to retrieve
the entire pack.
I was about to write "but then 'fetch recent' clients will have to get
the entire repo after the upstream does a 'git-repack -a -d'" but you
seem to have figured that out already.
I'm unclear: are you proposing new behavior for git-daemon in general,
or a special mode for resource-constrained servers? If general behavior,
are you suggesting that we never use 'git-repack -a' on repos which
might be cloned?
This would be a new general behavior. There are cases where git-daemon
is very resource hungry, rearranging things a little can remove this
need for everyone.
There are several ways to address the repack -a problem. But the
simplest solution may be the best, send existing packs only on an
initial clone. In all other cases continue with the current algorithm.
We could work on methods for making the middle case better but it is
so infrequent it is probably not worth bothering with.
Changing git-daemon only for the initial clone case also means that
people don't need to change the way they manage packs.
Posters have been saying, why worry about initial clone since it isn't
done that often. I agree that it isn't done that often, but if it is
done all on my NSLU2 it will take about 40hrs to complete. We can
easily see the impact of changing the the initial clone algorithm, the
http clone takes 3min.
BTW, if the NSLU2 needs a repack -a I can do it on another machine and
copy it over. Or maybe someone will write a repack that is happy in
20MB. The NSLU2 is a great home server, it is usually fast enough.
Power consumption is a tiny 8W, fine to leave on 24/7, My NSLU2 is as
powerful as the average desktop machine in the early 90's, how quickly
we forget.
Changing git-daemon only for the initial clone case also means that
people don't need to change the way they manage packs.
I do agree that we might want to do some special-case handling for the
initial clone (because it *is* kind of special), but it's not necessarily
as easy as just re-using an existing pack.
At a minimum, we'd need to have something that knows how to make a single
pack out of several packs and some loose objects. That shouldn't be
*hard*, but it's certainly nontrivial, especially in the presense of the
same objects possibly being available more than once in different packs.
[ The "duplicate object" thing does actually happen: even if you use only
"git native" protocols, you can get duplicate objects because a file was
changed back to an earlier version. The incremental packs you get from
push/pull'ing between two repositories try to send the minimal
incremental changes, but the keyword here is _try_: they will
potentially send objects that the receiver already has, if it's not
obvious that the receiver has them from the "commit boundary" cases ]
Maybe the client side will handle a pack with duplicate objects perfectly
fine, and it's not an issue. Maybe. It might even be likely (I can't think
of anything that would obviously break). But at a minimum, it would be
something that needs some code on the sending side, and a lot of
verification that the end result works ok on the receiving side.
And there's actually a deeper problem: the current native protocol
guarantees that the objects sent over are only those that are reachable.
That matters. It matters for subtle security issues (maybe you are
exporting some repository that was rebased, and has objects that you
didn't *intend* to make public!), but it also matters for issues like git
"alternates" files.
If you only ever look at a single repo, you'll never see the alternates
issue, but if you're seriously looking at serving git repositories, I
don't really see the "single repo" case as being at all the most common or
interesting case.
And if you look at something like kernel.org, the "alternates" thing is
*much* more important than how much memory git-daemon uses! Yes,
kernel.org would probably be much happier if git-daemon wasn't such a
memory pig occasionally, but on the other hand, the win from using
alternates and being able to share 99% of all objects in all the various
related kernel repositories is actually likely to be a *bigger* memory win
than any git-daemon memory usage, because now the disk caching works a
hell of a lot better!
So it's not actually clear how the initial clone thing can be optimized on
the server side.
It's easier to optimize on the *client* side: just do the initial clone
with rsync/http (and "git gc" it on the client afterwards), and then
change it to the git native protocol after the clone.
That may not sound very user-friendly, but let's face it, I think there is
exactly one person in the whole universe that tries to use an NSLU2 as a
git server. So the "client-side workaround" is likely to affect a very
limited number of clients ;)
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:31
On 8/26/07, Linus Torvalds [off-list ref] wrote:
And there's actually a deeper problem: the current native protocol
guarantees that the objects sent over are only those that are reachable.
That matters. It matters for subtle security issues (maybe you are
exporting some repository that was rebased, and has objects that you
didn't *intend* to make public!), but it also matters for issues like git
"alternates" files.
Are these objects visible through the other protocols? It seems
dangerous to leave something on an open server that you want to keep
hidden.
If you only ever look at a single repo, you'll never see the alternates
issue, but if you're seriously looking at serving git repositories, I
don't really see the "single repo" case as being at all the most common or
interesting case.
And if you look at something like kernel.org, the "alternates" thing is
*much* more important than how much memory git-daemon uses! Yes,
kernel.org would probably be much happier if git-daemon wasn't such a
memory pig occasionally, but on the other hand, the win from using
alternates and being able to share 99% of all objects in all the various
related kernel repositories is actually likely to be a *bigger* memory win
than any git-daemon memory usage, because now the disk caching works a
hell of a lot better!
Doesn't kernel.org use alternates or something equivalent for serving
up all those nearly identical kernel trees?
I've been handling the problem locally by using remotes and fetching
all the repos I'm interested in into a single git db.
So it's not actually clear how the initial clone thing can be optimized on
the server side.
It's easier to optimize on the *client* side: just do the initial clone
with rsync/http (and "git gc" it on the client afterwards), and then
change it to the git native protocol after the clone.
Even better, get them to clone from kernel.org and then just fetch in
the differences from my server. It's an educational problem.
How about changing initial clone to refuse to use the git protocol?
That may not sound very user-friendly, but let's face it, I think there is
exactly one person in the whole universe that tries to use an NSLU2 as a
git server. So the "client-side workaround" is likely to affect a very
limited number of clients ;)
I'll send you one and double the size of the user base. I have this
fancy new 20Mb FIOS connection and I can't come up with anything to
use the bandwidth on.
Anyway, I already gave up and moved on to a hosting provider. Repo is
here: http://git.digispeaker.com/ There's nothing there yet but a
clone of the 2.6 tree.
I don't think there is a solution for running a git daemon on a shared host.
Petr pointed out to me that an NSLU2 is late 90's equivalent not early
so my memory if faulty too.
And there's actually a deeper problem: the current native protocol
guarantees that the objects sent over are only those that are reachable.
That matters. It matters for subtle security issues (maybe you are
exporting some repository that was rebased, and has objects that you
didn't *intend* to make public!), but it also matters for issues like git
"alternates" files.
Are these objects visible through the other protocols? It seems
dangerous to leave something on an open server that you want to keep
hidden.
They'd be visible to any stupid walker, yes. But if you're
security-conscious, you'd simply not *allow* any stupid walkers.
One of the goals of "git-daemon" was to have a simple service that was
"obviously secure". Now, it's debatable just how obvious the daemon is,
but it really is pretty simple, and I do think it should be possible to
almost statically validate that it only ever reads files, and that it will
only ever read files that act like valid *git* data.
Some people may care about that kind of thing. I don't know how many, but
it really was one of the design criteria (which is why, for example, git
daemon will just silently close the connection if it finds something
fishy: no fishing expeditions with bad clients trying to figure out what
files exist on a server allowed!).
So the fact that a web server or rsync will expose everything is kind of
irrelevant - those are *designed* to expose everything. git-daemon was
designed *not* to do that.
Doesn't kernel.org use alternates or something equivalent for serving
up all those nearly identical kernel trees?
Absolutely. And that's the point. "git-daemon" will serve a nice
individualized pack, even though any particular repository doesn't have
one, but is really a combination of "the base Linus pack + extensions".
quoted
So it's not actually clear how the initial clone thing can be optimized on
the server side.
It's easier to optimize on the *client* side: just do the initial clone
with rsync/http (and "git gc" it on the client afterwards), and then
change it to the git native protocol after the clone.
Even better, get them to clone from kernel.org and then just fetch in
the differences from my server. It's an educational problem.
Yes.
How about changing initial clone to refuse to use the git protocol?
Absolutely not. It's quite often the best one to use (the ssh protocol
has the exact same issues, and is the only secure protocol).
But on a SNLU2, maybe *you* want to make your server side refuse it? I
would be easy enough: if the client doesn't report any existing SHA1's,
you just say "I'm not going to work with you".
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:31
On 8/26/07, Linus Torvalds [off-list ref] wrote:
quoted
Doesn't kernel.org use alternates or something equivalent for serving
up all those nearly identical kernel trees?
Absolutely. And that's the point. "git-daemon" will serve a nice
individualized pack, even though any particular repository doesn't have
one, but is really a combination of "the base Linus pack + extensions".
A really simple change to the git protocol would be to make the client
loop on the request. On the first request the server would see that
the client has no objects and send the "base Linus pack". The client
would then loop around and repeat the process which will trigger the
current pack building process.
Do pack files contain enough information about the heads of the object
chains for this to work? The client needs to be able to determine it's
state after receiving the pack and send the info back in the next
round.
I'm not buying the security argument. If you want something kept
hidden get it out of the public db. If I know the sha of the hidden
object can't I just add a head for it and git-deamon will happily send
it and the chain up to it to me?
--
Jon Smirl
jonsmirl@gmail.com
A really simple change to the git protocol would be to make the client
loop on the request. On the first request the server would see that
the client has no objects and send the "base Linus pack". The client
would then loop around and repeat the process which will trigger the
current pack building process.
Jon, just give it up. The fact is, the git protocol works the right way
already.
I'm not buying the security argument. If you want something kept hidden
get it out of the public db. If I know the sha of the hidden object
can't I just add a head for it and git-deamon will happily send it and
the chain up to it to me?
That's a particularly idiotic statement.
If you know the SHA1, there can *by*definition* not be any hidden objects.
The SHA1 depends on the object chain.
Linus
From: Daniel Hulme <hidden> Date: 2016-06-15 22:43:31
On Sun, Aug 26, 2007 at 10:15:24AM -0700, Linus Torvalds wrote:
It's easier to optimize on the *client* side: just do the initial clone
with rsync/http (and "git gc" it on the client afterwards), and then
change it to the git native protocol after the clone.
When I was working on Xen two years ago, they did the same thing with
their Mercurial repository. They had a proper repo that handled all the
push and fetch traffic, and a cron job would periodically pull from that
into a second repo. This second one was served by http. People were
encouraged to download the seed repo and then do a fetch (from the main
one) immediately.
I don't know whether they still do that, but in any case it shows your
idea is not unprecedented.
--
Kanga said to Roo, "Drink up your milk first, dear, and talk after-
wards." So Roo, who was drinking his milk, tried to say that he could do
both at once... and had to be patted on the back and dried for quite a
long time afterwards. A. A. Milne, 'Winnie-the-Pooh'
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:31
On Sun, Aug 26, 2007, Jeff King wrote:
On Sat, Aug 25, 2007 at 11:44:07AM -0400, Jon Smirl wrote:
quoted
A very simple solution is to sendfile() existing packs if they contain
any objects that the client wants and let the client deal with the
unwanted objects. Yes this does send extra traffic over the net, but
the only group significantly impacted is #2 which is the most
infrequent group.
Loose objects are handled as they are currently. To optimize this
scheme you need to let the loose objects build up at the server and
then periodically sweep only the older ones into a pack. Packing the
entire repo into a single pack would cause recent fetches to retrieve
the entire pack.
I was about to write "but then 'fetch recent' clients will have to get
the entire repo after the upstream does a 'git-repack -a -d'" but you
seem to have figured that out already.
I'm unclear: are you proposing new behavior for git-daemon in general,
or a special mode for resource-constrained servers? If general behavior,
are you suggesting that we never use 'git-repack -a' on repos which
might be cloned?
I think that "reuse existing packs if sensible" idea (instead of generating
always new pack) is a good one, even if at first limited to the clone case.
There are nevertheless a few complications.
1. When discussing this idea on git mailing list some time ago somebody
said that we don't need to implement "multi pack" extension (which was
at the beginning in the design, to add later, if I understand correctly),
it is enough to concatenate packs. The receiving side can then detect
boundaries between packs and split them appropriately. But is a
concatenated a proper pack? If not, then we can send concatenation of
packs only if the client (receiving side) understands it, and can split it;
it means checking for protocol extension...
2. How to detect that request is for a clone? git-clone is get all remote
heads and fetch from just received heads. But because fecthing refs and
fetching objects is separate, we cannot I think use this sequence for
detecting that we want a clone. We can use "no haves" as heuristic to
detect a clone request, but "no haves" occurs also for initial fetching of
single branch (i.e. using: git-remote; git-fetch sequence instead of
git-clone).
3. The problem with alternates mentioned by Linus is not much a problem,
as we can simply consider packs from the alternate repository/repositories.
For example if we use single alternate, we would send concatenation of
packs from this repository, and from alternate (and pack of loose objects
from this repository).
We would probably want to have some heuristic (besides configuring
git-daemon) to choose between reusing existing packs (and sending them
concatenated), and generating a pack for sending. Note that for dumb
transports we have the opposite problem and opposite idea: we always
send full packs for dumb transports; the idea was to use range downloading
(available at least for http and ftp protocols) to download only needed
fragments of packs. Perhaps if some % of pack (number of objects in the
pack or size of pack) is to be send then we reuse the pack, and remove
objects in the pack from consideration. No idea of how to implement that,
though. Or if number of objects in pack to be send crosses some threshold,
or generating pack/doing reachability analysis takes to loong, then reuse
existing packs.
Or you can wait fro the GitTorrent protocol to be implemented, or implement
it yourself... ;-)
--
Jakub Narebski
Poland
On Sun, Aug 26, 2007 at 11:26:07AM -0700, Linus Torvalds wrote:
quoted
How about changing initial clone to refuse to use the git protocol?
Absolutely not. It's quite often the best one to use (the ssh protocol
has the exact same issues, and is the only secure protocol).
But on a SNLU2, maybe *you* want to make your server side refuse it? I
would be easy enough: if the client doesn't report any existing SHA1's,
you just say "I'm not going to work with you".
What if the server sends a message which current clients interprets as
an error, and which newer clients could interpret as, "do a clone from
<this> URL, and then come back and talk to me". Basically an
automated redirect to get the "Linus base pack" somewhere else, and
then to go back to the original server. It certainly doesn't make
sense to change anything about the low-level protocol, but maybe a
higher level redirect would make sense, just as a user convenience thing.
- Ted
What if the server sends a message which current clients interprets as
an error, and which newer clients could interpret as, "do a clone from
<this> URL, and then come back and talk to me". Basically an
automated redirect to get the "Linus base pack" somewhere else, and
then to go back to the original server. It certainly doesn't make
sense to change anything about the low-level protocol, but maybe a
higher level redirect would make sense, just as a user convenience thing.
I agree, a redirect might be a good idea regardless of whether it's
something like "I'm a poor little NSLU2, please don't do anything but
incremental updates", or whether it's something like "this repository has
moved, use address xyz instead".
And it should be pretty easy from a high-level protocol, although it does
obviously need both server and client support.
Linus