From: Jan Hudec <hidden> Date: 2016-06-15 22:43:10
Hello,
Did anyone already think about fetching over HTTP working similarly to the
native git protocol?
That is rather than reading the raw content of the repository, there would be
a CGI script (could be integrated to gitweb), that would negotiate what the
client needs and then generate and send a single pack with it.
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and
https (you can tunel anything through, but only to port 443) through.
- Can be run on shared machine. If you have web space on machine shared
by many people, you can set up your own gitweb, but cannot/are not allowed
to start your own network server for git native protocol.
- Less things to set up. If you are setting up gitweb anyway, you'd not need
to set up additional thing for providing fetch access.
Than a question is how to implement it. The current protocol is stateful on
both sides, but the stateless nature of HTTP more or less requires the
protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but
make it stateless for server. That is server first sends it's heads and than
client repeatedly sends all it's wants and some haves until the server acks
all of them and sends the pack.
Alternatively I am thinking about using Bloom filters (somebody came with
such idea on the bzr list when I still followed it). It might be useful, as
over HTTP we need to send as many haves as possible in one go.
--
Jan 'Bulb' Hudec [off-list ref]
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:43:10
Jan Hudec wrote:
Hello,
Did anyone already think about fetching over HTTP working similarly to the
native git protocol?
That is rather than reading the raw content of the repository, there would be
a CGI script (could be integrated to gitweb), that would negotiate what the
client needs and then generate and send a single pack with it.
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and
https (you can tunel anything through, but only to port 443) through.
- Can be run on shared machine. If you have web space on machine shared
by many people, you can set up your own gitweb, but cannot/are not allowed
to start your own network server for git native protocol.
- Less things to set up. If you are setting up gitweb anyway, you'd not need
to set up additional thing for providing fetch access.
Than a question is how to implement it. The current protocol is stateful on
both sides, but the stateless nature of HTTP more or less requires the
protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but
make it stateless for server. That is server first sends it's heads and than
client repeatedly sends all it's wants and some haves until the server acks
all of them and sends the pack.
Alternatively I am thinking about using Bloom filters (somebody came with
such idea on the bzr list when I still followed it). It might be useful, as
over HTTP we need to send as many haves as possible in one go.
Bundles?
Client POSTs it's ref set; server uses the ref set to generate and
return the bundle.
Push over http(s) could work the same...
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:10
Jan Hudec [off-list ref] wrote:
Did anyone already think about fetching over HTTP working similarly to the
native git protocol?
No work has been done on this (that I know of) but I've discussed
it to some extent with Simon 'corecode' Schubert on #git, and I
think he also brought it up on the mailing list not too long after.
I've certainly thought about adding some sort of pack-objects
frontend into gitweb.cgi for this exact purpose. It is really
quite easy, except for the negotation of what the client has. ;-)
Than a question is how to implement it. The current protocol is stateful on
both sides, but the stateless nature of HTTP more or less requires the
protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but
make it stateless for server. That is server first sends it's heads and than
client repeatedly sends all it's wants and some haves until the server acks
all of them and sends the pack.
I think Simon was talking about doubling the number of haves the
client sends in each request. So the client POSTs initially all
of its current refs; then current refs and their parents; then 4
commits back, then 8, etc. The server replies to each POST request
with either a "send more please" or the packfile.
--
Shawn.
From: Martin Langhoff <hidden> Date: 2016-06-15 22:43:10
On 5/16/07, Johannes Schindelin [off-list ref] wrote:
On Wed, 16 May 2007, Martin Langhoff wrote:
quoted
Do the indexes have enough info to use them with http ranges? It'd be
chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers
speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it?
Sure! It'd probably take a few trips, and you'd end up fetching 20% of
the file, still better than 100%.
Besides, not all Web servers speak HTTP/1.1...
Are there any interesting webservers out there that don't? Hand-rolled
purpose-built webservers often don't but those don't serve files, they
serve web apps. When it comes to serving files, any webserver that is
supported (security-wise) these days is HTTP/1.1.
And for services like SF.net it'd be a safe low-cpu way of serving git
files. 'cause the git protocol is quite expensive server-side (io+cpu)
as we've seen with kernel.org. Being really smart with a cgi is
probably going to be expensive too.
cheers,
m
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:10
Martin Langhoff wrote:
On 5/16/07, Johannes Schindelin [off-list ref] wrote:
quoted
On Wed, 16 May 2007, Martin Langhoff wrote:
quoted
Do the indexes have enough info to use them with http ranges? It'd be
chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers
speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it?
Sure! It'd probably take a few trips, and you'd end up fetching 20% of
the file, still better than 100%.
That's why you should have something akin to backup policy for pack files,
like daily packs, weekly packs, ..., and the rest, just for the dumb
protocols.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:10
Hi,
On Thu, 17 May 2007, Martin Langhoff wrote:
On 5/16/07, Johannes Schindelin [off-list ref] wrote:
quoted
On Wed, 16 May 2007, Martin Langhoff wrote:
quoted
Do the indexes have enough info to use them with http ranges? It'd be
chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers
speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it?
Sure! It'd probably take a few trips, and you'd end up fetching 20% of
the file, still better than 100%.
Don't forget that those 10% probably do not do you the favour to be in
large chunks. Chances are that _every_ _single_ wanted object is separate
from the others.
quoted
Besides, not all Web servers speak HTTP/1.1...
Are there any interesting webservers out there that don't? Hand-rolled
purpose-built webservers often don't but those don't serve files, they
serve web apps. When it comes to serving files, any webserver that is
supported (security-wise) these days is HTTP/1.1.
And for services like SF.net it'd be a safe low-cpu way of serving git
files. 'cause the git protocol is quite expensive server-side (io+cpu)
as we've seen with kernel.org. Being really smart with a cgi is
probably going to be expensive too.
It's probably better and faster than relying on a feature which does not
exactly help.
Ciao,
Dscho
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:10
Johannes Schindelin [off-list ref] wrote:
Don't forget that those 10% probably do not do you the favour to be in
large chunks. Chances are that _every_ _single_ wanted object is separate
from the others.
That's completely possible. Assuming the objects even are packed
in the first place. Its very unlikely that you would be able to
fetch very large of a range from an existing packfile, you would be
submitting most of your range requests for very very small sections.
quoted
And for services like SF.net it'd be a safe low-cpu way of serving git
files. 'cause the git protocol is quite expensive server-side (io+cpu)
as we've seen with kernel.org. Being really smart with a cgi is
probably going to be expensive too.
It's probably better and faster than relying on a feature which does not
exactly help.
Yes. Packing more often and pack v4 may help a lot there.
The other thing is kernel.org should really try to encourage the
folks with repositories there to try and share against one master
repository, so the poor OS has a better chance at holding the bulk
of linux-2.6.git in buffer cache.
I'm not suggesting they share specifically against Linus' repository;
maybe hpa and the other admins can host one seperately from Linus and
enourage users to use that repository when on a system they maintain.
In an SF.net type case this doesn't help however. Most of SF.net
is tiny projects with very few, if any, developers. Hence most
of that is going to be unsharable, infrequently accessed, and uh,
not needed to be stored in buffer cache. For the few projects that
are hosted there that have a large developer base they could use
a shared repository approach as I just suggested for kernel.org.
aka the "forks" thing in gitweb, and on repo.or.cz...
--
Shawn.
And for services like SF.net it'd be a safe low-cpu way of serving git
files. 'cause the git protocol is quite expensive server-side (io+cpu)
as we've seen with kernel.org. Being really smart with a cgi is
probably going to be expensive too.
It's probably better and faster than relying on a feature which does not
exactly help.
Yes. Packing more often and pack v4 may help a lot there.
The other thing is kernel.org should really try to encourage the
folks with repositories there to try and share against one master
repository, so the poor OS has a better chance at holding the bulk
of linux-2.6.git in buffer cache.
do you mean more precisely share against one object store or do you really
mean repository?
David Lang
I'm not suggesting they share specifically against Linus' repository;
maybe hpa and the other admins can host one seperately from Linus and
enourage users to use that repository when on a system they maintain.
In an SF.net type case this doesn't help however. Most of SF.net
is tiny projects with very few, if any, developers. Hence most
of that is going to be unsharable, infrequently accessed, and uh,
not needed to be stored in buffer cache. For the few projects that
are hosted there that have a large developer base they could use
a shared repository approach as I just suggested for kernel.org.
aka the "forks" thing in gitweb, and on repo.or.cz...
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:10
david@lang.hm wrote:
On Wed, 16 May 2007, Shawn O. Pearce wrote:
quoted
The other thing is kernel.org should really try to encourage the
folks with repositories there to try and share against one master
repository, so the poor OS has a better chance at holding the bulk
of linux-2.6.git in buffer cache.
do you mean more precisely share against one object store or do you really
mean repository?
Sorry, I did mean "object store". ;-)
Repository is insanity, as the refs and tags namespaces are suddenly
shared. What a nightmare that would become.
--
Shawn.
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:10
"Shawn O. Pearce" [off-list ref] wrote:
david@lang.hm wrote:
quoted
On Wed, 16 May 2007, Shawn O. Pearce wrote:
quoted
The other thing is kernel.org should really try to encourage the
folks with repositories there to try and share against one master
repository, so the poor OS has a better chance at holding the bulk
of linux-2.6.git in buffer cache.
do you mean more precisely share against one object store or do you really
mean repository?
Sorry, I did mean "object store". ;-)
And even there, I don't mean symlink objects to a shared database,
I mean use the objects/info/alternates file to point to the shared,
read-only location.
Its not perfect. The hotter parts of the object database is almost
always the recent stuff, as that's what people are actively trying
to fetch, or are using as a base when they are trying to fetch from
someone else. The hotter parts are also probably too new to be
in the shared store offered by kernel.org admins, which means you
cannot get good IO buffering. Back to the current set of problems.
A single shared object directory that everyone can write new files
into, but cannot modify or delete from, would help that problem quite
a bit. But it opens up huge problems about pruning, as there is no
way to perform garbage collection on that database without scanning
every ref on the system, and that's just not simply possible on a
busy system like kernel.org.
--
Shawn.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:10
On Wed, 16 May 2007, Shawn O. Pearce wrote:
Johannes Schindelin [off-list ref] wrote:
quoted
Don't forget that those 10% probably do not do you the favour to be in
large chunks. Chances are that _every_ _single_ wanted object is separate
from the others.
That's completely possible. Assuming the objects even are packed
in the first place. Its very unlikely that you would be able to
fetch very large of a range from an existing packfile, you would be
submitting most of your range requests for very very small sections.
Well, in the commit objects case you're likely to have a bunch of them
all contigous.
For tree and blob objects it is less likely.
And of course there is the question of deltas for which you might or
might not have the base object locally already.
Still... I wonder if this could be actually workable. A typical daily
update on the Linux kernel repository might consist of a couple hundreds
or a few tousands objects. This could still be faster to fetch parts of
a pack than the whole pack if the size difference is above a certain
treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object,
parsing it to find out what tree object to fetch, then parse that tree
object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files,
finding out about the objects that the remote has that are not available
locally, and then fetching all those objects from within pack files
without even looking at them (except for deltas), then it should be
possible to issue a couple requests in parallel and possibly have decent
performances. And if it turns out that more than, say, 70% of a
particular pack is to be fetched (you can determine that up front), then
it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving
end of course, but storing them as loose objects and repacking them
afterwards should be just fine.
Of course you'll get objects from branches in the remote repository you
might not be interested in, but that's a price to pay for such a hack.
On average the overhead shouldn't be that big anyway if branches within
a repository are somewhat related.
I think this is something worth experimenting.
Nicolas
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:10
Hi,
On Wed, 16 May 2007, Nicolas Pitre wrote:
Still... I wonder if this could be actually workable. A typical daily
update on the Linux kernel repository might consist of a couple hundreds
or a few tousands objects. This could still be faster to fetch parts of
a pack than the whole pack if the size difference is above a certain
treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object,
parsing it to find out what tree object to fetch, then parse that tree
object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files,
finding out about the objects that the remote has that are not available
locally, and then fetching all those objects from within pack files
without even looking at them (except for deltas), then it should be
possible to issue a couple requests in parallel and possibly have decent
performances. And if it turns out that more than, say, 70% of a
particular pack is to be fetched (you can determine that up front), then
it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving
end of course, but storing them as loose objects and repacking them
afterwards should be just fine.
Of course you'll get objects from branches in the remote repository you
might not be interested in, but that's a price to pay for such a hack.
On average the overhead shouldn't be that big anyway if branches within
a repository are somewhat related.
I think this is something worth experimenting.
I am a bit wary about that, because it is so complex. IMHO a cgi which
gets, say, up to a hundred refs (maybe something like ref~0, ref~1, ref~2,
ref~4, ref~8, ref~16, ... for the refs), and then makes a bundle for that
case on the fly, is easier to do.
Of course, as with all cgi scripts, you have to make sure that DOS attacks
have a low probability of success.
Ciao,
Dscho
On Wed, May 16, 2007 at 09:45:42PM -0400, Shawn O. Pearce wrote:
Its not perfect. The hotter parts of the object database is almost
always the recent stuff, as that's what people are actively trying
to fetch, or are using as a base when they are trying to fetch from
someone else. The hotter parts are also probably too new to be
in the shared store offered by kernel.org admins, which means you
cannot get good IO buffering. Back to the current set of problems.
Actually, as long as objects/info/alternates is pointing at Linus's
kernel.org tree, I would think that it should work relatively well,
since everyone is normally basing their work on top of his tree as a
starting point.
- Ted
From: Petr Baudis <hidden> Date: 2016-06-15 22:43:10
Hi,
On Tue, May 15, 2007 at 10:10:06PM CEST, Jan Hudec wrote:
Did anyone already think about fetching over HTTP working similarly to the
That is rather than reading the raw content of the repository, there would be
a CGI script (could be integrated to gitweb), that would negotiate what the
client needs and then generate and send a single pack with it.
frankly, I'm not that excited. I'm not disputing that this would be
useful, but I have my doubts on just how *much* useful it would be - I'm
not so sure the set of users affected is really all that large. So I'm
just cooling people down here. ;-))
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and
https (you can tunel anything through, but only to port 443) through.
How many users really have this problem? I'm not so sure. There are
certainly some, but enough for this to be a viable argument?
- Can be run on shared machine. If you have web space on machine shared
by many people, you can set up your own gitweb, but cannot/are not allowed
to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so
it's not "zero-setup" solution anymore.
Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
- Less things to set up. If you are setting up gitweb anyway, you'd not need
to set up additional thing for providing fetch access.
Except, well, how do you "set it up"? You need to make sure
git-update-server-info is run, yes, but that shouldn't be a problem (I'm
not so sure if git does this for you automagically - Cogito would...).
I think 95% of people don't set up gitweb.cgi either for their small
HTTP repositories. :-)
Then again, it's not that it would be really technically complicated -
adding "give me a bundle" support to gitweb should be pretty easy.
However, this support has some "social" costs as well: no compatibility
with older git versions, support cost, confusion between dumb HTTP and
gitweb HTTP transports, more lack of motivation for improving dumb HTTP
transport...
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:10
On Thu, 17 May 2007, Johannes Schindelin wrote:
Hi,
On Wed, 16 May 2007, Nicolas Pitre wrote:
quoted
Still... I wonder if this could be actually workable. A typical daily
update on the Linux kernel repository might consist of a couple hundreds
or a few tousands objects. This could still be faster to fetch parts of
a pack than the whole pack if the size difference is above a certain
treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object,
parsing it to find out what tree object to fetch, then parse that tree
object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files,
finding out about the objects that the remote has that are not available
locally, and then fetching all those objects from within pack files
without even looking at them (except for deltas), then it should be
possible to issue a couple requests in parallel and possibly have decent
performances. And if it turns out that more than, say, 70% of a
particular pack is to be fetched (you can determine that up front), then
it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving
end of course, but storing them as loose objects and repacking them
afterwards should be just fine.
Of course you'll get objects from branches in the remote repository you
might not be interested in, but that's a price to pay for such a hack.
On average the overhead shouldn't be that big anyway if branches within
a repository are somewhat related.
I think this is something worth experimenting.
I am a bit wary about that, because it is so complex. IMHO a cgi which
gets, say, up to a hundred refs (maybe something like ref~0, ref~1, ref~2,
ref~4, ref~8, ref~16, ... for the refs), and then makes a bundle for that
case on the fly, is easier to do.
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place? After
all, they both boil down to running git-pack-objects and sending out the
result. I don't think such a solution really buys much.
On the other hand, if the client does all the work and provides the
server with a list of ranges within a pack it wants to be sent, then you
simply have zero special setup to perform on the hosting server and you
keep the server load down due to not running pack-objects there. That,
at least, is different enough from the Git daemon to be worth
considering. Not only does it provide an advantage to those who cannot
do anything but http out of their segregated network, but it also
provide many advantages on the server side too while the cgi approach
doesn't.
And actually finding out the list of objects the remote has that you
don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
2) From those .idx files, keep only a list of objects that are unknown
locally. A good starting point for doing this really efficiently is
the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each
object's size in the remote pack. The code to do this already exists
in builtin-pack-objects.c.
4) With the list of missing objects from (2) along with their offset and
size within a given pack file, fetch those objects from the remote
server. Either perform multiple requests in parallel, or as someone
mentioned already, provide the server with a list of ranges you want
to be sent.
5) Store the received objects as loose objects locally. If a given
object is a delta, verify if its base is available locally, or if it
is listed amongst those objects to be fetched from the server. If
not, add it to the list. In most cases, delta base objects will be
objects already listed to be fetched anyway. To greatly simplify
things, the loose delta object type from 2 years ago could be revived
(commit 91d7b8afc2) since a repack will get rid of them.
6 Repeat (4) and (5) until everything has been fetched.
7) Run git-pack-objects with the list of fetched objects.
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And
with this the server side remains really simple with no special setup
nor extra load beyond the simple serving of file content.
Nicolas
From: Martin Langhoff <hidden> Date: 2016-06-15 22:43:10
On 5/18/07, Nicolas Pitre [off-list ref] wrote:
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place?
And you probably _are_ running git daemon. But some clients may be on
shitty connections that only allow http. That's one of the scenarios
we're discussing.
cheers,
m
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:10
On Fri, 18 May 2007, Martin Langhoff wrote:
On 5/18/07, Nicolas Pitre [off-list ref] wrote:
quoted
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place?
And you probably _are_ running git daemon. But some clients may be on
shitty connections that only allow http. That's one of the scenarios
we're discussing.
That's not what I'm disputing at all.
I'm disputing the vertue of an HTTP solution involving a cgi with Git
bundles vs an HTTP solution involving static file range serving. The
clients on shitty connections don't care either ways.
Nicolas
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:10
On Thu, May 17, 2007 at 10:41:37 -0400, Nicolas Pitre wrote:
On Thu, 17 May 2007, Johannes Schindelin wrote:
quoted
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place? After
all, they both boil down to running git-pack-objects and sending out the
result. I don't think such a solution really buys much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate
server, at university while I studied and now I can get the same on friend's
server. Neither of them would probably be ok for serving larger busy git
repository, but something smaller accessed by several people is OK. I think
this is quite common for university students.
Of course your suggestion which moves the logic to client-side is a good one,
but even the cgi with logic on server side would help in some situations.
On the other hand, if the client does all the work and provides the
server with a list of ranges within a pack it wants to be sent, then you
simply have zero special setup to perform on the hosting server and you
keep the server load down due to not running pack-objects there. That,
at least, is different enough from the Git daemon to be worth
considering. Not only does it provide an advantage to those who cannot
do anything but http out of their segregated network, but it also
provide many advantages on the server side too while the cgi approach
doesn't.
And actually finding out the list of objects the remote has that you
don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
... for git it's 1.2 MiB. And that definitely isn't a huge source tree.
Of course the local side could remember which indices it already saw during
previous fetch from that location and not re-fetch them.
A slight problem is, that git-repack normally recombines everything to
a single pack, so the index would have to be re-fetched again anyway.
2) From those .idx files, keep only a list of objects that are unknown
locally. A good starting point for doing this really efficiently is
the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each
object's size in the remote pack. The code to do this already exists
in builtin-pack-objects.c.
4) With the list of missing objects from (2) along with their offset and
size within a given pack file, fetch those objects from the remote
server. Either perform multiple requests in parallel, or as someone
mentioned already, provide the server with a list of ranges you want
to be sent.
Does the git server really have to do so much beyond that? I didn't look at
the algorithm that finds what deltas should be based on, but depending on
that it might (or might not) be possible to proof the client has everything to
understand if the server sends the objects as it currently has them.
5) Store the received objects as loose objects locally. If a given
object is a delta, verify if its base is available locally, or if it
is listed amongst those objects to be fetched from the server. If
not, add it to the list. In most cases, delta base objects will be
objects already listed to be fetched anyway. To greatly simplify
things, the loose delta object type from 2 years ago could be revived
(commit 91d7b8afc2) since a repack will get rid of them.
6 Repeat (4) and (5) until everything has been fetched.
Unless I am really seriously missing something, there is no point in
repeating. For each pack you need to unpack a delta either:
- you have it => ok.
- you don't have it, but the server does =>
but than it's already in the fetch set calculated in 2.
- you don't have it and nor does server =>
the repository at server is corrupted and you can't fix it.
7) Run git-pack-objects with the list of fetched objects.
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And
with this the server side remains really simple with no special setup
nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the git
server approach, because at least the indices have to be transfered in
entirety. So each approach has it's own advantages.
--
Jan 'Bulb' Hudec [off-list ref]
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:10
On Thu, May 17, 2007 at 14:40:06 +0200, Petr Baudis wrote:
On Tue, May 15, 2007 at 10:10:06PM CEST, Jan Hudec wrote:
quoted
- Can be run on shared machine. If you have web space on machine shared
by many people, you can set up your own gitweb, but cannot/are not allowed
to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so
it's not "zero-setup" solution anymore.
Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
A particular case would be a group of students wanting to publish their
software project (I mean the PRG023 or equivalent). Private computers in the
hostel are not allowed to serve anything, so they'd use some of the lab
servers (eg. artax, ss1000...). All of them allow full CGI, but running
daemons is forbiden.
quoted
- Less things to set up. If you are setting up gitweb anyway, you'd not need
to set up additional thing for providing fetch access.
Except, well, how do you "set it up"? You need to make sure
git-update-server-info is run, yes, but that shouldn't be a problem (I'm
not so sure if git does this for you automagically - Cogito would...).
No. If it worked similar to git-upload-pack, only over http, it would work
without update-server-info, no?
I think 95% of people don't set up gitweb.cgi either for their small
HTTP repositories. :-)
Then again, it's not that it would be really technically complicated -
adding "give me a bundle" support to gitweb should be pretty easy.
However, this support has some "social" costs as well: no compatibility
with older git versions, support cost, confusion between dumb HTTP and
gitweb HTTP transports, more lack of motivation for improving dumb HTTP
transport...
The dumb transport is definitely useful. Extending it to use ranges if
possible would be useful as well (and maybe more than upload-pack-over-http).
--
Jan 'Bulb' Hudec [off-list ref]
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:10
On Thu, 17 May 2007, Jan Hudec wrote:
On Thu, May 17, 2007 at 10:41:37 -0400, Nicolas Pitre wrote:
quoted
On Thu, 17 May 2007, Johannes Schindelin wrote:
quoted
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place? After
all, they both boil down to running git-pack-objects and sending out the
result. I don't think such a solution really buys much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate
server, at university while I studied and now I can get the same on friend's
server. Neither of them would probably be ok for serving larger busy git
repository, but something smaller accessed by several people is OK. I think
this is quite common for university students.
Of course your suggestion which moves the logic to client-side is a good one,
but even the cgi with logic on server side would help in some situations.
You could simply wrap git-bundle within a cgi. That is certainly easy
enough.
quoted
On the other hand, if the client does all the work and provides the
server with a list of ranges within a pack it wants to be sent, then you
simply have zero special setup to perform on the hosting server and you
keep the server load down due to not running pack-objects there. That,
at least, is different enough from the Git daemon to be worth
considering. Not only does it provide an advantage to those who cannot
do anything but http out of their segregated network, but it also
provide many advantages on the server side too while the cgi approach
doesn't.
And actually finding out the list of objects the remote has that you
don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
... for git it's 1.2 MiB. And that definitely isn't a huge source tree.
Of course the local side could remember which indices it already saw during
previous fetch from that location and not re-fetch them.
Right. The name of the pack/index plus its time stamp can be cached.
If the remote doesn't repack too often then the overhead would be
minimal.
quoted
2) From those .idx files, keep only a list of objects that are unknown
locally. A good starting point for doing this really efficiently is
the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each
object's size in the remote pack. The code to do this already exists
in builtin-pack-objects.c.
4) With the list of missing objects from (2) along with their offset and
size within a given pack file, fetch those objects from the remote
server. Either perform multiple requests in parallel, or as someone
mentioned already, provide the server with a list of ranges you want
to be sent.
Does the git server really have to do so much beyond that?
Yes it does. The real thing perform a full object reachability walk and
only the objects that are needed for the wanted branch(es) are sent in a
custom pack meaning that the data transfer is really optimal.
quoted
5) Store the received objects as loose objects locally. If a given
object is a delta, verify if its base is available locally, or if it
is listed amongst those objects to be fetched from the server. If
not, add it to the list. In most cases, delta base objects will be
objects already listed to be fetched anyway. To greatly simplify
things, the loose delta object type from 2 years ago could be revived
(commit 91d7b8afc2) since a repack will get rid of them.
6 Repeat (4) and (5) until everything has been fetched.
Unless I am really seriously missing something, there is no point in
repeating. For each pack you need to unpack a delta either:
- you have it => ok.
- you don't have it, but the server does =>
but than it's already in the fetch set calculated in 2.
- you don't have it and nor does server =>
the repository at server is corrupted and you can't fix it.
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:43:10
On Thu, 17 May 2007, Jan Hudec wrote:
A particular case would be a group of students wanting to publish their
software project (I mean the PRG023 or equivalent). Private computers in the
hostel are not allowed to serve anything, so they'd use some of the lab
servers (eg. artax, ss1000...). All of them allow full CGI, but running
daemons is forbiden.
And wouldn't the admin authority for those lab servers be amenable to
install a Git daemon service? That'd be a much better solution to me.
Nicolas
On Thu, May 17, 2007 at 10:41:37 -0400, Nicolas Pitre wrote:
quoted
On Thu, 17 May 2007, Johannes Schindelin wrote:
quoted
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place? After
all, they both boil down to running git-pack-objects and sending out the
result. I don't think such a solution really buys much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate
server, at university while I studied and now I can get the same on friend's
server. Neither of them would probably be ok for serving larger busy git
repository, but something smaller accessed by several people is OK. I think
this is quite common for university students.
Of course your suggestion which moves the logic to client-side is a good one,
but even the cgi with logic on server side would help in some situations.
You could simply wrap git-bundle within a cgi. That is certainly easy
enough.
isn't this (or something very similar) exactly what we want for a smalrt
fetch via http?
after all, we're completely in control of the client software, and the
useual reason for HTTP-only access is on the client side rather then the
server side. so http access that wraps the git protocol in http would make
life much cleaner for lots of people
there are a few cases where all you have is static web space, but I don't
think it's worth trying to optimize that too much as you still have the
safety issues to worry about
David Lang
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:10
Hi,
On Thu, 17 May 2007, Jan Hudec wrote:
On Thu, May 17, 2007 at 10:41:37 -0400, Nicolas Pitre wrote:
quoted
And if you have 1) the permission and 2) the CPU power to execute such
a cgi on the server and obviously 3) the knowledge to set it up
properly, then why aren't you running the Git daemon in the first
place? After all, they both boil down to running git-pack-objects and
sending out the result. I don't think such a solution really buys
much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate
server, at university while I studied and now I can get the same on
friend's server. Neither of them would probably be ok for serving larger
busy git repository, but something smaller accessed by several people is
OK. I think this is quite common for university students.
1) This has nothing to do with the way the repo is served, but how much
you advertise it. The load will not be lower, just because you use a CGI
script.
2) you say yourself that git-daemon would have less impact on the load:
quoted
[...]
Et voilà. Oh, and of course update your local refs from the
remote's.
Actually there is nothing really complex in the above operations. And
with this the server side remains really simple with no special setup
nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the
git server approach, because at least the indices have to be transfered
in entirety.
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:11
On Thu, May 17, 2007 at 16:38:41 -0400, Nicolas Pitre wrote:
On Thu, 17 May 2007, Jan Hudec wrote:
quoted
A particular case would be a group of students wanting to publish their
software project (I mean the PRG023 or equivalent). Private computers in the
hostel are not allowed to serve anything, so they'd use some of the lab
servers (eg. artax, ss1000...). All of them allow full CGI, but running
daemons is forbiden.
And wouldn't the admin authority for those lab servers be amenable to
install a Git daemon service? That'd be a much better solution to me.
It would. But it would really depend on the administrator goodwill.
--
Jan 'Bulb' Hudec [off-list ref]
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:11
On Fri, May 18, 2007 at 10:01:52 +0100, Johannes Schindelin wrote:
Hi,
On Thu, 17 May 2007, Jan Hudec wrote:
quoted
On Thu, May 17, 2007 at 10:41:37 -0400, Nicolas Pitre wrote:
quoted
And if you have 1) the permission and 2) the CPU power to execute such
a cgi on the server and obviously 3) the knowledge to set it up
properly, then why aren't you running the Git daemon in the first
place? After all, they both boil down to running git-pack-objects and
sending out the result. I don't think such a solution really buys
much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate
server, at university while I studied and now I can get the same on
friend's server. Neither of them would probably be ok for serving larger
busy git repository, but something smaller accessed by several people is
OK. I think this is quite common for university students.
1) This has nothing to do with the way the repo is served, but how much
you advertise it. The load will not be lower, just because you use a CGI
script.
That won't. But that was never the purpose of "smart cgi". The purpose was to
minimize the bandwidth usage (and connectivity is still not so cheap that
you'd not care) while still working over http either because the users need
to access it from behind firewall or because administrator is not willing to
set up git-daemon for you, while CGI you can run yourself.
2) you say yourself that git-daemon would have less impact on the load:
NO, I didn't -- at least not in the paragraph below.
In the below paragraph I said, that *network* use will never be as good with
*dumb* solution, as it can be with smart solution, no matter whether it is
over special protocol or HTTP.
---
Of course it would be less efficient in both CPU and network load, because
there is the overhead of the web server and overhead of the http headers.
Actually I like the ranges solution. If accompanied with repack stategy that
does not pack everything together, but instead creates packs of limited
number of objects -- so that the indices don't exceed configurable size, say
64kB -- could not so much less efficient for the network and have the
advantage of working without ability to execute CGI.
quoted
quoted
[...]
Et voilà. Oh, and of course update your local refs from the
remote's.
Actually there is nothing really complex in the above operations. And
with this the server side remains really simple with no special setup
nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the
git server approach, because at least the indices have to be transfered
in entirety.