From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:18
Linus Torvalds [off-list ref] writes:
Anyway, _something_ like this is definitely needed. It could certainly be
better (if it showed the same kind of thing that git-unpack-objects did,
that would be much nicer, but would require parsing the object stream as
it comes in). But this is big step forward, I think.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
Comments? Hate-mail? Improvements?
It probably should default to quiet if (!isatty(1)).
The real improvement, independent of this client-side patch,
would be to reuse recently generated packs, but that needs
writable cache directory on the server side. Another thing that
I stumbled upon last time I tried it was that it did not look
totally trivial to modify the csum-file interface so that I can
splice the output from it into two different destinations (one
to cachefile, the other to the consumer).
On 11 Feb 2006, at 7:48 AM, Junio C Hamano wrote:
[snip]
The real improvement, independent of this client-side patch,
would be to reuse recently generated packs, but that needs
writable cache directory on the server side.
Speaking of improvements, I've noticed that my attempts to track
the 2.6 kernel via the git protocol result in inefficiencies from time
to time when the connection hangs or is terminated when my
flakey wireless link goes down. When I restart the pull, the data
that has already been downloaded is lost and things start from
scratch which is painful if it's a big update.
It would be nice if the "partial pack" or whatever that has been
downloaded at the time of the breakage could be re-used and
things could start "from that point onwards" or the bits that were
already received could be unpacked. Comments?
Thank you,
--Craig
On 11 Feb 2006, at 7:48 AM, Junio C Hamano wrote:
It would be nice if the "partial pack" or whatever that has been
downloaded at the time of the breakage could be re-used and
things could start "from that point onwards" or the bits that were
already received could be unpacked. Comments?
It even already works on plain http repos with git fetch.
(e.g. WineHQ repository)
Why git protocol doesn't support it?
+10
--
GPG Key id: 0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2
AstralStorm
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:18
Dear diary, on Sat, Feb 11, 2006 at 09:44:00AM CET, I got a letter
where Radoslaw Szkodzinski [off-list ref] said that...
Craig Schlenter wrote:
quoted
On 11 Feb 2006, at 7:48 AM, Junio C Hamano wrote:
It would be nice if the "partial pack" or whatever that has been
downloaded at the time of the breakage could be re-used and
things could start "from that point onwards" or the bits that were
already received could be unpacked. Comments?
It even already works on plain http repos with git fetch.
(e.g. WineHQ repository)
Why git protocol doesn't support it?
Because it works totally different. When downloading from plain HTTP
repos, you are just downloading files from the remote repository and it
is easy to pick up wherever you left (and last night, I just added a
possibility to Cogito to resume an interrupted cg-clone by just cd'ing
inside and running cg-fetch, as is; it's pretty neat) - you just resume
downloading of the file you downloaded last, and don't download again
the files you already have.
But the native git protocol works completely differently - you tell the
server "give me all objects you have between object X and head", the
object will generate a completely custom pack just for you and send it
over the network. The next time you fetch, you just ask for a pack
between object X and head again, but the head can be already totally
different. What we would have to do is to check for interrupted
packfiles before fetching, attempt to fix them (cutting out the
incomplete objects and broken delta chains, if applicable), and then
tell the remote side to skip those objects; but that may not be easy
because there can be a lot of "loose fibres". Another way would be to
just tell the server "if head is still Y, start sending the pack only
after N bytes". *shudder*
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Of the 3 great composers Mozart tells us what it's like to be human,
Beethoven tells us what it's like to be Beethoven and Bach tells us
what it's like to be the universe. -- Douglas Adams
But the native git protocol works completely differently - you tell the
server "give me all objects you have between object X and head", the
object will generate a completely custom pack just for you and send it
over the network. The next time you fetch, you just ask for a pack
between object X and head again, but the head can be already totally
different. What we would have to do is to check for interrupted
packfiles before fetching, attempt to fix them (cutting out the
incomplete objects and broken delta chains, if applicable), and then
tell the remote side to skip those objects; but that may not be easy
because there can be a lot of "loose fibres". Another way would be to
just tell the server "if head is still Y, start sending the pack only
after N bytes". *shudder*
The other way would be:
- generate pack file between X and Y
- start sending from N bytes
It could break if the repo has been rebased in the meantime.
But we could safeguard against it by sending the hash of the packfile
up to N bytes.
--
GPG Key id: 0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2
AstralStorm
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:18
BTW, some historical (from the very channel beginning) logs of #git for
fun, profit and late night reading are available at
http://pasky.or.cz/~pasky/cp/%23git/, e.g. the 2006-02-10 early morning
features the King Penguin explaining the deepness and intricacies of
pack files construction! Don't miss the opportunity!
New files won't be world-readable by default, but I hope to get some
irclogger with cutesy web interface set up for #git.
Dear diary, on Sat, Feb 11, 2006 at 06:48:55AM CET, I got a letter
where Junio C Hamano [off-list ref] said that...
Linus Torvalds [off-list ref] writes:
quoted
Anyway, _something_ like this is definitely needed. It could certainly be
better (if it showed the same kind of thing that git-unpack-objects did,
that would be much nicer, but would require parsing the object stream as
it comes in). But this is big step forward, I think.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
Comments? Hate-mail? Improvements?
It probably should default to quiet if (!isatty(1)).
isatty(2) or something, 1 is in practice always a ref generator. Perhaps
it would be better not to clutter stderr, though; what about directly
opening /dev/tty? Does Cygwin support that?
The real improvement, independent of this client-side patch,
would be to reuse recently generated packs, but that needs
writable cache directory on the server side. Another thing that
I stumbled upon last time I tried it was that it did not look
totally trivial to modify the csum-file interface so that I can
splice the output from it into two different destinations (one
to cachefile, the other to the consumer).
Yes, I said that on IRC yesterday as well. I don't think even a cache is
needed; just look at the repository and say:
* while there are packs containing only objects we are going to
send, pick the largest one and send it as-is.
* if there is a pack with more than a 75% (totally arbitrary)
overlap with the objects we are going to send, send it as-is.
* pack the loose objects.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Of the 3 great composers Mozart tells us what it's like to be human,
Beethoven tells us what it's like to be Beethoven and Bach tells us
what it's like to be the universe. -- Douglas Adams
From: Petr Baudis <hidden> Date: 2016-06-15 22:42:18
Dear diary, on Sat, Feb 11, 2006 at 02:33:40PM CET, I got a letter
where Petr Baudis [off-list ref] said that...
BTW, some historical (from the very channel beginning) logs of #git for
fun, profit and late night reading are available at
http://pasky.or.cz/~pasky/cp/%23git/, e.g. the 2006-02-10 early morning
features the King Penguin explaining the deepness and intricacies of
pack files construction! Don't miss the opportunity!
New files won't be world-readable by default, but I hope to get some
irclogger with cutesy web interface set up for #git.
Like,
http://colabti.de/irclogger/irclogger_logs/git
(Courtesy of Francois Beerten.)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Of the 3 great composers Mozart tells us what it's like to be human,
Beethoven tells us what it's like to be Beethoven and Bach tells us
what it's like to be the universe. -- Douglas Adams
From: Alex Riesen <hidden> Date: 2016-06-15 22:42:18
Petr Baudis, Sat, Feb 11, 2006 14:33:40 +0100:
quoted
It probably should default to quiet if (!isatty(1)).
isatty(2) or something, 1 is in practice always a ref generator. Perhaps
it would be better not to clutter stderr, though; what about directly
opening /dev/tty? Does Cygwin support that?
It can't. Windows has no terminals (as in "none at all"). It has a
Console, which is a special kind of window attached to an application
and where the unbuffered stdout and stderr are magically redirected.
A test for is stdout/err is a tty can only check if the process has
the console attached, and an attempt to open it for writing will
probably just create the thing.
It probably should default to quiet if (!isatty(1)).
Sounds fine. isatty(2), though, since we use stderr for these messages
(stdout is usually the data-stream).
The real improvement, independent of this client-side patch,
would be to reuse recently generated packs, but that needs
writable cache directory on the server side.
More importantly, it really wouldn't have helped that much in this
situation. At least for me, the network is 90% of the problem, the
pack-file generation is at most 10%. So cached packfiles really only
matter for server-side problems (high CPU load, or lack of memory, or
heavy disk activity).
So the problems really are very independent.
Linus
From: Keith Packard <keithp@keithp.com> Date: 2016-06-15 22:42:18
On Sat, 2006-02-11 at 09:45 -0800, Linus Torvalds wrote:
More importantly, it really wouldn't have helped that much in this
situation. At least for me, the network is 90% of the problem, the
pack-file generation is at most 10%. So cached packfiles really only
matter for server-side problems (high CPU load, or lack of memory, or
heavy disk activity).
I'd like to see git use less CPU than CVS does on my distribution host;
some mechanism for re-using either existing or cached packs would help a
whole lot with that. The alternative is to see people switch to rsync
instead, which seems like a far worse idea.
--
keith.packard@intel.com
From: Andreas Ericsson <hidden> Date: 2016-06-15 22:42:18
Keith Packard wrote:
On Sat, 2006-02-11 at 09:45 -0800, Linus Torvalds wrote:
quoted
More importantly, it really wouldn't have helped that much in this
situation. At least for me, the network is 90% of the problem, the
pack-file generation is at most 10%. So cached packfiles really only
matter for server-side problems (high CPU load, or lack of memory, or
heavy disk activity).
I'd like to see git use less CPU than CVS does on my distribution host;
some mechanism for re-using either existing or cached packs would help a
whole lot with that. The alternative is to see people switch to rsync
instead, which seems like a far worse idea.
A weird oddity; Cloning is faster over rsync, day-to-day pulling is not.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
From: Keith Packard <keithp@keithp.com> Date: 2016-06-15 22:42:18
On Sun, 2006-02-12 at 04:43 +0100, Andreas Ericsson wrote:
A weird oddity; Cloning is faster over rsync, day-to-day pulling is not.
Precisely. If the protocol could deliver existing packs instead of
unpacking and repacking them, then git would be as fast as rsync and I
wouldn't have to worry about supporting two protocols.
--
keith.packard@intel.com
From: Andreas Ericsson <hidden> Date: 2016-06-15 22:42:18
Keith Packard wrote:
On Sun, 2006-02-12 at 04:43 +0100, Andreas Ericsson wrote:
quoted
A weird oddity; Cloning is faster over rsync, day-to-day pulling is not.
Precisely. If the protocol could deliver existing packs instead of
unpacking and repacking them, then git would be as fast as rsync and I
wouldn't have to worry about supporting two protocols.
Caching features have been discussed, but that means the daemon needs to
have write-access to some directory within the repository. It would also
work poorly for projects that see very rapid development unless the
cached pack-files can be amended to. A sort of "create packs on demand".
It shouldn't be too difficult, really.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:19
On 2/12/06, Keith Packard [off-list ref] wrote:
On Sun, 2006-02-12 at 04:43 +0100, Andreas Ericsson wrote:
quoted
A weird oddity; Cloning is faster over rsync, day-to-day pulling is not.
Precisely. If the protocol could deliver existing packs instead of
unpacking and repacking them, then git would be as fast as rsync and I
wouldn't have to worry about supporting two protocols.
+1... there should be an easy-to-compute threshold trigger to say --
hey, let's quit being smart and send this client the packs we got and
get it over with. Or perhaps a client flag so large projects can
recommend that uses do their initial clone with --gimme-all-packs?
My workaround for large repos is to clone over http, and s/http:/git:/
on the origin file once it's done ;-)
martin
From: Keith Packard <keithp@keithp.com> Date: 2016-06-15 22:42:19
On Sun, 2006-02-12 at 12:02 +0100, Andreas Ericsson wrote:
Caching features have been discussed, but that means the daemon needs to
have write-access to some directory within the repository.
Caching seems a bit dicey to me; security concerns and all. I would much
rather have it discover packs on disk that provided a subset of the
necessary objects; repository cloning would then be a process of
delivering any available packs and then packing up the remaining
objects. Clever administration of the repository could then construct a
single pack of 'historical' data followed by periodic packs of
incremental data.
Yeah, I know, I should just implement this and see how well it works in
practice. I apologize for thinking in public.
--
keith.packard@intel.com