From: Jeff King <hidden> Date: 2016-06-15 22:53:35
On Sat, Apr 14, 2012 at 09:13:17PM -0500, Neal Kreitzinger wrote:
Does a file's delta-compression efficiency in the pack-file directly
correlate to its efficiency of transmission size/bandwidth in a
git-fetch and git-push? IOW, are big-files also a problem for
git-fetch and git-push by taking too long in a remote transfer?
Yes. The on-the-wire format is a packfile. We create a new packfile on
the fly, so we may find new deltas (e.g., between objects that were
stored on disk in two different packs), but we will mostly be reusing
deltas from the existing packs.
So any time you improve the on-disk representation, you are also
improving the network bandwidth utilization.
-Peff
On Sat, Apr 14, 2012 at 09:13:17PM -0500, Neal Kreitzinger wrote:
quoted
Does a file's delta-compression efficiency in the pack-file directly
correlate to its efficiency of transmission size/bandwidth in a
git-fetch and git-push? IOW, are big-files also a problem for
git-fetch and git-push by taking too long in a remote transfer?
Yes. The on-the-wire format is a packfile. We create a new packfile on
the fly, so we may find new deltas (e.g., between objects that were
stored on disk in two different packs), but we will mostly be reusing
deltas from the existing packs.
So any time you improve the on-disk representation, you are also
improving the network bandwidth utilization.
We use git to transfer database files from the dev server to
qa-servers. Sometimes these barf for some reason and I get called to
remediate. I assumed the user closed their session prematurely because
it was "taking too long". However, now I'm wondering if the git-pull
--ff-only is dying on its own due to the big-files. It could be that on
a qa-server that hasn't updated database files in awhile they are
pulling way more than another qa-server that does their git-pull more
requently. How would I go about troubleshooting this? Is there some
log files I would look at? (I'm using git 1.7.1 compiled with git
makefile on rhel6.) When I go to remediate do git-reset --hard to clear
out the barfed worktree/index and then run git-pull --ff-only manually
and it always works. I'm not sure if that proves it wasn't git that
barfed the first time. Maybe the first time git brought some stuff over
and barfed because it bit off more than it could chew, but the second
time its really having to chew less food because it already chewed some
of it the first time and therefore works the second time.
v/r,
neal
From: Jeff King <hidden> Date: 2016-06-15 22:53:36
On Sat, Apr 14, 2012 at 09:33:37PM -0500, Neal Kreitzinger wrote:
We use git to transfer database files from the dev server to
qa-servers. Sometimes these barf for some reason and I get called to
remediate. I assumed the user closed their session prematurely
because it was "taking too long". However, now I'm wondering if the
git-pull --ff-only is dying on its own due to the big-files. It
could be that on a qa-server that hasn't updated database files in
awhile they are pulling way more than another qa-server that does
their git-pull more requently. How would I go about troubleshooting
this? Is there some log files I would look at? (I'm using git 1.7.1
compiled with git makefile on rhel6.)
No, git doesn't keep logfiles. Errors go to stderr. So look wherever the
stderr for your git sessions is going (if you are doing this via cron
job or something, then that is outside the scope of git).
When I go to remediate do git-reset --hard to clear out the barfed
worktree/index and then run git-pull --ff-only manually and it always
works. I'm not sure if that proves it wasn't git that barfed the
first time. Maybe the first time git brought some stuff over and
barfed because it bit off more than it could chew, but the second time
its really having to chew less food because it already chewed some of
it the first time and therefore works the second time.
Try "git pull --no-progress" and see if it still works. If the server
has a very long delta-compression phase, there will be no output
generated for a while, which could cause intermediate servers to hang up
(git won't do this, but if, for example, you are pulling over
git-over-http and there is a reverse proxy in the middle, it may hit a
timeout). If the automated pulls are happening from a cron job, then
they won't have a terminal and progress-reporting will be off by
default.
-Peff
On Sat, Apr 14, 2012 at 09:13:17PM -0500, Neal Kreitzinger wrote:
quoted
Does a file's delta-compression efficiency in the pack-file directly
correlate to its efficiency of transmission size/bandwidth in a
git-fetch and git-push? IOW, are big-files also a problem for
git-fetch and git-push by taking too long in a remote transfer?
Yes. The on-the-wire format is a packfile. We create a new packfile on
the fly, so we may find new deltas (e.g., between objects that were
stored on disk in two different packs), but we will mostly be reusing
deltas from the existing packs.
So any time you improve the on-disk representation, you are also
improving the network bandwidth utilization.
The git-clone manpage says you can use the rsync protocol for the url.
If you use rsync:// as your url for your remote does that get you the
rsync delta-transfer algorithm efficiency for the network bandwidth
utilization part (as opposed to the on-disk representation part)? (I'm
new to rsync.)
v/r,
neal
From: Jeff King <hidden> Date: 2016-06-15 22:53:48
On Thu, May 10, 2012 at 04:43:26PM -0500, Neal Kreitzinger wrote:
quoted
Yes. The on-the-wire format is a packfile. We create a new packfile on
the fly, so we may find new deltas (e.g., between objects that were
stored on disk in two different packs), but we will mostly be reusing
deltas from the existing packs.
So any time you improve the on-disk representation, you are also
improving the network bandwidth utilization.
The git-clone manpage says you can use the rsync protocol for the
url. If you use rsync:// as your url for your remote does that get
you the rsync delta-transfer algorithm efficiency for the network
bandwidth utilization part (as opposed to the on-disk representation
part)? (I'm new to rsync.)
Well, yes. If you use the rsync transport, it literally runs rsync,
which will use the regular rsync algorithm. But it won't be better than
the git protocol (and in fact will be much worse) for a few reasons:
1. The object db files are all named after the sha1 of their content
(the object sha1 for loose objects, and the sha1 of the whole pack
for packfiles). Rsync will not run its comparison algorithm between
files with different names. It will not re-transfer existing loose
objects, but it will delete obsolete packfiles and retransfer new
ones in their entirety. So it's like re-cloning over again for any
fetch after an upstream repack.
2. Even if you could use the rsync delta algorithm, it will never be
as efficient as git. Git understands the structure of the packfile
and can tell the other side "Hey, I have these objects". Whereas
rsync must guess from the bytes in the packfiles. Which is much
less efficient to compute, and can be wrong if the representation
has changed (e.g., something used to be a whole object, but is now
stored as a delta).
3. Even if you could get the exact right set of objects to transfer,
and then use the rsync delta algorithm on them, git would still do
better. Git's job is much easier: one side has both sets of
objects (those to be sent and those not), and is generating and
sending efficient deltas for the other side to apply to their
objects. Rsync assumes a harder job: you have one set, and
the remote side has the other set, and you must agree on a delta by
comparing checksums. So it will fundamentally never do as well.
-Peff