Are binary xdeltas only used if you use git-gc?

Hi everyone.

I've been usig Git for the last couple of months and am quite happy with it.
In one of my Git repositories, I am storing uncompressed .tar files
(since being uncompressed allows git to detect and store
only their "real"differences).

However, when I introduce a new filename in the repos (with a minor
set of differences compared to an existing file with a different filename)
I've been unsuccessful in finding a way to tell Git to do it efficiently...

This is what I mean:

bash$ mkdir -p /var/tmp/tst
bash$ cd /var/tmp/tst
bash$ git init
bash$ cp /var/www/renderer-2.0e.tar .
bash$ git add renderer-2.0e.tar
bash$ git commit -m "First version"
bash$ du -s -k .git/
1724    .git/
bash$ cp renderer-2.0e.tar renderer-2.0f.tar
bash$ git add renderer-2.0f.tar
bash$ git commit -m "To add new version, first copy the first, so Git
detects it"
bash$ du -s -k .git/
1740    .git/
bash$ echo Good, Git detected it is the same
bash$ cp /var/www/renderer-2.0f.tar .
bash$ git add renderer-2.0f.tar
bash$ git commit -m "Real new version, slightly different to first"
bash$ du -s -k .git/
3344    .git/
bash$ echo What... did I do something wrong
bash$ xdelta delta renderer-2.0e.tar renderer-2.0f.tar delta
bash$ ls -l
total 7788
-rw-r--r-- 1 ttsiod ttsiod    8181 2008-10-31 11:27 delta
-rw-r--r-- 1 ttsiod ttsiod 3962880 2008-10-31 11:23 renderer-2.0e.tar
-rw-r--r-- 1 ttsiod ttsiod 3993600 2008-10-31 11:25 renderer-2.0f.tar
bash$ git-gc
bash$ du -s -k .git/
1660    .git/

So even though the xdelta is just 8KB, and git-gc actually finds out
that indeed
the new file is very similar to the old one, the initial commit of the
new version
in the repos is not taking advantage.

I found out about this when I tried to "git push" over a PSTN modem...

Then again, I must confess I only did the git-gc after I pushed.
Does the git-push actually take advantage of the similarities only if
I do a git-gc first?

If that is the case, I will create an alias to always git-gc after commits...

--
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph

On Fri, Oct 31, 2008 at 09:43:43AM +0000, Thanassis Tsiodras wrote:

So even though the xdelta is just 8KB, and git-gc actually finds out
that indeed
the new file is very similar to the old one, the initial commit of the
new version
in the repos is not taking advantage.

Have you tried to git repack with aggressive options, like:

    git repack --window=500 --depth=500 \
      --window-memory=<fair amount of your physical RAM>

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

"Thanassis Tsiodras" [off-list ref] writes:

I've been usig Git for the last couple of months and am quite happy with it.
In one of my Git repositories, I am storing uncompressed .tar files
(since being uncompressed allows git to detect and store
only their "real"differences).

I think you can use clean / smudge filter in gitattributes for that.

[...]

Then again, I must confess I only did the git-gc after I pushed.
Does the git-push actually take advantage of the similarities only if
I do a git-gc first?

Git does deltification _only_ in packfiles. But when you push via SSH
git would generate a pack file with commits the other side doesn't
have, and those packs are thin packs, so they also have deltas... but
the remote side then adds bases to those thin packs making them
standalone: you would have to git-gc on remote.

HTH
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Actually, after using git-gc, git-repack isn't really needed...
git-gc identifies that the two files are very similar and re-deltifies
(see the du -s -k outputs in the original mail, after git-gc we have
in fact lower usage than the first commit).

My question is basically...
(a) why doesn't git detect this during commit and needs a git-gc
(b) whether after git-gc I would have seen the massive difference
during a subsequent git-push or not

Thanassis.

Have you tried to git repack with aggressive options, like:

   git repack --window=500 --depth=500 \
     --window-memory=<fair amount of your physical RAM>

-- 
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph

On Fri, Oct 31, 2008 at 1:15 PM, Jakub Narebski [off-list ref] wrote:

I think you can use clean / smudge filter in gitattributes for that.

Thanks, I didn't know about that. Will look into it

Git does deltification _only_ in packfiles. But when you push via SSH
git would generate a pack file with commits the other side doesn't
have, and those packs are thin packs, so they also have deltas... but
the remote side then adds bases to those thin packs making them
standalone: you would have to git-gc on remote.

So I have to git-gc on my side (after the commits), git-gc on the remote,
and then git-push?

What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph

Thanassis Tsiodras wrote:

On Fri, Oct 31, 2008 at 1:15 PM, Jakub Narebski [off-list ref] wrote:

quoted

Git does deltification _only_ in packfiles. But when you push via SSH
git would generate a pack file with commits the other side doesn't
have, and those packs are thin packs, so they also have deltas... but
the remote side then adds bases to those thin packs making them
standalone: you would have to git-gc on remote.

So I have to git-gc on my side (after the commits), git-gc on the remote,
and then git-push?

Perhaps I haven't made myself clear.

On the local side: git-commit creates loose (compressed, but not
deltified) objects. git-gc packs and deltifies.

On the remote side (for smart protocols, i.e. git and ssh): git
creates _thin_ pack, deltified; on the remote side git either makes
pack thick/self contained by adding base objects (object + deltas),
or explodes pack into loose object (object). You need git-gc on
remote server to fully deltify on remote side. But transfer is fully
deltified.

On the remote side (for dumb protocols, i.e. rsync and http): git
finds required packs and transfers them whole. So the situation is
like on local side, but git might transfer more than really needed
because it transfers packs in full.

HTH.
-- 
Jakub Narebski
Poland

Jakub Narebski wrote:

"Thanassis Tsiodras" [off-list ref] writes:

quoted

Then again, I must confess I only did the git-gc after I pushed.
Does the git-push actually take advantage of the similarities only if
I do a git-gc first?

Git does deltification _only_ in packfiles. But when you push via SSH
git would generate a pack file with commits the other side doesn't
have, and those packs are thin packs, so they also have deltas...

AFAICT, git stopped pushing thin packs by default with 1.5.3.2, so
you have to explicitely ask for it.  The original poster might not
be clear about this (or even know what a thin pack is).

Thanassis, try to use "git push --thin".  'man git-push' says:

  --thin, --no-thin
      These options are passed to git-send-pack. Thin transfer spends
      extra cycles to minimize the number of objects to be sent and meant
      to be used on slower connection.

I did a quick test with big random files and it indeed only sends
small deltas on small changes, but if you don't pass --thin, it
will send the full objects.

I didn't find a configuration variable to change that default.  It
would make sense for people that regularly push over slow lines.

Hope this helps,
jlh

On Fri, 31 Oct 2008, Pierre Habouzit wrote:

On Fri, Oct 31, 2008 at 09:43:43AM +0000, Thanassis Tsiodras wrote:

quoted

So even though the xdelta is just 8KB, and git-gc actually finds out
that indeed
the new file is very similar to the old one, the initial commit of the
new version
in the repos is not taking advantage.

Have you tried to git repack with aggressive options, like:

    git repack --window=500 --depth=500 \
      --window-memory=<fair amount of your physical RAM>

That wouldn't bring any benefit in this case.


Nicolas

On Fri, 31 Oct 2008, Thanassis Tsiodras wrote:

Actually, after using git-gc, git-repack isn't really needed...
git-gc identifies that the two files are very similar and re-deltifies
(see the du -s -k outputs in the original mail, after git-gc we have
in fact lower usage than the first commit).

git-gc does call git-repack already.  In fact, git-gc is only a 
convenience wrapper for a couple maintenance commands.

My question is basically...
(a) why doesn't git detect this during commit and needs a git-gc

Because we want commit operations to be fast. One of many usage 
scenarios for git is to apply a large amount of patches in one go, 
meaning many commits per second.  The gc operation is potentially long, 
can be done unfrequently and deferred any time, like when you don't have 
to wait for it.

(b) whether after git-gc I would have seen the massive difference
during a subsequent git-push or not

No.  A push does create a pack with best size reduction already, whether 
or not your local or remote repositories are already packed.  The only 
advantage for having your local repository packed is in the time 
required to create that same pack to be pushed.

As mentioned already, you should consider the --thin switch if you are 
pushing over a slow link.  If the remote repository already has 
necessary objects then all the pushed pack will contain is all deltas.

The reason why --thin is not activated by default is because most people 
do pulls from a central server and only few people do pushes to such a 
server, and while thin packs do reduce the transmission size they do 
create slightly bigger packs on the receiving end which is best avoided 
on a busy server.


Nicolas