So I have to git-gc on my side (after the commits), git-gc on the remote,
and then git-push?
Perhaps I haven't made myself clear.
On the local side: git-commit creates loose (compressed, but not
deltified) objects. git-gc packs and deltifies.
On the remote side (for smart protocols, i.e. git and ssh): git
creates _thin_ pack, deltified;
I don't understand this point: the OP talks about pushing, so isn't
the pack created on the _local_ machine (and then sent to the remote)?
--
Matthieu
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:45:33
On Fri, 31 Oct 2008, Matthieu Moy wrote:
Jakub Narebski [off-list ref] writes:
quoted
Thanassis Tsiodras wrote:
quoted
So I have to git-gc on my side (after the commits), git-gc on the remote,
and then git-push?
Perhaps I haven't made myself clear.
On the local side: git-commit creates loose (compressed, but not
deltified) objects. git-gc packs and deltifies.
On the remote side (for smart protocols, i.e. git and ssh): git
creates _thin_ pack, deltified;
I don't understand this point: the OP talks about pushing, so isn't
the pack created on the _local_ machine (and then sent to the remote)?
Yes, the pack is created on the fly when pushing, regardless if the repo
is already packed or not locally. The only difference a locally packed
repo provides is a shorter "Compressing objects" phase when pushing
that's all. The packedness of the remote has no effect at all.
Nicolas
Thanks to everybody for your help.
I will setup an alias to always use "git push --thin".
For the reverse direction, I don't see a --thin for "git pull",
My understanding is that "git pull" is optimal,
and does what --thin does for push anyway, right?
On 10/31/08, Nicolas Pitre [off-list ref] wrote:
On Fri, 31 Oct 2008, Matthieu Moy wrote:
quoted
Jakub Narebski [off-list ref] writes:
quoted
Thanassis Tsiodras wrote:
quoted
So I have to git-gc on my side (after the commits), git-gc on the
remote,
and then git-push?
Perhaps I haven't made myself clear.
On the local side: git-commit creates loose (compressed, but not
deltified) objects. git-gc packs and deltifies.
On the remote side (for smart protocols, i.e. git and ssh): git
creates _thin_ pack, deltified;
I don't understand this point: the OP talks about pushing, so isn't
the pack created on the _local_ machine (and then sent to the remote)?
Yes, the pack is created on the fly when pushing, regardless if the repo
is already packed or not locally. The only difference a locally packed
repo provides is a shorter "Compressing objects" phase when pushing
that's all. The packedness of the remote has no effect at all.
Nicolas
--
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:45:34
On Sat, 1 Nov 2008, Thanassis Tsiodras wrote:
Thanks to everybody for your help.
I will setup an alias to always use "git push --thin".
For the reverse direction, I don't see a --thin for "git pull",
My understanding is that "git pull" is optimal,
and does what --thin does for push anyway, right?
Despair...
I just tested "git push --thin"...
Doesn't work.
It still sends the complete object, not a tiny pack as it could (should).
But perhaps I now understand why:
I run git-gc on both the remote end and the working end (before
changing anything,
i.e. with both repos being in sync - "git pull" and "git push" report all OK).
I then noticed that on the remote side, .git/objects/pack had one big pack file,
but on the local one I have two .pack files...!
I proceeded to try (many combinations of params on) git-repack in a vain attempt
to make my local repos also have one single .pack file (presumably, it
should be able
to exactly mirror the remote one, since it has the same objects inside
it!). No way...
git-prune and "git-fsck --full --strict --unreachable" report no errors either.
I'm at a loss as to why the two repos are having different "pack
representation" of the
same objects and why git-gc and git-repack fail to create a single
pack on my working
side, but I'm guessing that this is why "git push --thin" fails to
send small xdeltas...
Any help/advice on what to try next would be most welcome...
Thanassis.
On 11/1/08, Nicolas Pitre [off-list ref] wrote:
On Sat, 1 Nov 2008, Thanassis Tsiodras wrote:
quoted
Thanks to everybody for your help.
I will setup an alias to always use "git push --thin".
For the reverse direction, I don't see a --thin for "git pull",
My understanding is that "git pull" is optimal,
and does what --thin does for push anyway, right?
Exact.
Nicolas
--
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:45:34
On Mon, 3 Nov 2008, Thanassis Tsiodras wrote:
Despair...
I just tested "git push --thin"...
Doesn't work.
It still sends the complete object, not a tiny pack as it could (should).
But perhaps I now understand why:
I run git-gc on both the remote end and the working end (before
changing anything,
i.e. with both repos being in sync - "git pull" and "git push" report all OK).
I then noticed that on the remote side, .git/objects/pack had one big pack file,
but on the local one I have two .pack files...!
I proceeded to try (many combinations of params on) git-repack in a vain attempt
to make my local repos also have one single .pack file (presumably, it
should be able
to exactly mirror the remote one, since it has the same objects inside
it!). No way...
Please stop thinking that your repository layout has anything to do with
what is actually transferred on a push. It has not.
Here's a small test that you can do locally:
mkdir repo_a
mkdir repo_b
cd repo_a
git init
seq 1000000 > data
git add data
git commit -m "initial commit"
cd ../repo_b
git init
cd ../repo_a
git push ../repo_b master:master
Here you should see a line that says:
Writing objects: 100% (3/3), 2.01 MiB, done.
Therefore 2.1 MiB were transferred. Now let's continue:
echo "foo" >> data
git add data
git commit -m "second commit"
git push ../repo_b master:master
You should get:
Writing objects: 100% (3/3), 423 bytes, done.
And this means that you even don't need the --thin switch (which is
wrong -- this has been broken before but that's another story) for the
transfer to actually send only the difference and not the whole file
again. And note that none of those repositoryes actually contain any
pack as everything is still loose objects.
I'm at a loss as to why the two repos are having different "pack
representation" of the same objects
That's only because those objects entered each repositories in a
different way.
and why git-gc and git-repack fail
to create a single pack on my working side,
Maybe you have a .keep file in .git/objects/pack/ ? If so delete it and
run 'git repack -a -d'.
but I'm guessing that this is why "git push --thin" fails to send
small xdeltas...
Not at all.
Please provide a complete log of your tests and maybe we could find
something.
Nicolas
RESOLVED!!!
Finally...
What happened was actually quite reasonable, in hindsight...
As I said in the original mail, this was what I did:
cp version7.1.tar version7.2.tar
git add version7.2.tar
git commit -m "same data as old, so git will use old blob"
echo MAGICPLACE read below...
cp /path/to/work/realNewVersion7.2.tar version7.2.tar
git add version7.2.tar
git commit -m "and now, commit the really new version, so git can xdelta"
git push --thin
The problem was solved (that is, the "git push" became optimal,
when I added a "git push" right after the MAGICPLACE mark above...
In that way, the remote repo learned about the "dummy" commit that
referenced the old blob... and when I did the subsequent "git push"
at the end, the remote side could see that it already had this "dummy"
commit to "xdelta on", and that it only needed the delta...
Originally, when I used only one "git push --thin" at the end, the remote
side didn't have the "dummy" commit, so it probably said: "I can't
apply a delta, give me the full object".
Phew.
So it seems that if you must introduce a new file that is
very similar to an existing one (in my case, a new version
of software kept in an uncompressed .tar file),
you have to do what I did above to allow for optimal "git push"es:
that is:
1. Create the new filename by just copying the old
(so the old blob is used)
2. commit
3. PUSH
4. copy the real new file
5. commit
6. PUSH.
If you omit the middle PUSH in step 3, neither "git push", nor "git push --thin"
can realize that this new file can be "incrementally built" on the remote side
(even though git-gc totally squashes it in the pack).
Thanks to all the people who responded, and especially Nicolas...
Merci!
On 11/3/08, Nicolas Pitre [off-list ref] wrote:
On Mon, 3 Nov 2008, Thanassis Tsiodras wrote:
quoted
Despair...
I just tested "git push --thin"...
Doesn't work.
It still sends the complete object, not a tiny pack as it could (should).
But perhaps I now understand why:
I run git-gc on both the remote end and the working end (before
changing anything,
i.e. with both repos being in sync - "git pull" and "git push" report all
OK).
I then noticed that on the remote side, .git/objects/pack had one big pack
file,
but on the local one I have two .pack files...!
I proceeded to try (many combinations of params on) git-repack in a vain
attempt
to make my local repos also have one single .pack file (presumably, it
should be able
to exactly mirror the remote one, since it has the same objects inside
it!). No way...
Please stop thinking that your repository layout has anything to do with
what is actually transferred on a push. It has not.
Here's a small test that you can do locally:
mkdir repo_a
mkdir repo_b
cd repo_a
git init
seq 1000000 > data
git add data
git commit -m "initial commit"
cd ../repo_b
git init
cd ../repo_a
git push ../repo_b master:master
Here you should see a line that says:
Writing objects: 100% (3/3), 2.01 MiB, done.
Therefore 2.1 MiB were transferred. Now let's continue:
echo "foo" >> data
git add data
git commit -m "second commit"
git push ../repo_b master:master
You should get:
Writing objects: 100% (3/3), 423 bytes, done.
And this means that you even don't need the --thin switch (which is
wrong -- this has been broken before but that's another story) for the
transfer to actually send only the difference and not the whole file
again. And note that none of those repositoryes actually contain any
pack as everything is still loose objects.
quoted
I'm at a loss as to why the two repos are having different "pack
representation" of the same objects
That's only because those objects entered each repositories in a
different way.
quoted
and why git-gc and git-repack fail
to create a single pack on my working side,
Maybe you have a .keep file in .git/objects/pack/ ? If so delete it and
run 'git repack -a -d'.
quoted
but I'm guessing that this is why "git push --thin" fails to send
small xdeltas...
Not at all.
Please provide a complete log of your tests and maybe we could find
something.
Nicolas
--
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:45:36
On Tue, 4 Nov 2008, Thanassis Tsiodras wrote:
RESOLVED!!!
Finally...
What happened was actually quite reasonable, in hindsight...
As I said in the original mail, this was what I did:
cp version7.1.tar version7.2.tar
git add version7.2.tar
git commit -m "same data as old, so git will use old blob"
echo MAGICPLACE read below...
cp /path/to/work/realNewVersion7.2.tar version7.2.tar
git add version7.2.tar
git commit -m "and now, commit the really new version, so git can xdelta"
git push --thin
The problem was solved (that is, the "git push" became optimal,
when I added a "git push" right after the MAGICPLACE mark above...
In that way, the remote repo learned about the "dummy" commit that
referenced the old blob... and when I did the subsequent "git push"
at the end, the remote side could see that it already had this "dummy"
commit to "xdelta on", and that it only needed the delta...
Originally, when I used only one "git push --thin" at the end, the remote
side didn't have the "dummy" commit, so it probably said: "I can't
apply a delta, give me the full object".
Oh! But of course...
In fact, the way thin packs work is to store delta against a base object
which is not included in the pack. Those objects which are not included
but used as delta base are currently only the previous version of a file
which is part of the update to be pushed/fetched. In other words, there
must be a previous version under the same name for this to work. Doing
otherwise wouldn't scale if the previous commit had thousands of files
to test against.
But this particularity had escaped my mind somehow.
Phew.
So it seems that if you must introduce a new file that is
very similar to an existing one (in my case, a new version
of software kept in an uncompressed .tar file),
you have to do what I did above to allow for optimal "git push"es:
that is:
1. Create the new filename by just copying the old
(so the old blob is used)
2. commit
3. PUSH
4. copy the real new file
5. commit
6. PUSH.
If you omit the middle PUSH in step 3, neither "git push", nor "git push --thin"
can realize that this new file can be "incrementally built" on the remote side
(even though git-gc totally squashes it in the pack).
Right. Those thin packs were designed for different versions of the
same file in mind, not different files with almost the same content.
This could possibly be improved at some point...
Nicolas