From: Junio C Hamano <hidden> Date: 2016-06-15 22:43:02
"J. Bruce Fields" [off-list ref] writes:
On Thu, Mar 29, 2007 at 10:50:51PM +0200, Guennadi Liakhovetski wrote:
quoted
Now, my copy of Linus' tree was ATM 1.5GiB big... Slowly it's getting
scary.
On my laptop:
[bfields@pad linux]$ du -hs .
1.5G .
[bfields@pad linux]$ du -hs .git
334M .git
So it's mostly the checked out working directory and build
stuff.
If you really need a ton of build trees then you might just want to do
cp -al or something.
From: J. Bruce Fields <hidden> Date: 2016-06-15 22:43:02
On Thu, Mar 29, 2007 at 02:26:10PM -0700, Junio C Hamano wrote:
"J. Bruce Fields" [off-list ref] writes:
quoted
On Thu, Mar 29, 2007 at 10:50:51PM +0200, Guennadi Liakhovetski wrote:
quoted
Now, my copy of Linus' tree was ATM 1.5GiB big... Slowly it's getting
scary.
On my laptop:
[bfields@pad linux]$ du -hs .
1.5G .
[bfields@pad linux]$ du -hs .git
334M .git
So it's mostly the checked out working directory and build
stuff.
If you really need a ton of build trees then you might just want to do
cp -al or something.
How about suggesting "clone -l -s"?
If you really want to share as much as possible, then I guess you want
to share the working trees too, since (as evidenced above), they're at
least as large as the compressed history.
Though actually on a second look, clone -l -s produces something that's
only 377M. I hadn't realized how much space the build output takes up.
So judging from du the 1.5G Guennadi Liakhovetski mentions above seems
to break down into something like:
330M .git
380M working tree
750M build output
--b.
On Thu, Mar 29, 2007 at 02:26:10PM -0700, Junio C Hamano wrote:
quoted
How about suggesting "clone -l -s"?
Yes, but how do "advanced git users" kernel developers work? Do they just
do 1 clone and build / clean every time they want to test another
configuration / arch, or do they clone -l or what? Do they create branches
for each development thread, then pull / push between trees?...
If you really want to share as much as possible, then I guess you want
to share the working trees too, since (as evidenced above), they're at
least as large as the compressed history.
But I don't want to re-build. Apart from i386 I build for a couple of ARM
and PPC targets too...
Though actually on a second look, clone -l -s produces something that's
only 377M. I hadn't realized how much space the build output takes up.
So judging from du the 1.5G Guennadi Liakhovetski mentions above seems
to break down into something like:
330M .git
380M working tree
750M build output
Strange. Is my git 1.4.0 criminally broken? I have a clone of Linus' tree
on a USB disk on ext3 without any objects, which I just cloned at some
point and then did a couple of pulls from the same source. Now
1545084 /mnt/sda2/kernel-git/linux-2.6/
1255084 /mnt/sda2/kernel-git/linux-2.6/.git
Interestingly, both end up with 5084. For comparison:
465044 /mnt/sda2/kernel-git/powerpc
174980 /mnt/sda2/kernel-git/powerpc/.git
But that's a freshly cloned tree, without any pulls. I re-cloned it,
because the tree I had earlier had the problem with each pull:
Unpacking 12452 objects
100% (12452/12452) done
* refs/heads/origin: does not fast forward to branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc;
not updating.
Wonderful and strange git world...
Thanks
Guennadi
---
Guennadi Liakhovetski
Yes, but how do "advanced git users" kernel developers work? Do they just
do 1 clone and build / clean every time they want to test another
configuration / arch, or do they clone -l or what? Do they create branches
for each development thread, then pull / push between trees?...
I suspect it depends on the developer.
I end up just using different branches and switching between them, but
then, my branches tend to all be pretty small test-stuff (I only end up
using one main branch, since 99% of what I do is merge other peoples stuff
that has already gone through a test-cycle).
But I don't want to re-build. Apart from i386 I build for a couple of ARM
and PPC targets too...
You're probably fine with "git clone -l -s" then.
Strange. Is my git 1.4.0 criminally broken? I have a clone of Linus' tree
on a USB disk on ext3 without any objects, which I just cloned at some
point and then did a couple of pulls from the same source. Now
1545084 /mnt/sda2/kernel-git/linux-2.6/
1255084 /mnt/sda2/kernel-git/linux-2.6/.git
The old git that always exploded all pulls and generated lots of loose
objects? You can check with "git count-objects".
And to fix it, just do a "git gc" (or with older git versions, the secret
handshake is just a simple "git repack -a -d").
But that's a freshly cloned tree, without any pulls. I re-cloned it,
because the tree I had earlier had the problem with each pull:
Unpacking 12452 objects
100% (12452/12452) done
* refs/heads/origin: does not fast forward to branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc;
not updating.
Sounds like either Paul re-based his tree, or you did some work on your
"origin" branch..
Linus
On Fri, Mar 30, 2007 at 12:13:02AM +0200, Guennadi Liakhovetski wrote:
On Thu, 29 Mar 2007, J. Bruce Fields wrote:
quoted
Though actually on a second look, clone -l -s produces something that's
only 377M. I hadn't realized how much space the build output takes up.
So judging from du the 1.5G Guennadi Liakhovetski mentions above seems
to break down into something like:
330M .git
380M working tree
750M build output
Hmm.... That doesn't look right. My packed .git directory is 156 megs
(using post git 1.5 and repack.usedeltabaseoffset=true and
core.legacyheaders=false).
My working tree is 287M, and my build output (size of build tree minus
sources) is 1055M. (This will vary based on .config options, obviously).
The point though is the size of the .git tree is in the noise compared
to the size of the object files. I do share the .git repository
between trees; in fact I have a standard "base" repository which is
just a mirror of Linus's tree, and my other kernel repositories have
an .git/objects/info/alternates file which points at the base
repository for maximal sharing.
I could try to share the sources where the source files are identical,
but quite frankly, it's just never been worth the effort. After all,
when you have a 100gig laptop drive, 287 megs isn't that much, and
it's noise compared to the size of my object files.
Strange. Is my git 1.4.0 criminally broken? I have a clone of Linus' tree
on a USB disk on ext3 without any objects, which I just cloned at some
point and then did a couple of pulls from the same source.
You really, really, *REALLY* want to upgrade to at least git 1.5.0.
It's is *so* much better than git 1.5.0, and it's a lot easier to make
sure your repository is kept packed, using "git gc".
- Ted
From: J. Bruce Fields <hidden> Date: 2016-06-15 22:43:02
On Thu, Mar 29, 2007 at 10:43:27PM -0400, Theodore Tso wrote:
On Fri, Mar 30, 2007 at 12:13:02AM +0200, Guennadi Liakhovetski wrote:
quoted
On Thu, 29 Mar 2007, J. Bruce Fields wrote:
quoted
Though actually on a second look, clone -l -s produces something that's
only 377M. I hadn't realized how much space the build output takes up.
So judging from du the 1.5G Guennadi Liakhovetski mentions above seems
to break down into something like:
330M .git
380M working tree
750M build output
Hmm.... That doesn't look right. My packed .git directory is 156 megs
(using post git 1.5 and repack.usedeltabaseoffset=true and
core.legacyheaders=false).
I haven't run more than git-gc in a while, because I have local clones
and didn't want to figure out how to prune. Hm, now that I've looked it
up:
git prune -- $(cd ../linux-clone/ && echo $(git-rev-parse --all))
just gets me the git-prune usage message. In fact, contrary to the
prune man page, git-prune doesn't seem to accept any <head> arguments.
Isn't this a bug? I'm on 1.5.0.3.31.ge47c.
--b.
From: Andreas Herrmann <hidden> Date: 2016-06-15 22:43:02
On Fri, Mar 30, 2007 at 12:13:02AM +0200, Guennadi Liakhovetski wrote:
On Thu, 29 Mar 2007, J. Bruce Fields wrote:
quoted
On Thu, Mar 29, 2007 at 02:26:10PM -0700, Junio C Hamano wrote:
quoted
How about suggesting "clone -l -s"?
Yes, but how do "advanced git users" kernel developers work? Do they just
do 1 clone and build / clean every time they want to test another
configuration / arch, or do they clone -l or what? Do they create branches
for each development thread, then pull / push between trees?...
quoted
If you really want to share as much as possible, then I guess you want
to share the working trees too, since (as evidenced above), they're at
least as large as the compressed history.
But I don't want to re-build. Apart from i386 I build for a couple of ARM
and PPC targets too...
Seems to be trivial but:
Why don't you use "make O=/foo/bar/arch<x>-config<y>" to put output
files into separate directories? So you can have one source tree and
put each different kernel config and arch into a separate output
directory.
And if you have different sources for you trees put them into branches.
When switching between branches, atime of files are updated accordingly.
So even make should be happy with that.
Just one drawback:
Switching back and forth between two branches will cause
recompilation of sources that differ between that branches -
although nothing might have changed within a branch in the meantime.
(Not that I have used such an setup, yet.
But I think that should work.)
Regards,
Andreas
Strange. Is my git 1.4.0 criminally broken? I have a clone of Linus' tree
on a USB disk on ext3 without any objects, which I just cloned at some
point and then did a couple of pulls from the same source. Now
1545084 /mnt/sda2/kernel-git/linux-2.6/
1255084 /mnt/sda2/kernel-git/linux-2.6/.git
The old git that always exploded all pulls and generated lots of loose
objects? You can check with "git count-objects".
Installed 1.5.0.6 and its output of "git count-objects" is
180932 objects, 1112656 kilobytes
git gc removed everything (uh?) and then
linux-2.6$ du -ks .git
183040 .git
cool...
And to fix it, just do a "git gc" (or with older git versions, the secret
handshake is just a simple "git repack -a -d").
quoted
But that's a freshly cloned tree, without any pulls. I re-cloned it,
because the tree I had earlier had the problem with each pull:
Unpacking 12452 objects
100% (12452/12452) done
* refs/heads/origin: does not fast forward to branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc;
not updating.
Sounds like either Paul re-based his tree, or you did some work on your
"origin" branch..
It must be the former then:-) Did I have a chance to re-synchronize
locally to be able to pull normally again or was the only way to re-clone?
Thanks
Guennadi
---
Guennadi Liakhovetski
Seems to be trivial but:
Why don't you use "make O=/foo/bar/arch<x>-config<y>" to put output
files into separate directories? So you can have one source tree and
put each different kernel config and arch into a separate output
directory.
And if you have different sources for you trees put them into branches.
When switching between branches, atime of files are updated accordingly.
So even make should be happy with that.
Just one drawback:
Switching back and forth between two branches will cause
recompilation of sources that differ between that branches -
although nothing might have changed within a branch in the meantime.
Exactly, and since I have not only different configs, but also different
versions, and I don't commit all modifications, so... It would be
difficult. I think, the setup with "clone -l -s" should be the best.
Thanks to all for ideas
Guennadi
---
Guennadi Liakhovetski
Installed 1.5.0.6 and its output of "git count-objects" is
180932 objects, 1112656 kilobytes
Yeah, that is all the loose objects. That's 1.1GB of stuff as loose
objects, which also likely means that your tree was not only using up too
much disk-space, it was probably running about ten times slower than than
it needs to be for many operations.
Using pack-files ends up speeding up a lot of "big" operations a *lot*. If
all you do is look at individual commits, you'll probably not notice much,
but even something as simple as running "gitk" with no parameters (which
will thus traverse the whole history) should be a *lot* faster after it's
been all re-packed.
git gc removed everything (uh?) and then
Well, it will generally remove all loose objects, since they get put into
the pack-files instead. So it didn't remove "everything", but it does
remove everything that "git count-objects" normally counts (if you give
count-objects the "-v" flag for verbose output, it will talk about packed
objects too).
linux-2.6$ du -ks .git
183040 .git
cool...
That looks about right. Packing generally uses about a tenth of the
disk-space of loose objects - both from the use of deltas, and from the
fact that you don't any disk block fragmentation. And because looking at
objects then doesn't require any system calls any more (just the initial
"mmap pack and read index" stuff), it ends up being much faster too,
despite the added overhead of doing the whole delta-chain thing.
quoted
quoted
Unpacking 12452 objects
100% (12452/12452) done
* refs/heads/origin: does not fast forward to branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc;
not updating.
Sounds like either Paul re-based his tree, or you did some work on your
"origin" branch..
It must be the former then:-) Did I have a chance to re-synchronize
locally to be able to pull normally again or was the only way to re-clone?
You need to mark "refs/heads/origin" as always following the remote
"master", even if it gets re-based. You do that by adding a "+" before
the refspec that describes the remote.
These days, with git-1.5.x, "git clone" will do that for you (and make the
remotes fall under "refs/remotes/origin/*" instead - they're considered
separate branches from the local branches these days). However, since you
created the repository with an older git version, it still uses the
original format (and even though you upgraded your git binary, it will use
the old-fashioned branch format for remote branches and for
configuration).
So in your case, the remote is probably described by the
.git/remotes/origin file, and it looks something like
URL: git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
Pull: master:origin
and you should just change that "Pull:" line to have a "+" in front of the
refspec:
URL: git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
Pull: +master:origin
which tells git that the remote "master" branch should now
*unconditionally* be followed into the local "origin" branch when you
pull.
(In a more modern setup, you wouldn't have a .git/remotes/origin file at
all, insead you would have something like this in the .git/config file:
[remote "origin"]
url = git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
which means that there is a remote called "origin", and all of the remote
heads in that remote ("refs/heads/*") will be fetched into the local
repository under "refs/remotes/origin/*". And the "+" again means that we
do this even if it's not a fast-forward, ie we trust the remote
explicitly.
So "git fetch origin" will fetch *all* branches from "origin" into your
local repository, but to distinguish them from your *own* branches, they
will be under .git/refs/remotes/ instead of your "real" local branches
that are in ".git/refs/heads/. So you can now see the difference between
your *local* version of a branch X and the remote version of that same
branch by using "X" and "remotes/origin/X" respectively to describe that
branch.
Then, the second part of the above config file means that when you're on
the local "master" branch and do a "git pull", it will fetch it from the
remote "origin", and merge the remote "refs/heads/master" branch from that
remote (the same one that will be fetched into refs/remotes/origin/master
by a "git fetch".
Yeah, this is all a bit complex, and it takes a while to wrap your head
around it, but I have to say, once you do, the git-1.5.x layout really
*is* very powerful, and it's actually very natural too (but the "very
natural" part only comes after you have that "Aaahh!" moment!)
Linus
You need to mark "refs/heads/origin" as always following the remote
"master", even if it gets re-based. You do that by adding a "+" before
the refspec that describes the remote.
These days, with git-1.5.x, "git clone" will do that for you (and make the
remotes fall under "refs/remotes/origin/*" instead - they're considered
separate branches from the local branches these days). However, since you
created the repository with an older git version, it still uses the
original format (and even though you upgraded your git binary, it will use
the old-fashioned branch format for remote branches and for
configuration).
So in your case, the remote is probably described by the
.git/remotes/origin file, and it looks something like
URL: git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
Pull: master:origin
and you should just change that "Pull:" line to have a "+" in front of the
refspec:
URL: git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
Pull: +master:origin
which tells git that the remote "master" branch should now
*unconditionally* be followed into the local "origin" branch when you
pull.
(In a more modern setup, you wouldn't have a .git/remotes/origin file at
all, insead you would have something like this in the .git/config file:
[remote "origin"]
url = git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
which means that there is a remote called "origin", and all of the remote
heads in that remote ("refs/heads/*") will be fetched into the local
repository under "refs/remotes/origin/*". And the "+" again means that we
do this even if it's not a fast-forward, ie we trust the remote
explicitly.
So "git fetch origin" will fetch *all* branches from "origin" into your
local repository, but to distinguish them from your *own* branches, they
will be under .git/refs/remotes/ instead of your "real" local branches
that are in ".git/refs/heads/. So you can now see the difference between
your *local* version of a branch X and the remote version of that same
branch by using "X" and "remotes/origin/X" respectively to describe that
branch.
Then, the second part of the above config file means that when you're on
the local "master" branch and do a "git pull", it will fetch it from the
remote "origin", and merge the remote "refs/heads/master" branch from that
remote (the same one that will be fetched into refs/remotes/origin/master
by a "git fetch".
Yeah, this is all a bit complex, and it takes a while to wrap your head
around it, but I have to say, once you do, the git-1.5.x layout really
*is* very powerful, and it's actually very natural too (but the "very
natural" part only comes after you have that "Aaahh!" moment!)
Aha, so, that's how it is then! Why hasn't anybody explained this to me
strait away?!:-))))
Yeah, hopefully, I'll learn to at least use this thing efficiently enough.
Someone has to write a book on it though...
And, so, it's a pity I cloned Paul's tree yesterday with the "old" git.
And from your answer above it seems like some features of the "new" git
will not be available with this tree, like equally named local and remote
branches, etc. There isn't a way to convert such a "old style" tree to the
"new style", is there? Not a big deal, will re-clone at some point, maybe
when we get local git mirrors...
Many thanks for taking your time to answer, Linus!
Guennadi
---
Guennadi Liakhovetski
And, so, it's a pity I cloned Paul's tree yesterday with the "old" git.
And from your answer above it seems like some features of the "new" git
will not be available with this tree, like equally named local and remote
branches, etc. There isn't a way to convert such a "old style" tree to the
"new style", is there? Not a big deal, will re-clone at some point, maybe
when we get local git mirrors...
There's a conversion script to help you convert in place if you care.
Look at the git list for an email that looks something like this:
Date: Wed, 14 Mar 2007 02:16:12 -0400
From: Shawn O. Pearce [off-list ref]
To: git@vger.kernel.org
Subject: Upgrade to 1.5.0 utility
Yesterday on #git DrNick wanted a script to update a pre-1.5.0
Git repository to be like a 1.5.0 (and later) style repository.
...
which has a script in it to do this (it uses another script that is
already in git/contrib/ that just moves all the ".git/remotes" entries
as-is from the remotes files into the .git/config file)
I haven't tested it myself, so caveat emptor. But the config file format
really isn't *that* complicated - do the conversion with the script, and
then just go back and look at .git/config and verify that it looks sane,
or edit it to match your taste.
Linus
Yeah, hopefully, I'll learn to at least use this thing efficiently enough.
Someone has to write a book on it though...
The user-manual and the git tutorials are pretty good --- just make
sure you look at the ones that come with git 1.5.0, or the ones at
http://www.kernel.org/pub/software/svm/git/docs (which reflects the
Documentation as of the latest development version of git).
The problem is that there are some tutorials on the web which assume
git 1.4.x, or cogito, and that has been listed as a defect by people
who have been confused because those tutorials are out of date with
respect to modern git.[1]
[1] http://changelog.complete.org/posts/594-More-on-Git,-Mercurial,-and-Bzr.html
And, so, it's a pity I cloned Paul's tree yesterday with the "old" git.
And from your answer above it seems like some features of the "new" git
will not be available with this tree, like equally named local and remote
branches, etc. There isn't a way to convert such a "old style" tree to the
"new style", is there? Not a big deal, will re-clone at some point, maybe
when we get local git mirrors...
Yeah, we need to really make sure the word gets out that new git users
should really make sure they are using git 1.5, and not git 1.4.x; it
is such an improvement in terms of usability over git 1.4.x that there
really is no comparison. (Of course, guess what Debian is going to
ship in their soon-to-be-release "stable" distribution? git 1.4.4. Sigh...)
It is possible to get the new style naming, by using the git remote
command. Just do
git remote add origin <url>
...and that will adjust your git configuration file appropriate.
Check out the git 1.5.0 release notes, and the git-config man page,
and there are some other adjustments you can make as well that will
make git more efficiently, at the cost of breaking some level of
compatibility with git 1.4.x tools which replicate via the "dumb" http
protocol. But for local development repositories that generally isn't
a concern.
- Ted