From: Olaf Hering <hidden> Date: 2016-06-15 23:02:55
How can I reduce the disk usage for multiple copies of the same repo?
Up to now I just made copies like this, but since .git alone is already
2GB it becomes expensive:
# git clone git://host/repo.git repo-master
# cp -a repo-master repo-branchA
# cd repo-branchA
# git checkout -b branchA origin/branchA
# cd -
# cp -a repo-master repo-branchB
# cd repo-branchB
# git checkout -b branchB origin/branchB
# cd -
# cp -a repo-master repo-branchB-feature
# cd repo-branchB-feature
# git checkout -b branchB-feature origin/branchB
# cd -
Since each .git is almost identical I wonder if there is a reliable way
to "share" it. The "git clone" man page mentions --shared as a dangerous
way to do things. It does not give an advice how to manage such cloned
trees.
So how can I reduce the disk usage needed for the four .git dirs above?
I looked around in the docs that came with my git-2.1.3 package, but
found nothing that answers my question. Maybe we can workout something
and add it to one of the existing docs.
Thanks!
Olaf
From: Fredrik Gustafsson <hidden> Date: 2016-06-15 23:02:55
On Thu, Nov 13, 2014 at 12:14:44PM +0100, Olaf Hering wrote:
How can I reduce the disk usage for multiple copies of the same repo?
You can use --local och --shared. As you say --shared can be dangerous.
If you don't understand the man page enough to know how you should
manage your clones you should probably not use it.
--local seems to be what you're looking for.
However as a side note I'm curious about what your use case is. Why do
you need this many repos?
Your setup looks familiar to me for a subversion user switching to git
and trying to use git as subversion. The common usecase is not to have
multiple worktrees but to do a checkout to the worktree you need to work
on. This is possible with git since it's very fast and I recommend you
to try to use one worktree.
--
Med vänlig hälsning
Fredrik Gustafsson
tel: 0733-608274
e-post: iveqy@iveqy.com
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:55
On Thu, Nov 13, Fredrik Gustafsson wrote:
On Thu, Nov 13, 2014 at 12:14:44PM +0100, Olaf Hering wrote:
quoted
How can I reduce the disk usage for multiple copies of the same repo?
You can use --local och --shared. As you say --shared can be dangerous.
If you don't understand the man page enough to know how you should
manage your clones you should probably not use it.
Perhaps its more a lack of doc how to use the result.
--local seems to be what you're looking for.
I will try this.
However as a side note I'm curious about what your use case is. Why do
you need this many repos?
Because I do work in each copy, poke around, or do commits and push
them.
Your setup looks familiar to me for a subversion user switching to git
and trying to use git as subversion. The common usecase is not to have
multiple worktrees but to do a checkout to the worktree you need to work
on. This is possible with git since it's very fast and I recommend you
to try to use one worktree.
Switching branches will invalidate timestamps, causing a full rebuild.
Olaf
On Thu, Nov 13, 2014 at 6:14 PM, Olaf Hering [off-list ref] wrote:
Since each .git is almost identical I wonder if there is a reliable way
to "share" it. The "git clone" man page mentions --shared as a dangerous
way to do things. It does not give an advice how to manage such cloned
trees.
If you know what you are doing, you can try git-new-workdir in
contrib/workdir. A safe and reliable version of that is being worked
on, hopefully it'll be released in 2.3.0.
--
Duy
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:55
On Thu, Nov 13, Roger Gammans wrote:
Note the first sentence of the second paragraph.
eg:
# git clone git://host/repo.git repo-master
# git clone repo-master repo-branchA
# cd repo-branchA
# git checkout -b branchA origin/branchA
It fails right here because in this dir only "master" exists, but
branchA is expected.
So far the sequence of commands is:
# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB
Next step will be:
# $do_work ; git commit -avs ; git push
Will that work as expected? Will find out after lunch..
Olaf
From: Roger Gammans <hidden> Date: 2016-06-15 23:02:55
On Thu, 2014-11-13 at 12:14 +0100, Olaf Hering wrote:
How can I reduce the disk usage for multiple copies of the same repo?
Up to now I just made copies like this, but since .git alone is already
2GB it becomes expensive:
# git clone git://host/repo.git repo-master
# cp -a repo-master repo-branchA
# cd repo-branchA
# git checkout -b branchA origin/branchA
# cd -
# cp -a repo-master repo-branchB
# cd repo-branchB
# git checkout -b branchB origin/branchB
# cd -
# cp -a repo-master repo-branchB-feature
# cd repo-branchB-feature
# git checkout -b branchB-feature origin/branchB
# cd -
Since each .git is almost identical I wonder if there is a reliable way
to "share" it. The "git clone" man page mentions --shared as a dangerous
way to do things. It does not give an advice how to manage such cloned
trees.
But you're not using clone you are using cp .
The clone man page also says this:-
--local, -l
When the repository to clone from is on a local machine, this flag bypasses the normal "Git aware" transport
mechanism and clones the repository by making a copy of HEAD and everything under objects and refs directories. The
files under .git/objects/ directory are hardlinked to save space when possible.
If the repository is specified as a local path (e.g., /path/to/repo), this is the default, and --local is essentially
a no-op. If the repository is specified as a URL, then this flag is ignored (and we never use the local
optimizations). Specifying --no-local will override the default when /path/to/repo is given, using the regular Git
transport instead.
Note the first sentence of the second paragraph.
eg:
# git clone git://host/repo.git repo-master
# git clone repo-master repo-branchA
# cd repo-branchA
# git checkout -b branchA origin/branchA
# cd -
# git clone repo-master repo-branchB
# cd repo-branchB
# git checkout -b branchB origin/branchB
# cd -
# git clone repo-master repo-branchB-feature
# cd repo-branchB-feature
# git checkout -b branchB-feature origin/branchB
# cd -
Should work better for you. And there is probably a way to do it less
commands too.
--
Roger Gammans [off-list ref]
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:55
On Thu, Nov 13, Olaf Hering wrote:
So how can I reduce the disk usage needed for the four .git dirs above?
I looked around in the docs that came with my git-2.1.3 package, but
found nothing that answers my question. Maybe we can workout something
and add it to one of the existing docs.
While playing around with this I made some notes, this is the result:
Manage multiple branches as separate copies
To preserve disk space for each clone, use a master copy of the reop and do
local clones from such copy of a remote repository.
First clone the remote repository as usual. Then create a local branch for
each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
Now clone each work branch into its own directory. The work dir references the
master repo. All changes come from and go into this repo, instead of the
remote repo.
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB
To make changs in a work dir, commit as usual. The changes will be pushed from
the work copy into the local master repo. Its required to have some other
branch than branchA active in repo-master, or push from work copy to
repo-master will fail.
# cd repo-master
# git checkout master
# cd -
# cd repo-branchA
# git commit -avs
# git push origin branchA
# cd -
To publish the outstanding changes its required to do this from the master
repo. First checkout the work branch, then pull the local changes and finally
push them to the remote repo.
# cd repo-master
# git checkout branchA
# git pull
# git push origin branchA
# cd -
To receive changes from the remote repo its required to do this from the
master repo. First checkout the work branch, then pull the outstanding remote
changes into the local branch. And finally pull them into the work dir.
# cd repo-master
# git fetch --all (optional)
# git checkout branchB
# git pull
# cd -
# cd repo-branchB
# git pull
# cd -
# vim: set tw=72 et
From: Fredrik Gustafsson <hidden> Date: 2016-06-15 23:02:55
Thanks for sharing your notes! A few comments:
On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
First clone the remote repository as usual. Then create a local branch for
each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
Now clone each work branch into its own directory. The work dir references the
master repo. All changes come from and go into this repo, instead of the
remote repo.
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB
To make changs in a work dir, commit as usual. The changes will be pushed from
the work copy into the local master repo. Its required to have some other
branch than branchA active in repo-master, or push from work copy to
repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare
repository. You should clone your repo-master with the --bare option to
avoid having a work dir there.
To publish the outstanding changes its required to do this from the master
repo. First checkout the work branch, then pull the local changes and finally
push them to the remote repo.
# cd repo-master
# git checkout branchA
# git pull
# git push origin branchA
# cd -
It's not. You could just add your remote repository as a remote to each
of your clones of your master repo and push directly from them. It
would be much simplier and it would allow you to directly fetch changes
from your remote into your branches as well.
(however, I'm not sure but I think, that this will slowly increase the
difference between your repositories when you develop. So that they
won't change any new data since to local clone was made).
--
Med vänlig hälsning
Fredrik Gustafsson
tel: 0733-608274
e-post: iveqy@iveqy.com
From: Johan Herland <hidden> Date: 2016-06-15 23:02:55
On Thu, Nov 13, 2014 at 5:03 PM, Fredrik Gustafsson [off-list ref] wrote:
Thanks for sharing your notes! A few comments:
On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
quoted
First clone the remote repository as usual. Then create a local branch for
each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
Now clone each work branch into its own directory. The work dir references the
master repo. All changes come from and go into this repo, instead of the
remote repo.
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB
To make changs in a work dir, commit as usual. The changes will be pushed from
the work copy into the local master repo. Its required to have some other
branch than branchA active in repo-master, or push from work copy to
repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare
repository. You should clone your repo-master with the --bare option to
avoid having a work dir there.
quoted
To publish the outstanding changes its required to do this from the master
repo. First checkout the work branch, then pull the local changes and finally
push them to the remote repo.
# cd repo-master
# git checkout branchA
# git pull
# git push origin branchA
# cd -
It's not. You could just add your remote repository as a remote to each
of your clones of your master repo and push directly from them. It
would be much simplier and it would allow you to directly fetch changes
from your remote into your branches as well.
(however, I'm not sure but I think, that this will slowly increase the
difference between your repositories when you develop. So that they
won't change any new data since to local clone was made).
Can you not do this much simpler with --reference? Like this:
$ git clone --bare git://host/repo.git repo-master
$ git clone -b branchA --reference repo-master git://host/repo.git
repo-branchA
$ git clone -b branchB --reference repo-master git://host/repo.git
repo-branchB
All three repos now push/pull directly to/from git://host/repo.git,
but repo-branchA and repo-branchB reference objects from within the
bare repo-master. You have to make use to never delete objects from
repo-master (if those objects happen to be referenced from
repo-branchA|B). If you want to prevent the repos growing in size, you
must devise a way to add new objects into repo-master before
repo-branchA|B. (e.g. a nightly cron-job in repo-master that fetches
from origin), so that when repo-branchA|B pulls, they will find most
objects are already present in repo-master.
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: Jeff King <hidden> Date: 2016-06-15 23:02:56
On Thu, Nov 13, 2014 at 05:08:19PM +0100, Johan Herland wrote:
Can you not do this much simpler with --reference? Like this:
$ git clone --bare git://host/repo.git repo-master
$ git clone -b branchA --reference repo-master git://host/repo.git
repo-branchA
$ git clone -b branchB --reference repo-master git://host/repo.git
repo-branchB
All three repos now push/pull directly to/from git://host/repo.git,
but repo-branchA and repo-branchB reference objects from within the
bare repo-master. You have to make use to never delete objects from
repo-master
I think the "never delete" part is why we usually warn people off of
using alternates. I think at the least you would have to "git config
gc.auto 0" in the bare repository (otherwise your nightly fetches risk
pruning). Of course you'd probably want to repack eventually for
performance reasons. So maybe setting gc.pruneExpire is a better option
(to something like "20.years.ago").
If you want to prevent the repos growing in size, you must devise a
way to add new objects into repo-master before repo-branchA|B. (e.g. a
nightly cron-job in repo-master that fetches from origin), so that
when repo-branchA|B pulls, they will find most objects are already
present in repo-master.
You can also fetch from the children into repo-master periodically.
Like:
cd repo-master &&
for i in branchA branchB; do
git fetch ../$i +refs/*:refs/remotes/$i/*
done
after which it is actually safe to run "git gc" in the master (assuming
there isn't simultaneous activity in the children). This is how we
manage fork networks on GitHub (we take in objects to individual forks
via push, and then migrate them to the master repo via fetch).
-Peff
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:56
On Thu, Nov 13, Fredrik Gustafsson wrote:
Thanks for sharing your notes! A few comments:
On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
quoted
First clone the remote repository as usual. Then create a local branch for
each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
Now clone each work branch into its own directory. The work dir references the
master repo. All changes come from and go into this repo, instead of the
remote repo.
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB
To make changs in a work dir, commit as usual. The changes will be pushed from
the work copy into the local master repo. Its required to have some other
branch than branchA active in repo-master, or push from work copy to
repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare
repository. You should clone your repo-master with the --bare option to
avoid having a work dir there.
So my repo-master is now "bare". I pushed from repo-branchA into
repo-master and see my commits in both repos. But pushing from
repo-master to the remote fails because repo-master does not have
outstanding remote commits. However, git fetch doesnt do anything:
Fetching origin
From host:/remote/dir
* branch HEAD -> FETCH_HEAD
Obviously I miss something. The man page of git clone or fetch does not
mention how "bare" is supposed to be handled.
Olaf
From: Fredrik Gustafsson <hidden> Date: 2016-06-15 23:02:56
On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
So my repo-master is now "bare". I pushed from repo-branchA into
repo-master and see my commits in both repos. But pushing from
repo-master to the remote fails because repo-master does not have
outstanding remote commits. However, git fetch doesnt do anything:
Are you mixing up your branches? So that you're updating one branch in
your master repo but trying to push an other branch to your remote repo?
--
Med vänlig hälsning
Fredrik Gustafsson
tel: 0733-608274
e-post: iveqy@iveqy.com
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:56
On Fri, Nov 14, Fredrik Gustafsson wrote:
On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
quoted
So my repo-master is now "bare". I pushed from repo-branchA into
repo-master and see my commits in both repos. But pushing from
repo-master to the remote fails because repo-master does not have
outstanding remote commits. However, git fetch doesnt do anything:
Are you mixing up your branches? So that you're updating one branch in
your master repo but trying to push an other branch to your remote repo?
I dont think so. I have branchA in repo-branchA, and a 'git push origin
branchA' puts it into repo-master.
How is a bare repo supposed to be updated? Is a simple 'git fetch --all'
supposed to work?
Olaf
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:56
On Fri, Nov 14, Olaf Hering wrote:
On Fri, Nov 14, Fredrik Gustafsson wrote:
quoted
On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
quoted
So my repo-master is now "bare". I pushed from repo-branchA into
repo-master and see my commits in both repos. But pushing from
repo-master to the remote fails because repo-master does not have
outstanding remote commits. However, git fetch doesnt do anything:
Are you mixing up your branches? So that you're updating one branch in
your master repo but trying to push an other branch to your remote repo?
I dont think so. I have branchA in repo-branchA, and a 'git push origin
branchA' puts it into repo-master.
How is a bare repo supposed to be updated? Is a simple 'git fetch --all'
supposed to work?
Is there s a slim chance that I have to fetch in repo-master before
doing a git push in repo-branchA? git remote show origin shows that some
branches are out of date.
Olaf
From: Olaf Hering <hidden> Date: 2016-06-15 23:02:56
On Thu, Nov 13, Fredrik Gustafsson wrote:
That's one of the reason it's not recommended to push into a non-bare
repository. You should clone your repo-master with the --bare option to
avoid having a work dir there.
Even if I do a fresh clone with --bare, the result can not be updated
anymore with git fetch. What I'm doing wrong?
Olaf
From: Jakub Narębski <hidden> Date: 2016-06-15 23:02:56
W dniu 2014-11-13 13:03, Olaf Hering pisze:
> On Thu, Nov 13, Fredrik Gustafsson wrote:
[...]
quoted
Your setup looks familiar to me for a subversion user switching to git
and trying to use git as subversion. The common usecase is not to have
multiple worktrees but to do a checkout to the worktree you need to work
on. This is possible with git since it's very fast and I recommend you
to try to use one worktree.
Switching branches will invalidate timestamps, causing a full rebuild.
Wouldn't a better way of solving "full rebuild" issue be to use ccache
or similar solution?
Anyway, switching branches invalidates timestamps only on those files
that change between branches -- it is to avoid unnecessary rebuilds.
You can always clone with --reference, and use alternates (alternate
object store). Just don't delete objects in repository that other
repositories borrow from; GitHub uses refs/borrowers/ namespace for
that, IIRC.
HTH
--
Jakub Narębski