Hello,
I have an 'integration repo' which contains other git repos as submodules.
One of the submodules is to be split in two to extract a library.
A common way of doing that is to use git-filter-branch. A disadvantage
of that is that it results in duplicated partial-history in the
extracted repo. So, git log shows the entire history, but there is not
one canonical sha which represents the history at that point. The
split repo will contain 'false history', and checking it out will not
be useful.
So, I want to avoid git-filter-branch.
I have tried out using `git replace --graft` and
.git/objects/info/alternates to 'refer to' the history in the origin
repo instead of 'duplicating' it. This is similar to how Qt5 repos
refer to Qt 4 history in a different repo.
Question 1) Is this a reasonable thing to do for this scenario?
Question 2) Is there a way to push the `git replace` result and the
`info/alternates` content so that clients cloning the 'integration
repo' do not have to do that 'manually' or with a 'setup-repo.sh'
script?
The sequence of commands below can be pasted into a tmp directory to
see the scenario in action.
Thanks!
mkdir calculator
cd calculator
mkdir mainui libcalc
echo "print \"hello\"" > mainui/app.py
echo "print \"hello\"" > libcalc/adder.py
echo "print \"hello\"" > libcalc/subtracter.py
git init
git add .
git commit -am "Initial commit"
git checkout `git rev-parse HEAD`
cd ..
mkdir appsuite
cd appsuite
git init
git submodule add ../calculator
git commit -m "Add calculator submodule"
# Add other submodules in the suite...
cd calculator
echo "print \"goodbye\"" > libcalc/subtracter.py
git add libcalc/subtracter.py
git commit -am "Fix bug in subtracter"
echo "print \"Hi\"" > libcalc/adder.py
git add libcalc/adder.py
git commit -am "Make adder more efficient"
echo "print \"Hello, world!\"" > mainui/app.py
git add mainui/app.py
git commit -am "Improve app"
echo "print \"hello, hello\"" > libcalc/multiplier.py
git add libcalc/multiplier.py
git commit -am "Add multiplier"
cd ..
git add calculator
git commit -m "Update calculator submodule"
mkdir compute
cd calculator
mv libcalc ../compute
extraction_sha=`git rev-parse HEAD`
git commit -am "Remove libcalc from calculator repo" -m "It is moved
to a new compute repo"
removal_sha=`git rev-parse HEAD`
git push
cd ../compute
git init
git add .
git commit -m "Create the compute repo." -m "This commit will not be
normally visible after the replace --graft below."
echo "This is the compute framework. It contains the libcalc library." > README
git add README
git commit -m "Initialize the compute repo." -m "This has been
extracted from calculator.git at $removal_sha"
git checkout `git rev-parse HEAD`
cd ..
mv compute ..
git submodule add ../compute
git add calculator compute
git commit -m "Split compute framework out of calculator repo."
cd compute
git log --oneline
# We don't see older history from the calculator repo
# Let's add alternates
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
# ... and graft onto the extraction commit
git replace --graft HEAD $extraction_sha
git log --oneline
# Great, now we see history from the calculator repo.
cd ../..
git clone appsuite appsuite-clone
cd appsuite-clone
git submodule update --init
cd compute
ls ../.git/modules/compute/objects/info
git log --oneline
# The replacement and alternatives did not get cloned ... :(
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
git replace --graft HEAD $extraction_sha
# And now we see the history from the calculator repo. Great. But, it
required user action after the clone.
Hi,
On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly [off-list ref] wrote:
Hello,
I have an 'integration repo' which contains other git repos as submodules.
One of the submodules is to be split in two to extract a library.
A common way of doing that is to use git-filter-branch. A disadvantage
of that is that it results in duplicated partial-history in the
extracted repo. So, git log shows the entire history, but there is not
one canonical sha which represents the history at that point. The
split repo will contain 'false history', and checking it out will not
be useful.
So, I want to avoid git-filter-branch.
I have tried out using `git replace --graft` and
.git/objects/info/alternates to 'refer to' the history in the origin
repo instead of 'duplicating' it. This is similar to how Qt5 repos
refer to Qt 4 history in a different repo.
Question 1) Is this a reasonable thing to do for this scenario?
I think it should work without too much work, but see the answer to
the next question.
Question 2) Is there a way to push the `git replace` result and the
`info/alternates` content so that clients cloning the 'integration
repo' do not have to do that 'manually' or with a 'setup-repo.sh'
script?
In short no.
"git replace" creates replace refs in "refs/replace/". To fetch these
refs, people need to use either "git fetch
'refs/replace/*:refs/replace/*'" or add a "fetch =
refs/replace/*:refs/replace/*" line in their config.
For simplicity and security reasons fetching replace refs is not on by default.
Also changing the objects/info/alternates to make it point to another
repo cannot be done automatically when cloning.
The sequence of commands below can be pasted into a tmp directory to
see the scenario in action.
Thanks!
mkdir calculator
cd calculator
mkdir mainui libcalc
echo "print \"hello\"" > mainui/app.py
echo "print \"hello\"" > libcalc/adder.py
echo "print \"hello\"" > libcalc/subtracter.py
git init
git add .
git commit -am "Initial commit"
git checkout `git rev-parse HEAD`
cd ..
mkdir appsuite
cd appsuite
git init
git submodule add ../calculator
git commit -m "Add calculator submodule"
# Add other submodules in the suite...
cd calculator
echo "print \"goodbye\"" > libcalc/subtracter.py
git add libcalc/subtracter.py
git commit -am "Fix bug in subtracter"
echo "print \"Hi\"" > libcalc/adder.py
git add libcalc/adder.py
git commit -am "Make adder more efficient"
echo "print \"Hello, world!\"" > mainui/app.py
git add mainui/app.py
git commit -am "Improve app"
echo "print \"hello, hello\"" > libcalc/multiplier.py
git add libcalc/multiplier.py
git commit -am "Add multiplier"
cd ..
git add calculator
git commit -m "Update calculator submodule"
mkdir compute
cd calculator
mv libcalc ../compute
extraction_sha=`git rev-parse HEAD`
git commit -am "Remove libcalc from calculator repo" -m "It is moved
to a new compute repo"
removal_sha=`git rev-parse HEAD`
git push
cd ../compute
git init
git add .
git commit -m "Create the compute repo." -m "This commit will not be
normally visible after the replace --graft below."
echo "This is the compute framework. It contains the libcalc library." > README
git add README
git commit -m "Initialize the compute repo." -m "This has been
extracted from calculator.git at $removal_sha"
git checkout `git rev-parse HEAD`
cd ..
mv compute ..
git submodule add ../compute
git add calculator compute
git commit -m "Split compute framework out of calculator repo."
cd compute
git log --oneline
# We don't see older history from the calculator repo
# Let's add alternates
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
# ... and graft onto the extraction commit
git replace --graft HEAD $extraction_sha
git log --oneline
# Great, now we see history from the calculator repo.
cd ../..
git clone appsuite appsuite-clone
cd appsuite-clone
git submodule update --init
cd compute
ls ../.git/modules/compute/objects/info
git log --oneline
# The replacement and alternatives did not get cloned ... :(
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
git replace --graft HEAD $extraction_sha
Maybe use the following instead of the above line:
git fetch 'refs/replace/*:refs/replace/*'
# And now we see the history from the calculator repo. Great. But, it
required user action after the clone.
Yeah, but if the 2 above commands are in a script maybe it's
reasonable to ask the user to launch the script once after cloning.
Best,
Christian.
On 05/24/2015 07:28 AM, Christian Couder wrote:
Hi,
On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly [off-list ref] wrote:
quoted
I have tried out using `git replace --graft` and
.git/objects/info/alternates to 'refer to' the history in the origin
repo instead of 'duplicating' it. This is similar to how Qt5 repos
refer to Qt 4 history in a different repo.
Question 1) Is this a reasonable thing to do for this scenario?
I think it should work without too much work, but see the answer to
the next question.
Ok, thanks. The concern is that there is plenty of documentation for
git-filter-branch, but no documentation or porcelain for info/alternates
and little out on the internet about it or git replace and using them
together.
However, it seems to be a reasonable thing to do.
quoted
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
git replace --graft HEAD $extraction_sha
Maybe use the following instead of the above line:
git fetch 'refs/replace/*:refs/replace/*'
Thanks.
quoted
# And now we see the history from the calculator repo. Great. But, it
required user action after the clone.
Yeah, but if the 2 above commands are in a script maybe it's
reasonable to ask the user to launch the script once after cloning.
Would it be possible to do this in a hook in the 'integration repo'
which contains both submodules in the example I posted? Like a fetch
hook or something?
Thanks,
Steve.
On Mon, May 25, 2015 at 11:49 AM, Stephen Kelly [off-list ref] wrote:
On 05/24/2015 07:28 AM, Christian Couder wrote:
quoted
Hi,
On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly [off-list ref] wrote:
quoted
I have tried out using `git replace --graft` and
.git/objects/info/alternates to 'refer to' the history in the origin
repo instead of 'duplicating' it. This is similar to how Qt5 repos
refer to Qt 4 history in a different repo.
Question 1) Is this a reasonable thing to do for this scenario?
I think it should work without too much work, but see the answer to
the next question.
Ok, thanks. The concern is that there is plenty of documentation for
git-filter-branch, but no documentation or porcelain for info/alternates
and little out on the internet about it or git replace and using them
together.
However, it seems to be a reasonable thing to do.
Yeah.
By the way it looks like I was wrong in my answer to your second question.
You might want to clone using the --reference option to automatically
set up .git/objects/info/alternates properly.
(There is an ongoing related thread on the list:
http://thread.gmane.org/gmane.comp.version-control.git/269537)
You might also be able to clone using an option like "--config
remote.origin.fetch = 'refs/replace/*:refs/replace/*'" to fetch the
replace ref when cloning.
So it looks like you might just need to clone with a few more options
than usual.
I haven't tested it so please tell me if it works :-)
quoted
quoted
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
git replace --graft HEAD $extraction_sha
Maybe use the following instead of the above line:
git fetch 'refs/replace/*:refs/replace/*'
Thanks.
quoted
quoted
# And now we see the history from the calculator repo. Great. But, it
required user action after the clone.
Yeah, but if the 2 above commands are in a script maybe it's
reasonable to ask the user to launch the script once after cloning.
Would it be possible to do this in a hook in the 'integration repo'
which contains both submodules in the example I posted? Like a fetch
hook or something?
It is possible to do whatever you want in a hook, but the question is
why would you do it in a hook as it looks like it needs to be done
only once?
Or maybe I am missing something?
Best,
Christian.