GIT overlay repositories

Ok...

I started working on this a few weeks ago when it didn't appear 
anyone else had worked on support for this feature.  Since then, I
haven't kept pace with GIT development, so someone may have done 
something similar in the interim.  That said...

Start with two repositories, let's call them Repo-A and Repo-B.  Repo-A
is hosted on some server somewhere and contains lots of code 
(let's say its a kernel source repository).  Repo-B is only adding a 
small amount of changes to the repo (for argument sake, let's say the 
IPW2100 and IPW2200 projects) on top of what is already provided by Repo-A. 

For several reasons, we would like users to be able to get just the 
differences between Repo-A and Repo-B from me.

For example, the user gets the full Repo-A:

	rsync -avpr rsync://somewhere.com/repo-a/.git/objects/ \ 
		.git/objects/
	rsync -avpr rsync://somewhere.com/repo-a/.git/refs/heads/master \
		.git/refs/heads/parent

and then overlays just the delta, which they obtain from me:

	rsync -avpr rsync://myserver.com/repo-delta/.git/objects/ \
		.git/objects/
	rsync -avpr rsync://myserver.com/repo-delta/.git/refs/heads/master \
		.git/refs/heads/master

The user can then run:

	ln -s .git/HEAD refs/heads/master
	git-read-tree -m HEAD
	git-checkout-cache -a -q -f -u

And now their disk has the entire Repo-A + Repo-B object 
tree as if Repo-B had been entirely stored on myserver.com.

Adding this recursively is pretty straightforward if we introduce an 
'ancestors' file stored on the overlay repository that contains the URL 
and SHA of the tip of the parent when it was used to build the full 
repository.  For example:

% cat .git/refs/ancestors (edited to fit 80 columns)
rsync://rsync.kernel.org/...-2.6.git/#ieee80211	0c16...7f3
rsync://bughost.org/repos/ieee80211-delta/.git/ 5cb5...b84
rsync://bughost.org/repos/ipw-delta/.git/       f3ed...abe

A simple script (attached as 'git-grab-overlay') can then walk this 
ancestor file and pull down updates as needed from each of the parent 
repositories.

The process of creating the overlays is pretty straight forward (for 
simple cases anyway)

For example:

	A1   B  <-- HEAD SHA1
	|    |
	|    |
	| .->'  <-- parent SHA1
	|/     
	|
	A

In this state, the list of objects is easily obtained by walking the 
tree from HEAD back to the parent's SHA1 at the time the branch was 
made.  This can be done with something like: 

	git-rev-list --objects HEAD ^parent

To simplify things, I created the attached script 'git-make-overlay' 
which will create an overlay repository with for all objects needed 
between a given SHA1 and the HEAD of the repository:

	cd repository
	git-make-overlay ../target

This script will take the current HEAD and cached parent SHA (stored via 
the git-grab-overlay or manually when you create the intial tree) and 
create an overlay that you can then store on a remote server for others 
to pull.

Obviously you don't want the overlay repository to remain out of sync 
with the parent repository for too long.  The process of updating the 
full local repository is then a matter of:

  * Grabbing the latest parent objects
  * Caching the parent's latest master (store as new-parent)
  * Perform a merge of the child back into the master tree
  * Updating the cached heads (parent, master, etc.)

The merge is accomplished via:

	child=$(cat .git/HEAD)
	parent=$(cat .git/refs/heads/new-parent)
	base=$(git-merge-base $child $parent)
	# NOTE: base should be the same as .git/refs/heads/parent!

Is it best to merge the smaller subset back into the merge base, ie:

	git-read-tree -m $base $parent $child

In either case, the merge will fail if you have your local tree in a 
state consistent with child (vs. base).  So before you do the merge you 
need to either nuke the local tree or use paramters to the git- tools 
that I don't know about (definitely wouldn't suprise me).  When I run it, 
I create a temporary directory with a symlink of .git to the .git directory
where the merge will occur.  After the merge is done, I go back to the main
directory and resync the local file system with the GIT repository.

Anyway, I put together another simple script that attempts to do the 
above--minus the symlink hack (attached as git-merge-parent).  It invokes 
the git-merge-heads script which does some of what Cogito's cg-merge does, 
only with support for automatically launching an external merge conflict 
resolution program.  It relies on cogito's cg-Xmergefile so you need to 
have cogito to use git-merge-parent and git-merge-heads.
	
Frequently when remerging with a parent repository--especially one that 
may contain a version of the child within it, there will be merge 
conflicts.  The git-merge-parent accepts (well, requries) a MERGE 
environment variable to invoke as a merge conflict resolver.  If a 
conflict arrises, it will prompt if you want to resolve the merge, 
undo the update, or exit and allow you to finish the process manually.

If you chose to run the conflict editor, it will invoke it by providing 
three versions of the conflicting file:

1.  The base version (as captured in the tree identified 
    with git-merge-base on the parent and child) -- named FILE.base
2.  The parent version named FILE.parent
3.  The child version named FILE

After all merge conflicts have been manually updated, the 
git-merge-parent script will commit those changes, along with the rest 
of the merge, and move the child's HEAD to the new tree identity.  In 
addition it will cache the HEAD of the parent's merge.

I haven't really tested the whole merge extensiveloy; it has worked 
(more or less) with some simple parent updates I've done here.  However, 
it may be of use to some folks out there, and if others can take it 
forward beyond what I have, or merge the functionality into cogito (or 
wherever) great. 

I have a tree currently up that demonstrates the above in use.  
You can access it via:

  git-grab-overlay rsync://bughost.org/repos/ipw-delta/.git/ ipw-dev/

James