git-union-merge proposal

Lately some tools are storing data in git branches or refs, that is not
source code, and that is designed in some way to be automatically
merged. Generally merge=union will work for it, but the problem is that
git-merge can only operate on the checked out branch, not other
branches.

So these things all deal with merging in their own ad-hoc ways:

* pristine-tar commits the info it needs to reconstruct tarballs
  to a pristine-tar branch; files in the branch should not easily conflict
  as each includes the name of the tarball.. but when git pull
  cannot fast-forward the pristine-tar branch, the user is left to
  manually fix it.
* git-annex stores location tracking information to log files in
  .git-annex/; gitattributes is configured to use merge=union,
  and the log files have timestamps or are otherwise structured to be
  safely merged.
* git notes merge -s cat_sort_uniq
  Notes are stored in a tree using the object sha, which can be
  union merged, when the notes' format is a series of independant lines.
* probably other tools do things like this too, or will ...

So I've written a prototype of a git-union-merge that could be used
for all of these. It works like this:

git union-merge foo origin/foo refs/heads/foo 

That looks up foo and origin/foo and union merges them together,
producing the new branch refs/heads/foo. New blobs are injected
as needed for unioned files, and the merge commit is generated,
without affecting the current working tree, and without any
expensive checkouts of the branches. It's pretty fast, it only
needs to write out a temporary index file.

Prototype is attached, I doubt it would be suitable for git as-is,
but it does show how this is accomplished, if you've not already
seen how to do it -- just look for ls-tree, diff-tree,
show, hash-object, and update-index. Note that merging file modes is
not yet dealt with.

I imagine a git that can have union merge or other custom automated
merge strategies configured on a per-branch basis, so that git pull
automatically merges branches. That could be a good basis for adding
Fossil-like features to git.

-- 
see shy jo

Hi Joey,

Joey Hess wrote[1]:

* git-annex stores location tracking information to log files in
  .git-annex/; gitattributes is configured to use merge=union,
  and the log files have timestamps or are otherwise structured to be
  safely merged.
* git notes merge -s cat_sort_uniq
  Notes are stored in a tree using the object sha, which can be
  union merged, when the notes' format is a series of independant lines.
* probably other tools do things like this too, or will ...

So I've written a prototype of a git-union-merge that could be used
for all of these. It works like this:

git union-merge foo origin/foo refs/heads/foo 

Hm, this makes a lot of sense.  Often a person needs a worktree anyway
to check the merge result for sanity, but as you say, that needn't
always be the case.

[...]

Prototype is attached, I doubt it would be suitable for git as-is,

Does the GitRepo module that it uses come from git-annex?

If the prototype were self-contained, I would encourage you to submit
it for inclusion under contrib/ so it can evolve and eventually
graduate out of there.  Cc-ing Johan (who has no doubt thought through
these things in the context of "git notes") in case he has thoughts on
it.

Regards,
Jonathan

[1] http://thread.gmane.org/gmane.comp.version-control.git/176119

On Tuesday 21 June 2011, Jonathan Nieder wrote:

Joey Hess wrote[1]:

quoted

* git notes merge -s cat_sort_uniq

  Notes are stored in a tree using the object sha, which can be
  union merged, when the notes' format is a series of independant
  lines.

[...]

So I've written a prototype of a git-union-merge that could be used
for all of these. It works like this:

git union-merge foo origin/foo refs/heads/foo

[...]

If the prototype were self-contained, I would encourage you to submit
it for inclusion under contrib/ so it can evolve and eventually
graduate out of there.  Cc-ing Johan (who has no doubt thought through
these things in the context of "git notes") in case he has thoughts on
it.

Thanks for the CC.

I must confess that my Haskell skills are exactly nil, but AFAICS the script 
depends on the filename as the only criteria to identify files that need a 
line-level merge. How does the script deal with renamed and copied files?

If you depend on the filename only, this script simply will not work for 
notes. The notes tree reorganizes itself dynamically for optimum 
performance, and this affects how notes trees can be merged.

E.g. given a note for object 01234567..., this note may exist as 
"01234567..." in one notes tree, while it may exist as "01/234567..." in a 
bigger notes tree, or even "01/23/4567..." in an even bigger notes tree. 
Even though the filenames differ, they all refer to the same note, and you 
cannot merge notes trees correctly without taking that fact into account.

Furthermore, if you (union-)merge two notes trees that both have 
"01/234567...", the result does not necessarily belong in "01/234567...". It 
could be that the sum/union of the two notes trees have pushed the number of 
notes in the result so high that "01/23/4567..." is now a more optimal name 
for this note.


...Johan

-- 
Johan Herland, [off-list ref]
www.herland.net

Jonathan Nieder wrote:

Does the GitRepo module that it uses come from git-annex?

If the prototype were self-contained, I would encourage you to submit
it for inclusion under contrib/ so it can evolve and eventually
graduate out of there.  Cc-ing Johan (who has no doubt thought through
these things in the context of "git notes") in case he has thoughts on
it.

Yes, this was written in the context of git-annex. I would probably not want
to submit the haskell implementation to contrib/, but a shell implementation
could be done that would be perhaps less robust, but also less unusual in
the context of git's code base.

Johan Herland wrote:

I must confess that my Haskell skills are exactly nil, but AFAICS the script 
depends on the filename as the only criteria to identify files that need a 
line-level merge. How does the script deal with renamed and copied files?

If you depend on the filename only, this script simply will not work for 
notes.

It simply depends on filenames. I saw there was additional complexity
in notes and I don't see how a general purpose merger can handle that.
(I wish I could just use notes for my application, but I have data that
is not tied to any one object in git.)

Although, this is an obvious extension that would add some flexability
to handle for files that cannot be merged with a naive union:

git union-merge foo origin/foo refs/heads/foo -c "sort * | uniq"

Where the files would be passed in as temp files.

Hmm, that makes it look not unlike git-filter-branch, except
it's generating a new commit at the tip. I *think* that filter-branch
can't do this.

-- 
see shy jo