Storing state in $GIT_DIR

Is there a convention of where/how it is safe to store additional
(non-git) data in $GIT_DIR?

The arch import needs to keep a cache with arch-commit-id  =
git-commit-id mappings, and some notes about what patch-trading Arch
recorded. It'd be great to be able to store those in
$GIT_DIR/archimport/ . Is that supported?

It does not need to be replicated with push or pull, merely preserved. 

cheers,


martin

[ Junio, the fact that you can't script the initial commit with "git 
  commit" is _really_ irritating. ]

On Thu, 25 Aug 2005, Martin Langhoff wrote:

Is there a convention of where/how it is safe to store additional
(non-git) data in $GIT_DIR?

There's no convention, but I have a suggestion.

The arch import needs to keep a cache with arch-commit-id  =
git-commit-id mappings, and some notes about what patch-trading Arch
recorded. It'd be great to be able to store those in
$GIT_DIR/archimport/ . Is that supported?

Git won't care, so it will work, but things like clone/pull etc also won't
actually ever look there, so it will only work for that one repo.

Now, I have what I consider a clever idea (I've mentioned variations on it 
before), but it's entirely possible that people hate it.

The thing is, I think you _do_ want to revision-control the git-commit-id
mappings, but at the same time, you do _not_ want to mess up the resulting
git commit history with arch information.

The reason you want to revision-control them is that that way you get them 
on clones, and you can use push/pull to update them. And the reason you 
don't want to mess up the commit history is that it's just wrong and ugly.

The git solution to this (which nobody has ever _used_, but which
technically is wonderful) is to have a "side branch" that does not share
any commits (or files, for that matter) in common with the "real branch",
and which is used to track any metadata. In fact, you can obviously have 
any number of side branches.

So that "metadata branch" is a real git branch in its own right, but it
doesn't share the same root as the "normal" branch, and it's really
totally independent: you can pull just the main branch (ie somebody who
isn't arch-aware and has no reason to want the arch mappings), or you
could pull just the metadata branch (for example, somebody who doesn't
want to use git, but is trying to match up a git commit ID to whatever
ID's arch uses).

The way to maintain a metadata branch is to have not only a different 
branch name (obviously), but also use a totally different index file, so 
that you can index both branches in parallell, and you don't actually need 
to check out one or the other.

Now, your arch import tools would then use the raw git commands explicitly 
to maintain the metadata branch. Every time you do an incremental import 
from an arch project, your import scripts would save away the mapping 
information into the metadata branch.

I'll make a _really_ stupid example for you, just to make this a bit more 
concrete:

	mkdir silly-example
	cd silly-example

	#
	# The normal "main branch": use regular git
	# infrastructure
	#
	git init-db
	echo "Hello" > file
	git update-cache --add file
	git commit -m "Main branch"

	#
	# The metadata branch: magic, very special stuff
	#
	echo "initial commit:" $(git-rev-parse HEAD) > .archdata
	GIT_INDEX_FILE=.git/archindex git-update-cache --add .archdata
	arch_index_tree=$(GIT_INDEX_FILE=.git/archindex git-write-tree)
	echo "arch index" | git-commit-tree $arch_index_tree > .git/refs/heads/arch-index

(Btw, the above example shows that the initial "git commit" won't take a 
"-m" flag, which is really irritating for scripts.)

Then do a "gitk --all", see the two different branches, and realize that 
the "arch-index" branch can now contain all the tracking information 
necessary to go back-and-forth. 

		Linus

On Fri, 26 Aug 2005, Martin Langhoff wrote:

OTOH, storing the metadata in a branch will allow us to run the import
in alternating repositories. But as Junio points out, unless I can
guarantee that the metadata and the tree are in sync, I cannot
trivially resume the import cycle from a new repo.

But you can.

Remember: the metadata is the pointers to the original git conversion, and 
objects are immutable.

In other words, if you just have a "last commit" pointer in your 
meta-data, then git is _by_definition_ in sync. There's never anything to 
get out of sync, because objects aren't going to change.

So you can think of your meta-data as a strange kind of head ref. Or 
rather, a _collection_ of these strange refs.

And it doesn't matter if somebody ends up committing on top of an arch 
import. The metadata by definition doesn't know about it, so the "import" 
head doesn't move anywhere (if you do git and arch work in parallell, you 
can then merge the two heads with git, of course).

			Linus

Linus, 

I like the solution you are suggesting, but I suspect it will create
more problems that it will solve, and while the coolness factor is
drawing me in.... we ain't gonna need it, as the xp people say.

More below...

On 8/26/05, Linus Torvalds [off-list ref] wrote:

Git won't care, so it will work, but things like clone/pull etc also won't
actually ever look there, so it will only work for that one repo.

Storing things there _works_ in the sense that it will be ignored, and
that is fine with me. So I could just be lazy and have it strictly
tied to the repo. In practice, if you are tracking an external Arch
repo, you really have it scripted, and use a dedicated git repo for
that.

Not using a dedicated repo is quite... messy. If you do other things
in that particular repo, the import script may find it dirty, and mess
things up on import. And after the import, you'll probably run
git-push-script --all because it's bringing a dynamically growing
forest of heads from the arch repo. That's another reason why your
private branches should be elsewhere.

OTOH, storing the metadata in a branch will allow us to run the import
in alternating repositories. But as Junio points out, unless I can
guarantee that the metadata and the tree are in sync, I cannot
trivially resume the import cycle from a new repo.

The git solution to this (which nobody has ever _used_, but which
technically is wonderful) is to have a "side branch" that does not share
any commits (or files, for that matter) in common with the "real branch",
and which is used to track any metadata. In fact, you can obviously have
any number of side branches.

A couple of days ago, playing with the import, I realised that the git
repo can hold unrelated projects, too, if you just commit orphan trees
as new heads. I mean - it was a bug in my script but I thought it was
cool. ;)

The way to maintain a metadata branch is to have not only a different
branch name (obviously), but also use a totally different index file, so
that you can index both branches in parallell, and you don't actually need
to check out one or the other.

Hmmm. Now that's voodoo magic! I was thinking of reading the file by
asking directly for the object by its sha, or doing a checkout in a
tmpdir. Interesting.

cheers,


martin

Yes it was irritating.

Signed-off-by: Junio C Hamano <redacted>

---

    Linus Torvalds [off-list ref] writes:
    > [ Junio, the fact that you can't script the initial commit with "git 
    >   commit" is _really_ irritating. ]

 git-commit-script |   86 ++++++++++++++++++++++++-----------------------------
 1 files changed, 39 insertions(+), 47 deletions(-)

c038244ac9260c8c895bf791ff587103bacadaba

diff --git a/git-commit-script b/git-commit-script
--- a/git-commit-script
+++ b/git-commit-script

@@ -110,57 +110,51 @@ t)
 	fi
 esac
 
+if [ ! -r "$GIT_DIR/HEAD" ]
+then
+	echo "#"
+	echo "# Initial commit"
+	echo "#"
+	git-ls-files | sed 's/^/# New file: /'
+	echo "#"
+elif [ -f "$GIT_DIR/MERGE_HEAD" ]; then
+	echo "#"
+	echo "# It looks like your may be committing a MERGE."
+	echo "# If this is not correct, please remove the file"
+	echo "#	$GIT_DIR/MERGE_HEAD"
+	echo "# and try again"
+	echo "#"
+fi >.editmsg
+if test "$log_message" != ''
+then
+	echo "$log_message"
+elif test "$logfile" != ""
+then
+	if test "$logfile" = -
+	then
+		test -t 0 &&
+		echo >&2 "(reading log message from standard input)"
+		cat
+	else
+		cat <"$logfile"
+	fi
+elif test "$use_commit" != ""
+then
+	git-cat-file commit "$use_commit" | sed -e '1,/^$/d'
+fi | git-stripspace >>.editmsg
+
 PARENTS="-p HEAD"
 if [ ! -r "$GIT_DIR/HEAD" ]; then
 	if [ -z "$(git-ls-files)" ]; then
 		echo Nothing to commit 1>&2
 		exit 1
 	fi
-	{
-		echo "#"
-		echo "# Initial commit"
-		case "$no_edit" in
-		t) echo "# (ignoring your commit message for initial commit)"
-		   no_edit= 
-		esac
-		echo "#"
-		git-ls-files | sed 's/^/# New file: /'
-		echo "#"
-	} >.editmsg
 	PARENTS=""
-	no_edit=
 else
 	if [ -f "$GIT_DIR/MERGE_HEAD" ]; then
-		{
-		echo "#"
-		echo "# It looks like your may be committing a MERGE."
-		echo "# If this is not correct, please remove the file"
-		echo "#	$GIT_DIR/MERGE_HEAD"
-		echo "# and try again"
-		case "$no_edit" in
-		t) echo "# (ignoring your commit message for merge commit)"
-		   no_edit= 
-		esac
-		echo "#"
-		} |
-		git-stripspace >.editmsg
 		PARENTS="-p HEAD -p MERGE_HEAD"
-	elif test "$log_message" != ''
-	then
-		echo "$log_message" |
-		git-stripspace >.editmsg
-	elif test "$logfile" != ""
-	then
-		if test "$logfile" = -
-		then
-			test -t 0 &&
-			echo >&2 "(reading log message from standard input)"
-			cat
-		else
-			cat <"$logfile"
-		fi |
-		git-stripspace >.editmsg
-	elif test "$use_commit" != ""
+	fi
+	if test "$use_commit" != ""
 	then
 		pick_author_script='
 		/^author /{

@@ -188,22 +182,20 @@ else
 		export GIT_AUTHOR_NAME
 		export GIT_AUTHOR_EMAIL
 		export GIT_AUTHOR_DATE
-		git-cat-file commit "$use_commit" |
-		sed -e '1,/^$/d' |
-		git-stripspace >.editmsg
 	fi
-
 	case "$signoff" in
 	t)
 		git-var GIT_COMMITTER_IDENT | sed -e '
 			s/>.*/>/
-			s/^/Signed-off-by: /' >>.editmsg ;;
+			s/^/Signed-off-by: /
+		' >>.editmsg
+		;;
 	esac
 	git-status-script >>.editmsg
 fi
 if [ "$?" != "0" -a ! -f $GIT_DIR/MERGE_HEAD ]
 then
-	cat .editmsg
+	sed -ne '/^#/p' .editmsg
 	rm .editmsg
 	exit 1
 fi

On 8/26/05, Linus Torvalds [off-list ref] wrote:

quoted

OTOH, storing the metadata in a branch will allow us to run the import
in alternating repositories. But as Junio points out, unless I can
guarantee that the metadata and the tree are in sync, I cannot
trivially resume the import cycle from a new repo.

But you can.

Remember: the metadata is the pointers to the original git conversion, and
objects are immutable.

In other words, if you just have a "last commit" pointer in your
meta-data, then git is _by_definition_ in sync. There's never anything to
get out of sync, because objects aren't going to change.

Hmmm. That repo is in sync, but there are no guarantees that they will
travel together to a different repo. In fact, the push/pull
infrastructure wants to push/pull one head at a time.

And if they are not in sync, I have no way of knowing. Hmpf. I lie:
the arch metadata could keep track of what it expects the last head
commits to be, and complain bitterly if something smells rotten.

let me think about it ;)


martin