publishing a forked^W cloned directory with ancestry

Dear list,

I am the mdadm (software RAID management) maintainer for Debian and
a recent git convert. While preparing a new mdadm package tonight,
I found myself screaming too often at SVN [0] and decided to convert
Debian mdadm package maintenance to git. Upstream also uses git, so
I said to myself "jolly good", clapped hands, and… froze as the
pieces weren't fitting together nicely, at least from my
understanding of how git works.

There are multiple ways to maintain a Debian package and I shall
today only concentrate on tracking an upstream VCS repo and
packaging it for Debian "when I please" (others call that
snapshotting). Thus, I don't wait for upstream to publish tarballs,
I make the package from a given HEAD. There are pros and cons to
this, but let's just assume that it's "the way".

So I clone upstream and find that git-branch -r includes
upstream/master (s/origin/upstream/ for clarity). I then branch
'debian' off upstream/master and make some required changes. With
utter enjoyment of git, I wrap it up and package a new mdadm.deb.
Yay.

And then I wonder: how do I now publish this result of my work? I'd
like to push my repository to git.debian.org so that others can
clone it and help or submit patches against the debianised upstream.

But the remote branch upstream/master only really exists in
$GIT_DIR, which is local and can't be pushed. Or well, even if
I tried, the people cloning from the push location wouldn't see it
[1].

I saw two solutions to this: using two separate upstreams/origins,
and submodules:

1. I could tell my $GIT_DIR/config that upstream/* comes from mdadm
upstream and debian/* comes from git.debian.org and then merge
happily across branches locally and be done with it. However, John
Doe, who on a rainy Saturday afternoon has two hours to spend and
wants to fix some mdadm bugs would have to jump through hoops to
replicate the setup: all the ties between upstream and the
git.debian.org repo are local to my machine and can't be pushed
anywhere (except to verbose documentation).

2. Let's assume for a moment that all Debian changes go to ./debian,
then submodules seem to want to save the day. Except that Debian
changes are not confined to ./debian, and from the perspective of
the Debian mdadm maintainer and with the semantics of git-submodules
in the back of my head [2], I'd rather want it the other way around:
upstream be a submodule of my Debian-specific repo; but upstream
needs to live in . and TTBOMK, submodules can't live in .

So neither of these work and thus I am turning to you. I want to
publish my Debian "fork" of mdadm in such a way that people can
easily clone it, without pulling all the components together. How
would you do it?

I guess the cleanest solution I can come up with is to branch off
upstream/master into branch "upstream" whenever *I* decide it's time
to snapshot. Then, people using my repo would basically be confined
to the state of the tree as it was the last time I rebased
"upstream", but could work freely on the Debian-specific stuff.
I think this is actually quite okay, but I am still interested in
any comments you may have.

Cheers,
m


Footnotes:

[0] fine "too often" tonight meant "once" but it's once too many.
    Thanks Linus and Junio, since I touched git I can't work most
    other VCS anymore (this is sarcastic... not).
[1] I tried cloning B from A, then cloning C from B. Within C, there
    is no reference to A's master branch, so unless B pulled changes
    from A and C pulled changes from B, C could not be updated.
[2] The submodule's commit ID is stored in the supermodules index
    and a commit to the submodule also requires a commit to the
    supermodule to restore consistency.

-- 
"if builders built buildings the way
 programmers wrote programs,
 then the first woodpecker that came along
 would destroy civilization."
                                                  -- gerald weinberg
 
spamtraps: madduck.bogus@madduck.net

also sprach martin f krafft [off-list ref] [2007.08.30.2125 +0200]:

And then I wonder: how do I now publish this result of my work? I'd
like to push my repository to git.debian.org so that others can
clone it and help or submit patches against the debianised upstream.

With this I mean: it would just be nice if people cloning the
git.debian.org repo could also use the upstream refs. I am aware
that the pack they download is complete in the sense that they can
just build after cloning (thanks to Harri Ilari Tapio Liusvaara
though for clearing this up for me a bit). After all, this is what
I meant when I wrote:

I guess the cleanest solution I can come up with is to branch off
upstream/master into branch "upstream" whenever *I* decide it's time
to snapshot. Then, people using my repo would basically be confined
to the state of the tree as it was the last time I rebased
"upstream", but could work freely on the Debian-specific stuff.

But I guess in the Debian world, a bug may be fixed upstream before
it is fixed in Debian (not only because Debian is allegedly
outdated, also because our users and developers often cooperate
directly with upstream), and then when someone jumps in for me to
release a new mdadm package, they might just need to rebase to the
latest upstream HEAD.

Is it possible to enable this with git without asking those people
to first set up their local repo to reference git.debian.org *and*
upstream, all of which is likely going to be more than four
commands?

Harri hinted at using Makefiles, and sure, I can use the Makefile to
set up upstream if it's not already present, but I'd much rather
have a standard way that's going to be the same across all
git-maintained Debian packages.

Comments welcome,

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
"no woman should ever be quite accurate about her age.
 it looks so calculating."
                                                        -- oscar wilde
 
spamtraps: madduck.bogus@madduck.net

On Thu, Aug 30, 2007 at 09:25:33PM +0200, martin f krafft wrote:

So I clone upstream and find that git-branch -r includes
upstream/master (s/origin/upstream/ for clarity). I then branch
'debian' off upstream/master and make some required changes. With
utter enjoyment of git, I wrap it up and package a new mdadm.deb.
Yay.

And then I wonder: how do I now publish this result of my work? I'd
like to push my repository to git.debian.org so that others can
clone it and help or submit patches against the debianised upstream.

So in the setup you describe if they clone your repo then they'll get a
single branch called 'debian' with your work in it.  That sounds fine to
me, actually.

But the remote branch upstream/master only really exists in
$GIT_DIR, which is local and can't be pushed. Or well, even if
I tried, the people cloning from the push location wouldn't see it

They can always just fetch from upstream as well if they'd like.  They
could do something like:

	git clone git://coolproject.org/cool.git
	cd cool
	git remote add debian git://git.debian.org/cool.git
	git fetch debian

Then they have a repository where git-branch -r reports something like

	origin/master
	debian/debian

Or they could do it the other way around, with "origin" pointing to you
and an "upstream" remote pointing to coolproject.org.  The naming's
obviously up to them.

1. I could tell my $GIT_DIR/config that upstream/* comes from mdadm
upstream and debian/* comes from git.debian.org and then merge
happily across branches locally and be done with it. However, John
Doe, who on a rainy Saturday afternoon has two hours to spend and
wants to fix some mdadm bugs would have to jump through hoops to
replicate the setup: all the ties between upstream and the
git.debian.org repo are local to my machine and can't be pushed
anywhere (except to verbose documentation).

Maybe the one extra "git remote add ...; git remote fetch" isn't such a
big deal?

I guess the cleanest solution I can come up with is to branch off
upstream/master into branch "upstream" whenever *I* decide it's time
to snapshot. Then, people using my repo would basically be confined
to the state of the tree as it was the last time I rebased
"upstream", but could work freely on the Debian-specific stuff.
I think this is actually quite okay, but I am still interested in
any comments you may have.

Sure, you can do that.  I don't think it's really necessary.

My local kernel repository, for example, currently knows about five
other repos:

	$ git remote
	labiaga		# server pnfs work
	linux-nfs	# my public repo
	origin		# Linus's repo
	richterd	# a coworker's nfs work
	trond		# Trond's nfs stuff

Sure, each of those could add a "linus" branch that tracked upstream, so
I could still get some idea what Linus's tree was even if I didn't
happen to already have it.  But then I'd end up with 4 different
slightly-out-of-date pointers to the head of linus's repo in each of
those trees, which would end up being just be a bunch of cruft that I'd
have to ignore whenever I looked at them.

--b.

* martin f krafft [off-list ref] [070830 15:25]:

[1] I tried cloning B from A, then cloning C from B. Within C, there
    is no reference to A's master branch, so unless B pulled changes
    from A and C pulled changes from B, C could not be updated.

I think a .git/config like this will do what you want:

[remote "upstream"]
        url = git://mdadm-upstream-repo
        fetch = +refs/heads/*:refs/remotes/upstream/*

[remote "debian"]
        url = git://debian-repo-you-want-to-publish-to
        fetch = +refs/heads/*:refs/remotes/debian/*
        push = refs/remotes/upstream/master:refs/heads/upstream
        push = refs/heads/master:refs/heads/master

Now when you 'git push debian' it will populate the 'upstream' and
'master' branches properly.

When someone clones your repo, they will get origin/master (your branch)
and origin/upstream (the official mdadm branch).

Did I understand the problem correctly?

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/