So I tried to apply my new get-mm-patches-from-git methodology on this and
came unstuck.
-mm kernels consist of a series of patches against the most recent release
(2.6.12-rc3 today):
linus.patch (Linus' changes since 2.6.12-rc3)
git-net.patch (Davem's changes wrt Linus's latest tree)
etc...
The algorithm is:
a) Set up the git repo
mkdir git26
git init rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
(Futz around in `git log' output to identify the v2.6.12-rc3 commit, do
`git tag v2.6.12-rc3 a2755a80f40e5794ddc20e00f781af9d6320fafb')
b) Add davem's repo:
git addremote git-net rsync://rsync.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git
c) To generate -mm's linus.patch (patch against 2.6.12-rc3):
git pull origin
git diff -r v2.6.12-rc3 > ../25/patches/linus.patch
d) To generate davem's tree (patch against linus's current tree (ie: patch
against 2.6.12-rc3+linus.patch)):
git pull git-net
MERGE_BASE=$(merge-base $(cat .git/heads/origin ) $(cat .git/heads/git-net))
git diff -r $MERGE_BASE:$(cat .git/heads/git-net) > ../25/patches/git-net.patch
e) Repeat d) for all known git trees.
But git-net.patch has a bunch of bluetooth stuff in it which is already in
Linus's tree. And git-net.patch modifies net/sched/simple.c, which doesn't
appear in either 2.6.12-rc3 or in current -linus.
Doing step d) by hand:
bix:/usr/src/git26> cat .git/heads/origin
b453257f057b834fdf9f4a6ad6133598b79bd982
bix:/usr/src/git26> cat .git/heads/git-net
5523662c4cd585b892811d7bb3e25d9a787e19b3
bix:/usr/src/git26> merge-base b453257f057b834fdf9f4a6ad6133598b79bd982 5523662c4cd585b892811d7bb3e25d9a787e19b3
25ee7e3832951cf5896b194f6cd929a44863f419
bix:/usr/src/git26> cat-file commit 25ee7e3832951cf5896b194f6cd929a44863f419
tree 5b6486ded5188e41ac9bc81ad4a5e2bd746f7ede
parent 056de2fa12febe02597f971eb6ea8f2cc9c9b06e
author Adrian Bunk [off-list ref] 1114442294 -0700
committer Linus Torvalds [off-list ref] 1114442294 -0700
[PATCH] fs/aio.c: make some code static
This patch makes some needlessly global code static.
Signed-off-by: Adrian Bunk [off-list ref]
Acked-by: Benjamin LaHaise [off-list ref]
Signed-off-by: Linus Torvalds [off-list ref]
That seems to be a reasonable gca. It's the last thing which Linus added
prior to merging the ARM patches, which presumably weren't in Dave's tree.
So let's try to grab davem's diff wrt that gca:
bix:/usr/src/git26> git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:5523662c4cd585b892811d7bb3e25d9a787e19b3 | diffstat
drivers/net/tg3.c | 73 ++++++++++++++-------------
net/bluetooth/af_bluetooth.c | 1
net/bluetooth/bnep/sock.c | 1
net/bluetooth/cmtp/capi.c | 1
net/bluetooth/cmtp/core.c | 1
net/bluetooth/cmtp/sock.c | 1
net/bluetooth/hci_conn.c | 1
net/bluetooth/hci_core.c | 1
net/bluetooth/hci_event.c | 1
net/bluetooth/hci_sock.c | 1
net/bluetooth/hidp/core.c | 1
net/bluetooth/hidp/sock.c | 1
net/bluetooth/l2cap.c | 1
net/bluetooth/rfcomm/sock.c | 1
net/bluetooth/sco.c | 1
net/core/rtnetlink.c | 1
net/core/scm.c | 1
net/core/sock.c | 1
net/ipv4/af_inet.c | 1
net/ipv4/ip_output.c | 2
net/ipv4/netfilter/ip_conntrack_ftp.c | 4 -
net/ipv4/netfilter/ip_conntrack_standalone.c | 7 --
net/ipv4/tcp_input.c | 1
net/ipv6/af_inet6.c | 1
net/netlink/af_netlink.c | 1
net/sched/simple.c | 18 ------
net/unix/af_unix.c | 1
27 files changed, 46 insertions(+), 80 deletions(-)
And that's the bad patch.
What did I do wrong?
Can someone suggest a better approach?
Thanks.
So I tried to apply my new get-mm-patches-from-git methodology on this and
came unstuck.
Yes. You cannot just apply patches, since that will inevitably fail if
there is any overlap. Which there quite often is.
For this to work in general, you really have to merge the different git
trees, and generate patches from _that_.
a) Set up the git repo
mkdir git26
git init rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Yes. In the long run, you really should need to do this just once, since
if there is one thing git should be good at, it's just keeping tons of
random collections of objects around.
The only thing you should be a bit careful about is to remember what the
"heads" at different points were. In particular, you want to remember
where you merged with me last was. I've started tagging my releases with
the git tag facility (_not_ the pasky one, but I think pasky will start
picking up on that soon enough), so finding a specific release will be
easy, but if you ever do a non-release merge you'll just have to tag it
yourself.
b) Add davem's repo:
git addremote git-net rsync://rsync.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git
c) To generate -mm's linus.patch (patch against 2.6.12-rc3):
git pull origin
git diff -r v2.6.12-rc3 > ../25/patches/linus.patch
You should now also remember the HEAD at this point. That's your "base"
for any future patches, since you expect other patches to apply on top of
that.
I think cogito remembers it in the "origin" thing, but you should check.
Save it away in (for example) .git/last-diff-head:
cat .git/HEAD > .git/last-diff-head
d) To generate davem's tree (patch against linus's current tree (ie: patch
against 2.6.12-rc3+linus.patch)):
git pull git-net
Yes. This should have merged the two (assuming "git pull" does what I
think it does).
No. Now you ended up looking at the last common ancestor of the thing you
merged, and you _should_ have looked at what the difference was _before_
the merge.
So assuming it does remember it in "origin", you should just have done
git diff -r $(cat .git/last-diff-head):$(cat .git/HEAD)
which basically says "diff between the tree at the time of my last diff,
and the result of the merge".
Then you just update your last-diff-head to reflect that:
cat .git/HEAD > .git/last-diff-head
and you go on:
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:41:54
On Tue, 26 Apr 2005, Linus Torvalds wrote:
On Tue, 26 Apr 2005, Andrew Morton wrote:
quoted
The only thing you should be a bit careful about is to remember what the
"heads" at different points were. In particular, you want to remember
where you merged with me last was. I've started tagging my releases with
the git tag facility (_not_ the pasky one, but I think pasky will start
picking up on that soon enough), so finding a specific release will be
easy, but if you ever do a non-release merge you'll just have to tag it
yourself.
Your tag system is in the "cogito-0.8" release, plus a pasky-style way of
keeping track of what tags you have in your repository.
quoted
d) To generate davem's tree (patch against linus's current tree (ie: patch
against 2.6.12-rc3+linus.patch)):
git pull git-net
Yes. This should have merged the two (assuming "git pull" does what I
think it does).
I think git pull only downloads the contents of the repo and saves the new
head in a separate file. You're left to do the merge yourself, in case
what you actually wanted to do was just read the patches in the remote
repo without merging them.
You probably need a "git merge git-net" here, and things will be in the
state that Linus expects.
-Daniel
*This .sig left intentionally blank*
From: Jan Harkes <jaharkes@cs.cmu.edu> Date: 2016-06-15 22:41:54
Forgot to cc: git@vger when I sent this originally.
On Tue, Apr 26, 2005 at 12:57:25AM -0700, Andrew Morton wrote:
27 files changed, 46 insertions(+), 80 deletions(-)
And that's the bad patch.
Both Linus and DaveM merged the same patch from Al Viro, but they are
ofcourse different commits. So both branches have diverged and would
need to be merged, where someone decides if Linus's commit or Dave's
commit should be used.
What did I do wrong?
Can someone suggest a better approach?
Well, I looked at the history, and if I skip the Viro patch in davem's
branch by hand I get,
$ git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:088dd3a45fdb8fb726cd50575856562c4f6f1c3e | diffstat
drivers/net/tg3.c | 73 ++++++++++++++-------------
net/ipv4/ip_output.c | 2
net/ipv4/netfilter/ip_conntrack_ftp.c | 4 -
net/ipv4/netfilter/ip_conntrack_standalone.c | 7 --
net/ipv4/tcp_input.c | 1
net/sched/simple.c | 18 ------
6 files changed, 46 insertions(+), 59 deletions(-)
So now, how to do this without picking through the logs. I figured the
patchutils stuff might have a solution for this. So I got the diffs on
Linus's branch wrt to the gca, and from Dave's branch wrt to the gca.
$ git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:b453257f057b834fdf9f4a6ad6133598b79bd982 > git-linus.patch
$ git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:5523662c4cd585b892811d7bb3e25d9a787e19b3 > git-net.patch
$ interdiff --no-revert-omitted -p1 git-linus.patch git-net.patch | diffstat
drivers/net/tg3.c | 73 ++++++++++++++-------------
net/ipv4/ip_output.c | 2
net/ipv4/netfilter/ip_conntrack_ftp.c | 4 -
net/ipv4/netfilter/ip_conntrack_standalone.c | 7 --
net/ipv4/tcp_input.c | 1
net/sched/simple.c | 18 ------
6 files changed, 46 insertions(+), 59 deletions(-)
That looks like it matched.
Jan
From: Petr Baudis <hidden> Date: 2016-06-15 22:41:54
Dear diary, on Tue, Apr 26, 2005 at 05:40:08PM CEST, I got a letter
where Daniel Barkalow [off-list ref] told me that...
On Tue, 26 Apr 2005, Linus Torvalds wrote:
quoted
On Tue, 26 Apr 2005, Andrew Morton wrote:
quoted
The only thing you should be a bit careful about is to remember what the
"heads" at different points were. In particular, you want to remember
where you merged with me last was. I've started tagging my releases with
the git tag facility (_not_ the pasky one, but I think pasky will start
picking up on that soon enough), so finding a specific release will be
easy, but if you ever do a non-release merge you'll just have to tag it
yourself.
Your tag system is in the "cogito-0.8" release, plus a pasky-style way of
keeping track of what tags you have in your repository.
It came in with merge with Linus, but there's no support in the Cogito
toolkit for it whatsoever.
I'm personally unlikely to get any time for doing it (or much anything
else; but this is probably the priority now) until Sunday evening. :-(
quoted
quoted
d) To generate davem's tree (patch against linus's current tree (ie: patch
against 2.6.12-rc3+linus.patch)):
git pull git-net
Yes. This should have merged the two (assuming "git pull" does what I
think it does).
I think git pull only downloads the contents of the repo and saves the new
head in a separate file. You're left to do the merge yourself, in case
what you actually wanted to do was just read the patches in the remote
repo without merging them.
You probably need a "git merge git-net" here, and things will be in the
state that Linus expects.
Yes. git pull only pulls the stuff, doesn't merge it (that is, unless it
is tracked branch; but I think this isn't the case; this was cleaned up
in cogito-0.8 and cg-pull now always only pulls, cg-update
pulls+merges).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
From: Petr Baudis <hidden> Date: 2016-06-15 22:41:54
Dear diary, on Tue, Apr 26, 2005 at 09:57:25AM CEST, I got a letter
where Andrew Morton [off-list ref] told me that...
c) To generate -mm's linus.patch (patch against 2.6.12-rc3):
git pull origin
git diff -r v2.6.12-rc3 > ../25/patches/linus.patch
(Mainly for people not familiar with git-pasky - this merged origin to
the working tree since it was tracked branch - that's what origin is by
default after git init.)
d) To generate davem's tree (patch against linus's current tree (ie: patch
against 2.6.12-rc3+linus.patch)):
git pull git-net
MERGE_BASE=$(merge-base $(cat .git/heads/origin ) $(cat .git/heads/git-net))
git diff -r $MERGE_BASE:$(cat .git/heads/git-net) > ../25/patches/git-net.patch
This is the bad way; I think this suffers of basically the same problems
as my ancient merging by "forward-patching". You should probably do a
regular merge:
git pull git-net
git merge git-net
git diff -p
The last command will show diff between current tree and the first
parent; that amounts the merged patch in this case.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
From: Andrew Morton <hidden> Date: 2016-06-15 22:41:54
Petr Baudis [off-list ref] wrote:
> d) To generate davem's tree (patch against linus's current tree (ie: patch
> against 2.6.12-rc3+linus.patch)):
>
> git pull git-net
> MERGE_BASE=$(merge-base $(cat .git/heads/origin ) $(cat .git/heads/git-net))
> git diff -r $MERGE_BASE:$(cat .git/heads/git-net) > ../25/patches/git-net.patch
This is the bad way; I think this suffers of basically the same problems
as my ancient merging by "forward-patching". You should probably do a
regular merge:
git pull git-net
git merge git-net
git diff -p
The last command will show diff between current tree and the first
parent; that amounts the merged patch in this case.
Bear in mind that there will be 20 or 30 different trees which I'll need
the diffs for, not just the one git-net.
I don't know if it'll be successful continually merging all those trees
together. The way I did this with bk was to have a separate repo for each
tree, but I don't think I'll want 30-40 separate git trees.
My little methodology worked nicely for git-ia64.
Jan Harkes has pointed out that the problem here is that Linus and Dave
both applied the same patch from Al and that interdiff was able to fix it
up:
$ git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:b453257f057b834fdf9f4a6ad6133598b79bd982 > git-linus.patch
$ git diff -r 25ee7e3832951cf5896b194f6cd929a44863f419:5523662c4cd585b892811d7bb3e25d9a787e19b3 > git-net.patch
$ interdiff --no-revert-omitted -p1 git-linus.patch git-net.patch | diffstat
drivers/net/tg3.c | 73 ++++++++++++++-------------
net/ipv4/ip_output.c | 2
net/ipv4/netfilter/ip_conntrack_ftp.c | 4 -
net/ipv4/netfilter/ip_conntrack_standalone.c | 7 --
net/ipv4/tcp_input.c | 1
net/sched/simple.c | 18 ------
6 files changed, 46 insertions(+), 59 deletions(-)
So hm. I guess git did what it was supposed to do here, and that a `git
merge' would have removed the common patch. But if I take the approach of
merging all those subsystem trees I do wonder if things will come
unstuck...
I don't know if it'll be successful continually merging all those trees
together. The way I did this with bk was to have a separate repo for each
tree, but I don't think I'll want 30-40 separate git trees.
You really don't. With git, the only thing you need is one object store,
and some way to _track_ those 30-40 separate git trees (and "track" really
means "remember a single SHA1 name for their top-of-tree").
Then you can merge any combination of the 40 in the same tree.
You'll get confused easily, but if you do this all with tools, it
shouldn't be too bad.
So hm. I guess git did what it was supposed to do here, and that a `git
merge' would have removed the common patch. But if I take the approach of
merging all those subsystem trees I do wonder if things will come
unstuck...
Well, git isn't as good at merging as BK is, and your usage sure as hell
would be a horrible worst-case example, but it might actually work fine.
Git merges are _cheap_ (with a capital C, and probably H as well) when
they work out, and quite frankly, so far they have always worked out for
me.
But yeah, you'd be doing some pretty aggressive merging, using a tool that
is two weeks old. It might work. I'd be interested to know ;)
Linus
From: Andrew Morton <hidden> Date: 2016-06-15 22:41:54
Linus Torvalds [off-list ref] wrote:
On Tue, 26 Apr 2005, Andrew Morton wrote:
quoted
I don't know if it'll be successful continually merging all those trees
together. The way I did this with bk was to have a separate repo for each
tree, but I don't think I'll want 30-40 separate git trees.
You really don't. With git, the only thing you need is one object store,
and some way to _track_ those 30-40 separate git trees (and "track" really
means "remember a single SHA1 name for their top-of-tree").
Petr's wrappers do all that head tracking OK (which is why I'm using a
combo of those and of lower-level gittiness). They do handy remote repo
tracking too.
Then you can merge any combination of the 40 in the same tree.
You'll get confused easily, but if you do this all with tools, it
shouldn't be too bad.
quoted
So hm. I guess git did what it was supposed to do here, and that a `git
merge' would have removed the common patch. But if I take the approach of
merging all those subsystem trees I do wonder if things will come
unstuck...
Well, git isn't as good at merging as BK is, and your usage sure as hell
would be a horrible worst-case example, but it might actually work fine.
Git merges are _cheap_ (with a capital C, and probably H as well) when
they work out, and quite frankly, so far they have always worked out for
me.
But yeah, you'd be doing some pretty aggressive merging, using a tool that
is two weeks old. It might work. I'd be interested to know ;)
Hm. For now I might try what I have plus Jan's fancy interdiff command to
remove duped patches. We'll see. I obtained a copy of Tony's ia64 diff
quite happily - it didn't have any duplicates.
(It's fairly common for two subsystem maintainers to apply the same patch.
With bk I was resolving that by just smashing the patches on top of each
other, ignoring the rejects and refreshing the topmost patch. That
approach actually resolved this linus-vs-davem dupe as well. But the duped
patch was so damn BIG that it threw me. And I wasn't feeling gitty enough
to go hunting about looking for the particular patch(es) which caused the
problem)
With bk I was resolving that by just smashing the patches on top of each
other, ignoring the rejects and refreshing the topmost patch. That
approach actually resolved this linus-vs-davem dupe as well.
Oh, wow. I didn't realize that your scripts were quite _that_ stupid, and
didn't actually take advantage of any automatic merges at all.
If so, git should trivially do everything that BK ever did for you. Which
is not saying a lot ;)
Linus
With bk I was resolving that by just smashing the patches on top of each
other, ignoring the rejects and refreshing the topmost patch. That
approach actually resolved this linus-vs-davem dupe as well.
Oh, wow. I didn't realize that your scripts were quite _that_ stupid, and
didn't actually take advantage of any automatic merges at all.
If so, git should trivially do everything that BK ever did for you. Which
is not saying a lot ;)
No version control system will do a particularly good job of merging
content which got passed around outside of the system. They can be made to
sort-of handle some simple cases well, but fundamentally too much
information is getting dropped.
The solution is to get everyone using the same version control system,
which is actually quite a workable solution if (a) the version control
system in question is quite nice, and (b) there isn't some deep political
reason why many people will never agree to use it.
-Bram
From: Ryan Anderson <hidden> Date: 2016-06-15 22:41:55
On Tue, Apr 26, 2005 at 01:35:55PM -0700, Bram Cohen wrote:
Linus Torvalds wrote:
quoted
On Tue, 26 Apr 2005, Andrew Morton wrote:
quoted
With bk I was resolving that by just smashing the patches on top of each
other, ignoring the rejects and refreshing the topmost patch. That
approach actually resolved this linus-vs-davem dupe as well.
Oh, wow. I didn't realize that your scripts were quite _that_ stupid, and
didn't actually take advantage of any automatic merges at all.
If so, git should trivially do everything that BK ever did for you. Which
is not saying a lot ;)
No version control system will do a particularly good job of merging
content which got passed around outside of the system. They can be made to
sort-of handle some simple cases well, but fundamentally too much
information is getting dropped.
One thing to keep in mind, about the way Linux development works (and
honestly, the way I think "git" development is currently working) is
that one of the things the version control system has to provide an easy
method to do is to abandon history that is messy.
For example, I'm adding a new driver, "foobar".
It uses the fancy quantum-bus, which is fairly new to the kernel.
The bus driver is new, and I go off, work in my private tree for a
month, fixing all kinds of quantum-entanglement related bugs, and
committing as I go.
I get everything working, and submit my driver.
The quantum-bus maintainer replies and says, "Hey, I reworked the API so
that you don't need to worry about all this quantum-entanglement stuff
anymore, just call compensate_for_heisenberg() before doing the DMA."
I swear for a day or two, and rework my driver, and resubmit it.
Now, all that history I had, with the duplicated imlementation, and
useless code is in my tree.
The current (as I understand it) policy is, "We don't want that
history." This means that the developer will build a new tree (maybe),
export his patch and reimport it into a clean tree, making a much
simpler history graph.
What Andrew is doing isn't too far from this, in concept, it's just a
lot more complicated because he's pulling something insane, like 27
seperate trees, plus several hundred stand alone patches.
So, there's a *deliberate* desire to drop history and move some content
around outside of version control.
Now, the desire to pull a bunch of seperate trees, merge them, produce a
diff that roughly pertains to what came in from each tree, and collect
that as a patch series may be strange, but it seems to be working really
well at the moment, for Linux development.
The solution is to get everyone using the same version control system,
which is actually quite a workable solution if (a) the version control
system in question is quite nice, and (b) there isn't some deep political
reason why many people will never agree to use it.
Git (well, cogito, really) seems to be getting there awfully fast - I'm
rather impressed with the speed of it, and annoyed that I haven't had
the time to build up a test suite for merging!
--
Ryan Anderson
sometimes Pug Majere
From: Daniel Barkalow <hidden> Date: 2016-06-15 22:41:55
On Wed, 27 Apr 2005, Ryan Anderson wrote:
Now, all that history I had, with the duplicated imlementation, and
useless code is in my tree.
The current (as I understand it) policy is, "We don't want that
history." This means that the developer will build a new tree (maybe),
export his patch and reimport it into a clean tree, making a much
simpler history graph.
I've been doing just this. I actually import it in pieces, with a commit
between each, so it's just like I applied the patch series I'm about to
send out. It actually works beautifully, and, someday, I'll have the
series up on my site so that a maintainer can just pull it.
Honestly, I'm not interested long-term in my buggy history, even
locally; I'm interested in the clean history in which I make a series of
self-contained, logical modifications and they get merged upstream.
What Andrew is doing isn't too far from this, in concept, it's just a
lot more complicated because he's pulling something insane, like 27
seperate trees, plus several hundred stand alone patches.
So, there's a *deliberate* desire to drop history and move some content
around outside of version control.
I think it's more a desire to drop history as it actually happened, and
replace it with history as it should have happened. The one thing I would
like is the ability to provide merging help to poor souls who got part of
the messy history without preserving that history. I think having the head
of the clean series have a bunch of lines: "replaces <sha1>", where people
aren't supposed to have or want that commit, but if they've merged it,
they should know that the clean series includes its content.
-Daniel
*This .sig left intentionally blank*