Linus Torvalds [off-list ref] writes:
I think that's actually likely the exception rather than the rule. It's
much more likely that people have almost _all_ active development done on
side branches, and that - together with rebasing of the side branches -
inevitably means that the "main branch" ends up not having such a clean
set of "topic branch" merges.
You are absolutely right about "Andrew patchbomb" which is
linear and does not have the series boundary. Import from
mostly linear foreign SCM would have the same issue. Merge
topology would not help us at all in these cases.
In addition, on a more mature tree, a lot (probably _most_) of the commits
aren't really "topics" at all, but "maintenance", which exacerbates the
problem: you don't have a "line of development of this feature",
...
Put another way: bugs get fixed one by one, not in a nice linear fashion
by "topic".
Again, you are right, but that only means topic based grouping
is not for everybody, and certainly is not suitable for a long
stretch of commits on the trunk of a mature project because they
tend to touch everywhere and not all that clustered. If those
bugs were fixed by committing on separate topic branches and
then later merged, the topology based clustering would get the
grouping right, but I would imagine we would end up seeing
hundreds of such short groups which would not be useful at all.
In such cases, it would be much more useful to have one huge
group that says "these are small fixes, each of which may touch
different areas -- they are not related but grouped together
because they are all small, obviously correct and harmless
fixes". So I suspect that is a slightly different issue -- it
just illustrates the need for an "ungrouped" bin.
So I'm coming at it from a totally different project - where "topic
branches" simply aren't delineated as much, and even when they are, they
tend to be merged in multiple steps (and they pull both ways when they
aren't re-based).
I agree multiple steps merge and merging both ways would happen
in real life, but I had an impression that fpc handles that
topology reasonably well, unless that "merge from upstream" are
of "too frequent, automated and useless" kind of merges.
... but in the kernel, I pretty much guarantee
that you probably get better "topic clustering" by going simply by author,
like the old standard "git shortlog" does. Because that will tend to get
the clustering at a finer granularity (ie not just "networking", but
things like "packet filtering" etc).
So the "sort by people" actually works fairly well, but it's kind of an
"incidental" thing, and it _would_ be potentially useful to have other
ways of grouping things.
I think "networking" vs "packet filtering" largely depends on
how the networking subsystem you pull from is managed. If
netfilter comes as e-mailed patches to DaveM and are applied
onto the trunk of networking subsystem, we will face exactly the
same problem as we have with Andrew's patchbomb to your trunk.
If it were managed on a separate topic branch in the networking
subsystem repository (either DaveM manages them in his
repository as a topic, or DaveM pulls from netfilter git
repository -- I do not know how that part of the patchflow
works), I would imagine you would get the same "per topic"
grouping.
Another factor is that the author population of a wide and
mature project like the kernel tends to be more diverse, and a
single person tends to be focused on one thing at a time while
others work on different things. There is enough work in one
specific area for one person to do, and the project is too wide
for one person to be everywhere.
On Sun, 26 Nov 2006, Junio C Hamano wrote:
I think "networking" vs "packet filtering" largely depends on
how the networking subsystem you pull from is managed. If
netfilter comes as e-mailed patches to DaveM and are applied
onto the trunk of networking subsystem, we will face exactly the
same problem as we have with Andrew's patchbomb to your trunk.
Most of the subsystems end up using patches - they're simply better ways
to move things around and have people comment on them than saying "please
pull on this tree to see my suggestion". I do it myself: even when I
_generate_ the diff in my tree, I will often just do a
git diff > ~/diff
and then import the thing into my mailer, and say "Maybe something like
this?".
So I think patches are fundamentally the core way to get things in the
periphery into just about any system. Maybe we do it more than most just
because we're so _used_ to them, but I actually think that if the kernel
does it more than most (and I'm not sure it does), it's simply because the
thing about patches is that they really _work_.
So yes, the network subsystem tends to be entirely linear by the time it
hits me. That's true of a lot of other subsystems too (SCSI etc). There's
a _few_ subsystems that actually have real topic branches: ACPI and
network driver development comes to mind, but it seems to actually be the
exception rather than the rule.
(I think that a lot of people work like I occasionally do: they do have
their own local branches for some stuff, but they end up re-linearizing
and keeping them active with "git rebase", so the branches really are
purely local, rather than something that is visible in the end result).
But the REAL reason I'd love to see a smarter "data-mining" git log
(whether it does things by bayesian clustering or any other kind of
grouping technology) is that this is actually something that people ask
for: when I make my "git shortlog" for major releases, the thing is often
thousands of lines long, and it would be _beautiful_ if that could be
data-mined somewhat more intelligently.
So, for example, do a simple
git shortlog v2.6.17..v2.6.18
(with the shortlog in "next" that can do this - btw, why doesn't it
default to using PAGER like "git log" does?), and realize that it's about
8500 lines of stuff, and nobody can really be expected to read it. It's
not a "shortlog" in other words.
So what would a _nice_ "shortlog" do? I'd _love_ to see ways to make it
more concise, more "short" for something like this. Look at the output as
a _non_kernel_ person, and what does it tell you? Not a lot. It's just too
big.
Examples of what I think would be _really_ useful (much more so than
going by "topic branches", even if they existed):
- Clustering.
The author-based clustering does work, but it would be even better to
cluster by other methods ("subsystem" - either by subdirectory, or by
noticing filename patters, or even patterns in the patches: there's a
lot of academic work on clustering human text, perhaps not as much on
clustering patches).
- Shortening
The "shortlog" often isn't. It's wonderful for small things as-is, but
once it reaches a hundred lines or more, it's less so. It would often
be nice to be able to say "only show the 100 biggest patches" (or
preferably something smarter like "the 25 biggest clusters, with a
short 4-line clustering explanation", but even just the "biggest
patches" is useful in itself and much simpler)
- External annotations (eventually)
One of the things that people like LWN editor Jonathan Corbet would
want is a way to say which patches are "important". But the thing is,
"importance" is (a) fleeting and (b) not necessarily as obvious when
the commit is made as it is afterwards. So you cannot (and must not)
mark things "important" at commit-time, and it thus can't really be
part of the repo itself, but at the same time, this is definitely
something that _could_ be somehow logged/annotated externally.
Now, I realize that these are all pipe-dreams, but so was my old "a better
annotate than annotate" a year or two ago. So I'm not saying that people
should work on this, I'm just saying that it's worth perhaps thinking
about, because I think the git model does actually give us the power to
_do_ things like this. Eventually.
And the reason? Performance! Git is fast enough that we really _can_
afford to do things like "generate diffs for every single commit in the
range v2.6.17..v2.6.18" and it takes me just 20 seconds to do on a
reasonable machine with "git log -p". So good performance means that we
can _afford_ to do a diffstat for everything (or, just raw diffs to make
it even cheaper - quite often you care more about _which_ files and how
many files something touched than the actual size of the diff in those
files itself), and using that diffstat to some day generate shortlogs that
are more useful for people like Jonathan Corbet and others that just want
to get an overview of "what happened"?