Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

On Tue, 4 Jul 2017 21:03:22 +0200
Thorsten Leemhuis [off-list ref] wrote:

On 03.07.2017 18:30, Steven Rostedt wrote:

quoted

On Sun, 2 Jul 2017 19:51:43 +0200
Thorsten Leemhuis [off-list ref] wrote:  

quoted

 * How to get subsystems maintainer involved more in regression tracking
to better make sure that reported regressions are tracked and not
forgotten accidentally.  

We should push harder for all reproducer tests to be put into
selftests. I try to do that myself [...]
[...]
By adding reproducing tests to selftests, we can easily see what
regressions are still there.
[...]
What is selftests?  (Jeopardy answer for all of the above ;-)  

Sure, writing and running selftests is a good idea. But as you said
yourself in the later part of your mail: it won't help much in
situations where the kernel (or a selftest) needs to run on a certain
hardware or a specific (and maybe rare or complex) configuration. Sadly
a lot of the regressions in my recent reports were of this kind afaics :-/

In fact I got the impression that most of the regressions that might get
caught by selftests were directly handled by the subsystem maintainer
and never made it to me or my reports -- and thus I can't ask
maintainers to write selftests. *If* I got better aware of those
problems I (a) could make sure they are not forgotten and (b) sooner or
later could publicly state something like "hey, you had ten regressions
recently in your subsystem where writing a selftest might have been a
good idea, but you didn't even write one -- why?" (if we want something
like that).

I'm betting there's a lot of reproducer code that never makes it into a
test. How do we solve that? Perhaps we need people looking at LKML for
any signs "I did this, and it caused a bug" or "Here's a test case
which can trigger the bug". Each of these instances should end up in
selftests, and I'm sure they are not.

We can't do much for special hardware, even though those tests should
still be in the selftests for those that have the hardware, but we can
do something about special configs. Perhaps selfttests should have a
"config test" section. I have that in my own tests, but I use ktest to
build them.

quoted

[…]  

quoted

 * how to make the Linux kernel development so good that the mainstream
distros stop their kernel forks and do what they do with Firefox: Ship
the latest stable version (users get a new version with new features
every few weeks) or a longterm branch (makes a big version jump about
once a year; see Firefox ESR).  

Hehe, I maybe left the field "regression tracking" to much here and
wandered too far into QA territory.

quoted

This wont ever happen (famous last words). Distros want "stable
kernels" with new features.   

Ha, yes, it's a long shot (and maybe more a vague idea to work towards
to). And maybe Debian stable and RHEL will always use the model they use
today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and
some others are updating to the latest Linux kernel release every few
weeks already and it works fine for them. Maybe we can get Ubuntu and
others to follow sooner or later.

Sure, for some people a version jump to a major new kernel release will
sound crazy, but when Linus introduced the current development scheme  a
lot of people also said "that will never fly" -- that was 13 years ago
now and it works quite well. The situation was similar with Firefox as well.

quoted

That's not what stable is about.  

That afaics (disclaimer: English is not my mother tongue) depends on the
interpretation of the word, as it can mean "nothing changes" or "rock
solid/reliable" (even when two people have a "stable relationship" it
does not mean that nothing changes between them...).

Nothing to do with what language your mother tongue is ;-)

When the stable releases were created, there was some pretty strict
requirements for what should go into stable. Of course the requirements
have changed throughout the years. But there are big differences in
what Red Hat considers something "stable" and what the Linux stable
releases consider to be stable. That is where I meant that things wont
change.

-- Steve