Re: Article about "git bisect run" on LWN

On Thu, 5 Feb 2009, Ingo Molnar wrote:

* Christian Couder [off-list ref] wrote:

quoted

Hi,

For information, an article from me, 'Fully automated bisecting with "git
bisect run"' has been published in today's edition of LWN on the
development page:

http://lwn.net/Articles/317154/

Nice article!

In terms of possible future enhancements of git bisect, here's a couple of
random ideas that would help my auto-bisection efforts:

- Feature: support "Bisection Redundancy"

  This feature helps developers realize if a bug is sporadic. This happens
  quite often in the kernel space: a bug looks deterministic, but down the
  line it becomes sporadic. Sometimes a boot crash only occurs with a 75%
  probability - and if one is unlucky it can cause a _lot_ of wasted
  bisection time. The wrong commit gets blamed and the wrong set of
  developers start scratching their heads. It's a reoccuring theme on lkml.

  What git could do here is to allow testers to inject a bit of extra
  "redundancy" automatically, and use the redundant test-points to detect
  conflicts in good/bad constraints.

  It would work like this:

     git bisect start --redundancy=33%

  It would mean that for every third bisection points, Git would
  _not_ chose the ideal (estimated) 'middle point' from the set of "unknown
  quality" changes that are still outstanding - but would intentionally
  "weer outside" and select one commit from the _known_ set of commits.

  If such a redundant re-test of the known-good or known-bad set yields a
  nonsensical result then Git aborts the bisection with a "logic
  inconsistency detected" kind of message - and people could at this point
  realize the non-determinism of the test.

  ( Git can do this when a "redundant" test point is marked as 'bad' -
    despite an earlier bisection already categorizing that test point as
    'good' - or if it's the other way around. Git will only continue with
    the bisection if the test point has the expected quality. )

  This essentially means an automated re-test - but it's much better than
  just a repeated bisection - i've often met non-deterministic bugs that
  yield the _exact same_ nonsensical commit even on repeat bisections. That
  happens when a timing bug depends on the exact kernel layout, or a
  miscompilation or linker bug depends on the exact kernel layout, etc.

  It's also faster than a re-done bisection: 33% more testpoints is better
  than twice as many test-points. Also, auto-bisection can deal with
  redundancy just fine - it does not really matter whether i have to wait
  20 or 30 minutes for a test result since there's no manual intervention
  needed - but it _very_ much matters whether i can trust the validity of
  the bisection result.

when you gave this the title of redundnancy and described the problem I 
assumed that you would then propose running the test multiple times (so 
"git bisect run X --redundancy 5" would run each test 5 times, it would 
pass IFF it passed the test all 5 times. that would seem to be a better 
match for the name, as well as being a better test

David Lang