Re: [PATCH/RFC/GSOC] make git-pull a builtin

Hi Paul,

On 2015-03-21 14:23, Paul Tan wrote:

Thanks for the review, though I would like to work on the proposal now
before the deadline passes :)

That makes sense.

On Thu, Mar 19, 2015 at 1:52 AM, Matthieu Moy
[off-list ref] wrote:

quoted

Paul Tan [off-list ref] writes:

quoted

Ideally, I think the solution is to
improve the test suite and make it as comprehensive as possible, but
writing a comprehensive test suite may be too time consuming.

time-consuming, but also very beneficial since the code would end up
being better tested. For sure, a rewrite is a good way to break stuff,
but anything untested can also be broken by mistake rather easily at any
time.

I'd suggest doing a bit of manual mutation testing: take your C code,
comment-out a few lines of code, see if the tests still pass, and if
they do, add a failing test that passes again once you uncomment the
code.

Maybe code coverage tools could help here so we only need to focus on
the code paths that are untested by the test suite. At the minimum,
all of the non-trivial code paths in both the shell script and the
converted builtin must be covered by tests. This should help to
eliminate most sources of breakages. Anything further than that would
require an experienced understanding of all the possible important
inputs to be tested, which I personally feel would make the project
quite tedious.

I see git already has gcov support. For shell scripts, maybe kcov[1]
could be used. With some slight code changes, I managed to generate a
report for the git-pull tests[2] which should at least provide a good
starting point for how the tests can be improved.

While it is often a tempting idea to make test suites as thorough as possible, there lies a true danger herein. True war story: in one of the projects I was involved in, the test suite grew to a size that one complete run lasted two weeks. Yes, that is fourteen days. Needless to say: this test suite was run rarely. How useful is a test suite that is run rarely? More useful than a non-existent one, to be sure, but it is still more of a burden than a boon.

Now, on Windows the test suite takes almost three hours to run. This really, really slows down development.

So while we are not yet at the "too large to be useful state", I would caution against trying to get there.

Instead, I would really like to focus on the *usage*. Calling `git grep "git pull" t/` should give you an idea what usage of `git pull` is already tested. It should be pretty easy to come up with a list of *common* use cases, and if any of them are not covered, adding tests for them is simple and straight-forward, too.

Ciao,
Johannes