Re: [PATCH/RFC/GSOC] make git-pull a builtin

Hi,

On Sun, Mar 22, 2015 at 1:35 AM, Johannes Schindelin
[off-list ref] wrote:

quoted

Maybe code coverage tools could help here so we only need to focus on
the code paths that are untested by the test suite. At the minimum,
all of the non-trivial code paths in both the shell script and the
converted builtin must be covered by tests. This should help to
eliminate most sources of breakages. Anything further than that would
require an experienced understanding of all the possible important
inputs to be tested, which I personally feel would make the project
quite tedious.

I see git already has gcov support. For shell scripts, maybe kcov[1]
could be used. With some slight code changes, I managed to generate a
report for the git-pull tests[2] which should at least provide a good
starting point for how the tests can be improved.

While it is often a tempting idea to make test suites as thorough as possible, there lies a true danger herein. True war story: in one of the projects I was involved in, the test suite grew to a size that one complete run lasted two weeks. Yes, that is fourteen days. Needless to say: this test suite was run rarely. How useful is a test suite that is run rarely? More useful than a non-existent one, to be sure, but it is still more of a burden than a boon.

Now, on Windows the test suite takes almost three hours to run. This really, really slows down development.

So while we are not yet at the "too large to be useful state", I would caution against trying to get there.

Instead, I would really like to focus on the *usage*. Calling `git grep "git pull" t/` should give you an idea what usage of `git pull` is already tested. It should be pretty easy to come up with a list of *common* use cases, and if any of them are not covered, adding tests for them is simple and straight-forward, too.

The code coverage tools can help here as well. The kcov output clearly
shows which options of git-pull are currently not being tested. But
yes, I agree that the test suite shouldn't be relied too much on
compared to code inspection and review.

On another important topic, though, along with git-pull.sh, I'm
looking for another script to convert in parallel with git-pull.sh so
that there will be no blocks due to patch review. Generally, I think
rewriting scripts that are called frequently by users, or spawn a lot
of processes due to loops, would be most desirable because the runtime
gains would be much higher. A quick review of the scripts shows that
git-am.sh, git-rebase--interactive.sh and git-quiltimport.sh have
pretty heavy loops with lots of process spawning that grows with
input.

I'm currently leaning with git-am because not only is it a frequently
used command, git-rebase--am.sh (for non-interactive rebase) calls it
as well. In fact, quick tests show that it takes up 98% of
git-rebase's execution time on Windows, so if git-am's performance
improves it would be a huge win on many fronts. git-am's code also
seems to be manageable for a 3-month project.

Anyway, I would like to know if you (or anyone else) have any scripts in mind.

(I also think that just 2 scripts would be enough to fill the 3
months, but that might be me just being too conservative)

Regards,
Paul