Re: [PATCH 6/6] t/README: A new section about test coverage

Ævar Arnfjörð Bjarmason wrote:

quoted

Ævar Arnfjörð Bjarmason wrote:

quoted

quoted

+ - Check the test coverage for your tests. See the "Test coverage"
+   below.

[...]

What I was going for here is that you should try to make sure that the
code you're adding is covered by tests by running the coverage tests.

I.e. if I add a new function "blah" to git-whatever which is
implemented by the "do_blah" function checking if every line of
"do_blah" is covered is an excellent indicator of whether that code is
being exhaustively tested, as opposed to just superficially tested.

In most cases a low test coverage counts is telling about the overall
quality of the tests.

But, the wording can probably be improved. Do you have a suggestion
for the above intent compressed into a sentence or two? I can't come
up with anything right now.

What I meant is that when developing a new feature, I think paying
too much attention to coverage numbers is a very dangerous thing.

It produces two hazards: too many tests and too few tests.

 - too many tests because when I write my "do_blah" function
   that is about a case no one cares about in practice, to write
   artificial tests to exercise would actually be to do harm.

 - too few tests because if I focus on testing all the code I
   just wrote, then I am very unlikely to include tests for the
   cases I did /not/ write code for.  Some important cases are
   just easy; we should still test them because that will help
   if the code is ever refactored later.  Some important cases
   may be just not implemented yet; test cases for them are
   very helpful indeed to readers and future implementers.

In other words, I would rather that when writing tests, authors would
forget about the implementation for a moment and just think about what
a user wants to do.

That said:

The rest of the time, checking test coverage does provide a very good
indication about what features might not be well tested yet.  So it is
still a good way to decide where to /start/ writing tests.

Plus it’s great fun to look at. :)

Thanks,
Jonathan