Re: [PATCH] t/known-leaky: add list of known-leaky test scripts

Martin Ågren [off-list ref] writes:

On 19 February 2018 at 22:29, Jeff King [off-list ref] wrote:
...

quoted

Or alternatively, we could just not bother with checking this into the
repository, and it becomes a local thing for people interested in
leak-testing. What's the value in having a shared known-leaky list,
especially if we don't expect most people to run it.

This sums up my feeling about this.

Even though keeping track of list of known-leaky tests may not be so
interesting, we can still salvage useful pieces from the discussion
and make them available to developers, e.g.  something like

    prove --dry --state=failed |
    perl -lne '/^(t[0-9]{4})-.*\.sh$/ and print $1' | sort >$@+
    if cmp >/dev/null $@ $@+; then rm $@+; else mv $@+ $@; fi

could be made into a target to stash away the list of failing tests
after a test run?

On Wed, Feb 21, 2018 at 08:53:16AM -0800, Junio C Hamano wrote:

Even though keeping track of list of known-leaky tests may not be so
interesting, we can still salvage useful pieces from the discussion
and make them available to developers, e.g.  something like

    prove --dry --state=failed |
    perl -lne '/^(t[0-9]{4})-.*\.sh$/ and print $1' | sort >$@+
    if cmp >/dev/null $@ $@+; then rm $@+; else mv $@+ $@; fi

could be made into a target to stash away the list of failing tests
after a test run?

Unfortunately there are some caveats in that snippet:

  1. You are using prove.

  2. You are using --state=save in the initial run.

I think we might be better off having the test scripts write to
test-results/*.counts even when run under a TAP harness, and then we can
have a consistent way to get the list of failed tests (we already have a
"make failed" that works this way).

-Peff