Re: [PATCH 0/6 + 1] test-lib: make --verbose output valid TAP
From: SZEDER Gábor <hidden>
Date: 2021-03-09 22:07:54
On Tue, Mar 09, 2021 at 10:03:54PM +0100, Ævar Arnfjörð Bjarmason wrote:
On Tue, Mar 09 2021, SZEDER Gábor wrote:quoted
On Tue, Mar 09, 2021 at 05:02:12PM +0100, Ævar Arnfjörð Bjarmason wrote:quoted
The eventual goal not included in this series is to have multiple output targets, and e.g. convert ci/print-test-failures.sh to use a TAP parser. Machine-readable "TAP --verbose -x" output can bring us a lot of nice things down the line, I have some local WIP code that's a smarter version of ci/print-test-failures.sh that knows how to spew out only the output relevant to the failing test(s).I wonder what you mean by the word "relevant" here, as I can't imagine how you could possibly identify what is relevant for a failing test and what isn't. If you didn't at all meant "relevant", but that it will show only the output of the failing test(s), then this is a bad example. Our test cases depend too much on previous test cases, and a failure of one test can be the result of a change in a previous successful tests. Therefore, any such change to 'ci/print-test-failures.sh' will have my firm NACK.On e.g. the github CI every step in the run is an collapsable button, so we could have our cake and eat it too here.
Well, one of the things I didn't like in GitHub CI is that I had to click a lot to get to the information I wanted...
It seems to me like a sane default would be to have an equivalent to "print-test-failures.sh" that only prints the --verbose output for the failing tests, and a "print-all-output-for-failing-tests.sh" or whatever which gave you the full output. Even then, there seem to me to be some low hanging fruit for abbreviating even that output. E.g. if you we have 100 tests and we failed only on the 5th, isn't the --verbose -x output up to and including the 5th going to be enough, or do we need it for the other 95?
I have a test fix waiting to be sent out, where a failing git command in the 'test_when_finished' block of test #21 went unnoticed because of the broken && chain, and ultimately caused the failure of test #91. In my opinion your proposed changes to 'ci/print-test-failures.sh' would make things worse.