Hi Eric,

On Wed, 5 Sep 2018, Eric Sunshine wrote:

On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin
[off-list ref] wrote:

quoted

So let's hear some ideas how to improve the situation, m'kay?  Just as
a reminder, this is the problem I want to solve: I want to run the
tests in a light-weight manner, with minimal output, and only in case
of an error do I want to crank up the verbosity. Instead of wasting
most of the effort to log everything and then throwing it away in most
of the common cases, I suggest to re-run the entire test.

What about the very different approach of capturing the full "verbose"
output the executed tests in addition to whatever is actually output
to the terminal?

I fear it is not really possible to do a "verbose but not really" mode. I
want the console output to be quiet, there is no use in chatting up the
build log with these messages, as we have to run the tests in parallel, so
the output would be utterly hard to interpret anyway. At the same time, I
want verbose output for use in the test results. It is not really possible
to `tee` all output, then "quiet down" the part that makes it into the
log.

If a test fails, then (and only then) you can insert the captured
verbose output into the JUnit XML file.

Yep. That's what my patch series does.

If a test fails, then (and only then) I re-run the script with verbose
output that is immediately moved into that JUnit XML file.

This way (if we always have the full verbose output at hand), you don't
need to re-run the test at all.

But that way, if we always have the full verbose output, the build log
will be unnecessarily verbose and confusing.

I've cc:'d Peff and Jonathan since I believe they are more familiar
with how all the capturing / output-redirection works in the test
suite.

Okay.

Ciao,
Dscho

P.S.: An unintended side effect of re-running the tests is to identify
flakey tests. I do not yet have a way to represent this outcome in the
test result, but I deem this an additional benefit in favor of keeping my
current strategy.