Re: [PATCH 6/7] test-lib: make --verbose output valid TAP

On Wed, Mar 10 2021, Ævar Arnfjörð Bjarmason wrote:

On Tue, Mar 09 2021, SZEDER Gábor wrote:

quoted

On Tue, Mar 09, 2021 at 09:57:03PM +0100, Ævar Arnfjörð Bjarmason wrote:

[...]

quoted

quoted

Becuse the advantage of this series is that that sort of thing becomes
really trivial without everything needing to be hardcoded in
test-lib.sh, observe (this is with my series):

    
    0 $ PERL5LIB=. prove -v --formatter=SZEDERVerboseLog ./t0201-gettext-fallbacks.sh :: --verbose-log

Well, this doesn't look trivial at all, does it?  In fact, I consider
this unusably convoluted.

... I meant to say something closer to "does that output look
ok?". Obviously we'd then make the --verbose-log run something like that
under the hood.

But in any case, I found it easier to just add this feature to my "tee"
program than doing it with Perl's TAP libraries, i.e. something like
this on top (will integrate it in an eventual re-roll):

FWIW I implemented this. I've got it at https://github.com/avar.git's
avar/support-test-verbose-under-prove-2 if you want to take an early
peek at it. It turned into a 50-some patch slog through refactoring
various test-lib.sh code, so I'm seeing about how I submit it.

The end result is that -V now powered by a proper TAP-parsing tap-tee
program. So you'll have e.g. this working as before:
    
    $ ./t9004-example.sh -V
    ok 1 - decorate
    # passed all 1 test(s)
    1..1

But you can also tweak levels of verbosity to stdout (while retaining
full logging) by suppling -V N to get N-level comments, as now indicated
by N number of prefixed "#"'s:

   # -V0 is a special-case to get 0th level logging (couldn't think of a
   # better name), --verbose-log=0 also works
   $ ./t9004-example.sh -V0
   ok 1 - decorate
   1..1

At level 3 you start getting the source of passing tests (failing tests
are level 1, unexpected TODO tests level 2, trace level 4):
    
    $ ./t9004-example.sh -VVV 
    ok 1 - decorate
    ###
    ###     test-tool example-decorate
    ###
    # passed all 1 test(s)
    1..1

So obviously the devil's in the details, but it would be useful to know
what you think about the approach before I try to roll this into
submitted patches sooner than later.

I'm mainly interesetd in this for this providing building blocks for
more advanced features down the road.

E.g. it would now be pretty easy to do something like run -x behind the
scenes even if the user doesn't ask for it, and then when a test fail
annotate the specific line that failed using fuzzy-matching on the -x
output.

We could do that now, sort of, but once we've got unambiguous TAP
start/end markers that can't be fooled it becomes trivial to write a
function like get_trace_lines_for_nth_test(N-1) to get your own trace
output for test N.