Re: [PATCH] t6018-rev-list-glob: fix 'empty stdin' test

On Wed, Aug 22, 2018 at 7:53 PM Eric Sunshine [off-list ref] wrote:

Can you say a word or two (here in the email thread) about how you're
finding these failures (across the various test fixes you've posted
recently)? Are you instrumenting the code in some fashion? Or, finding
them by visual inspection?

Errors from system commands in our tests look like these:

  grep: file3: No such file or directory
  sed: -e expression #1, char 2: extra characters after command
  diff: sub1/.git: No such file or directory
  tar: rmtlseek not stopped at a record boundary
  tar: Error is not recoverable: exiting now

while errors from the shell running the test like these:

  t0020-crlf.sh: 8: eval: cannot open two: No such file
  t6018-rev-list-glob.sh: 4: eval: cannot open expect: No such file
  t7408-submodule-reference.sh: 615: test: =: unexpected operator

i.e. lines starting with various system commands' or test scripts'
names, followed by ': '.

So I've modified t/Makefile to not remove the 'test-results' directory
after a successful 'make test':

diff --git a/t/Makefile b/t/Makefile
index ea36cf7ac7..c7b1655593 100644
--- a/t/Makefile
+++ b/t/Makefile

@@ -54,10 +54,11 @@ pre-clean:
        $(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'

 clean-except-prove-cache:
-       $(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+       $(RM) -r 'trash directory'.*
        $(RM) -r valgrind/bin

 clean: clean-except-prove-cache
+       $(RM) '$(TEST_RESULTS_DIRECTORY_SQ)'

 distclean: clean
        $(RM) .prove

And then scanned the results of a '--verbose-log -x' test run with:

  grep -E '^(awk|basename|cat|cd|chmod|cmp|cp|cut|diff|dirname|egrep|find|fgrep|grep|gunzip|gzip|ln|mkdir|mkfifo|mktemp|mv|readlink|rmdir|sed|sort|tar|touch|tr|ulimit|umask|uniq|unzip|wc|zipinfo|t[0-9][0-9][0-9][0-9]-[^:]*\.sh):
' test-results/*.out

and then, for lack of something better to do ;), I started looking at
the simpler looking errors.

I've though about how a check like this could be automated, but
haven't had any workable idea yet.  There are commands that can
legitimately print errors, e.g. when checking for a prereq which the
system doesn't have (e.g. the 'tar' errors above, I think).  And the
list of system commands in the grep pattern above is surely incomplete
and will likely change in the future...