Re: [PATCH v2 7/7] wildmatch test: create & test files on disk in addition to in-memory

On Monday 25 December 2017 at 12:28 am +0000, Ævar Arnfjörð Bjarmason wrote:

There has never been any full roundtrip testing of what git-ls-files
and other functions that use wildmatch() actually do, rather we've
been satisfied with just testing the underlying C function.

Due to git-ls-files and friends having their own codepaths before they
call wildmatch() there's sometimes differences in the behavior between
the two, and even when we test for those (as with
9e4e8a64c2 ("pathspec: die on empty strings as pathspec", 2017-06-06))
there was no one place where you can review how these two modes
differ.

Now there is. We now attempt to create a file called $haystack and
match $needle against it for each pair of $needle and $haystack that
we were passing to test-wildmatch.

If we can't create the file we skip the test. This ensures that we can
run this on all platforms and not maintain some infinitely growing
whitelist of e.g. platforms that don't support certain characters in
filenames.

As a result of doing this we can now see the cases where these two
ways of testing wildmatch differ:

 * Creating a file called 'a[]b' and running ls-files 'a[]b' will show
   that file, but wildmatch("a[]b", "a[]b") will not match

 * wildmatch() won't match a file called \ against \, but ls-files
   will.

 * `git --glob-pathspecs ls-files 'foo**'` will match a file
   'foo/bba/arr', but wildmatch won't, however pathmatch will.

   This seems like a bug to me, the two are otherwise equivalent as
   these tests show.

This also reveals the case discussed in 9e4e8a64c2 above, where '' is
now an error as far as ls-files is concerned, but wildmatch() itself
happily accepts it.

Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>

I'm seeing this test script failing on the pu branch as a result of this
commit when building on Cygwin.  Specifically, the test fails at
9d45e1ca4 ("Merge branch 'bw/oidmap-autoinit' into pu", 2017-12-28), and
bisecting points the blame at 2ee0c785a ("wildmatch test: create & test
files on disk in addition to in-memory", 2017-12-25).

I've copied the verbose error output for the first error below, and
uploaded the full output, including verbose and trace output for the
unexpectedly failing tests, at [0].  (With 42 failures among 1512 tests,
there's a lot of it, so I didn't want to include it in an email.)

    expecting success:
                                    wildtest_file_setup &&
                                    printf '%s' '\' >expect &&
                                    git --glob-pathspecs ls-files -z -- '00' >actual.raw 2>actual.err &&
                                    wildtest_stdout_stderr_cmp
    
    ++ wildtest_file_setup
    ++ test_when_finished '
                    rm -rf -- * &&
                    git reset
            '
    ++ test 0 = 0
    ++ test_cleanup='{
                    rm -rf -- * &&
                    git reset
    
                    } && (exit "$eval_ret"); eval_ret=$?; :'
    ++ git add -A
    ++ printf %s '\'
    ++ git --glob-pathspecs ls-files -z -- 00
    ++ wildtest_stdout_stderr_cmp
    ++ tr -d '\0'
    ++ test_cmp expect.err actual.err
    ++ diff -u expect.err actual.err
    ++ test_cmp expect actual
    ++ diff -u expect actual
    --- expect      2018-01-03 12:43:11.116611200 +0000
    +++ actual      2018-01-03 12:43:11.216180400 +0000
    @@ -1 +0,0 @@
    -\
    \ No newline at end of file
    + test_eval_ret_=1
    + want_trace
    + test t = t
    + test t = t
    + set +x
    error: last command exited with $?=1
    ++ rm -rf -- actual actual.err actual.raw expect expect.err
    ++ git reset
    ++ exit 1
    ++ eval_ret=1
    ++ :
    + test_eval_ret_=0
    + want_trace
    + test t = t
    + test t = t
    + set +x
    not ok 490 - wildmatch(ls): match '00' '\'
    #
    #                                       wildtest_file_setup &&
    #                                       printf '%s' '\' >expect &&
    #                                       git --glob-pathspecs ls-files -z -- '00' >actual.raw 2>actual.err &&
    #                                       wildtest_stdout_stderr_cmp
    #

I'm digging into the failures myself now, but wanted to report the
problem in the name of getting more eyes on it.

Adam

[0]: https://gist.github.com/me-and/04443bcb00e12436f0eacce079b56d02