On Wed, Dec 02, 2020 at 12:01:49PM +0100, Han-Wen Nienhuys wrote:

quoted

I'm not sure if the long-term goal is to have this opaque unit-test
program or not. If it is, I was likewise going to suggest that its
ad-hoc output be replaced with TAP. But it looks like on your branch
that "test-tool reftable" does not produce output at all. So I may be a
bit behind on what the current state and forward plans are...

The most important requirement is that something fails if the
unittests don't work. I surmised that that meant running tests from
test-helper in some way, so this is what happens now. Looking for
"unit.?test" across the git codebase didn't turn up much info. Happy
to explore other solutions if you can give me pointers.

Normally we do a combination of:

  - git plumbing exposes scriptable commands, and we make sure those
    work. This is a much coarser-grained unit than testing individual C
    functions, but produces resilient tests because that interface is
    user-visible and stable (and in fact seeing test breakages is often
    a sign that one will be breaking real users).

    These are obviously driven by shell script tests emulating what
    users might run.

  - for unit tests of individual data types where it's not appropriate
    to have a user-facing command, we often add a test-tool helper that
    exposes specific functions (e.g., t/helper/test-date.c exposes date
    parsing and formatting routines, test-oid-array.c exercises methods
    of that data type, etc).

    The C parts of these are usually generic, and then are driven by
    shell scripts providing the actual data in individual tests (and
    handling success/failure, skipping tests, etc).

    This is still coarser than you might get unit-testing inside C.
    E.g., you could not generally check the difference between passing
    an empty array vs NULL to a function. But our philosophy has
    generally been _not_ to test at that level. The C interfaces are
    internal, and Git is not that big a project. If there's a function
    whose caller does something unexpected, it's usually easier to fix
    the caller and add a test that triggers the caller's code path.

I agree that a test that simply runs a bunch of C code and either exits
with failure or success is better than nothing, in the sense of finding
tests. And wrapping that with a single test_expect_success does that.
But it's unfortunate that we get none of the fine-grained control that
the test suite provides, nor much support in debugging failing tests.

One middle ground would be for a battery of C tests to output
TAP-compatible output (outputting "ok 1 - foo works", and "not ok 2 -
bar does not work", etc). That at least gives more info on what fails,
and does it in a way that the rest of the test suite can interpret
(though not manipulate).

-Peff