Re: [PATCH v5] unit tests: Add a project plan document

On 2023.08.14 14:29, Phillip Wood wrote:

Hi Josh

On 08/08/2023 00:07, Josh Steadmon wrote:

quoted

Reviewers can help this series progress by discussing whether it's
acceptable to rely on `prove` as a test harness for unit tests. We
support this for the current shell tests suite, but it is not strictly
required.

If possible it would be good to be able to run individual test programs
without a harness as we can for our integration tests. For running more than
one test program in parallel I think it is fine to require a harness.

Sounds good. This is working in v6 which I hope to send to the list
soon.

I don't have a strong preference for which harness we use so long as it
provides a way to (a) run tests that previously failed tests first and (b)
run slow tests first. I do have a strong preference for using the same
harness for both the unit tests and the integration tests so developers
don't have to learn two different tools. Unless there is a problem with
prove it would probably make sense just to keep using that as the project
test harness.

To be clear, it sounds like both of these can be done with `prove`
(using the various --state settings) without any further support from
our unit tests, right? I see that we do have a "failed" target for
re-running integration tests, but that relies on some test-lib.sh
features that currently have no equivalent in the unit test framework.

quoted

TODOs remaining:
- Discuss pre-existing harnesses for the current test suite
- Figure out how to evaluate frameworks on additional OSes such as *BSD
   and NonStop

We have .cirrus.yml in tree which I think gitgitgadget uses to run our test
suite on freebsd so we could leverage that for the unit tests. As for
NonStop I think we'd just have to rely on Randall running the tests as he
does now for the integration tests.

Thanks for the pointer. I've updated .cirrus.yml (as well as the GitHub
CI configs) for v6.

quoted

Changes in v5:
- Add comparison point "License".
- Discuss feature priorities
- Drop frameworks:
   - Incompatible licenses: libtap, cmocka
   - Missing source: MyTAP
   - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit
- Drop comparison point "Coverage reports": this can generally be
   handled by tools such as `gcov` regardless of the framework used.
- Drop comparison point "Inline tests": there didn't seem to be
   strong interest from reviewers for this feature.
- Drop comparison point "Scheduling / re-running": this was not
   supported by any of the main contenders, and is generally better
   handled by the harness rather than framework.
- Drop comparison point "Lazy test planning": this was supported by
   all frameworks that provide TAP output.

These changes all sound sensible to me

The trimmed down table is most welcome. The custom implementation supports
partial parallel execution. It shouldn't be too difficult to add some signal
handling to it depending on what we want it to do.

It sounds like we're getting to the point where we have pinned down our
requirements and the available alternatives well enough to make a decision.

Yes, v6 will include your TAP implementation (I assume you are still OK
if I include your patch in this series?).

Best Wishes

Phillip