Re: [RFC/PATCH] Makefile: add test-all target

On Wed, Dec 08, 2021 at 12:04:50PM -0800, Junio C Hamano wrote:

quoted

That is an interesting way to demonstrate how orthogonal the issues
are, which in turn means that it is not such a big deal to add back
the coverage to the part that goes to contrib/scalar/.  As the actual
implementation, it is a bit too icky, though.

So, how about doing it this way?  This is based on 'master' and does
not cover contrib/scalar, but if we want to go this route, it should
be trivial to do it on top of a merge of ab/ci-updates and js/scalar
into 'master'.  Good idea?  Terrible idea?  Not good enough?

I don't mind the general direction, but...

+# Additional tests from places in contrib/ that are prepared to take
+# "make -C $there test", but expects that the primary build is done
+# already.
+test-extra: all
+	$(MAKE) -C contrib/diff-highlight test
+	$(MAKE) -C contrib/mw-to-git test
+	$(MAKE) -C contrib/subtree test

I'm not sure of the quality of tests in some of the contrib stuff. The
tests in diff-highlight worked for me when I added them, but it's not
like I ever run them regularly, or that they've been tested on a wide
variety of platforms.

So I think this is as likely to cause somebody a headache due to a dumb
portability problem or random bitrot as it is to actually find a bug. I
guess test-extra wouldn't be run by default, but only via CI, so maybe
that limits the blast radius sufficiently.

For diff-highlight in particular, you need to have a working perl, so
you'd probably want to at least wrap it with a NO_PERL ifndef. For
mw-to-git, you need to have MediaWiki::API installed, though I think the
tests at least notice this and skip everything if you don't.

-Peff