[PATCH/RFC] tests: WIP Infrastructure for Git smoke testing

Add the capability to send smoke reports to the Git test suite.

Currently we only notice bugs in the test suite when it's run
manually. Bugs in Git that only occur on obscure platforms or setups
that the core developers aren't using may thus go unnoticed until the
bug makes it into a release.

This series aims to change that. With it anyone that's interested in
avoiding bitrot in Git can volunteer to run a smoke tester. A smoke
tester periodically compiles the latest version of Git, runs the test
suite, and submits a report to a central server indicating how the
test run went.

A smoke tester might run something like this in cron:

    #!/bin/sh
    cd ~/g/git
    git fetch
    for branch in maint master next pu; do
        git checkout origin/$i &&
        make clean all &&
        cd t &&
        make smoke_report
    done

The smoker might want to compile git with non-default flags. A script
which does that is outside the scope of this patch. But it's something
we'll want to include eventually.

What this does now is add smoke and smoke_report targets to
t/Makefile:

    $ make clean smoke
    rm -f -r 'trash directory'.* test-results
    rm -f t????/cvsroot/CVSROOT/?*
    rm -f -r valgrind/bin
    rm -f .prove
    perl ./harness --git-version="1.7.2.1.173.gc9b40" \
                    --no-verbose \
                    --archive="test-results/git-smoke.tar.gz" \
                    t0000-basic.sh t0001-init.sh t0002-gitfile.sh t0003-attributes.sh t0004-unwritable.sh t0005-signals.sh t0006-date.sh
    t0000-basic.sh ....... ok
    t0001-init.sh ........ ok
    t0002-gitfile.sh ..... ok
    t0003-attributes.sh .. ok
    t0004-unwritable.sh .. ok
    t0005-signals.sh ..... ok
    t0006-date.sh ........ ok
    All tests successful.

    Test Summary Report
    -------------------
    t0000-basic.sh     (Wstat: 0 Tests: 46 Failed: 0)
      TODO passed:   5
    Files=7, Tests=134,  3 wallclock secs ( 0.06 usr  0.05 sys +  0.23 cusr  1.33 csys =  1.67 CPU)
    Result: PASS

    TAP Archive created at /home/avar/g/git/t/test-results/git-smoke.tar.gz

The smoke target uses TAP::Harness::Archive to aggregate the test
results into a tarball. The tarball contains two things, the output of
every test file that was run, and a metadata file:

Tarball contents:

    $ tar xzvf git-smoke.tar.gz
    t0004-unwritable.sh
    t0001-init.sh
    t0002-gitfile.sh
    t0005-signals.sh
    t0000-basic.sh
    t0003-attributes.sh
    t0006-date.sh
    meta.yml

A test report, this could also include --verbose output:

    $ cat t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

A metadata file:

    ---
    extra_properties:
      git:
        version: 1.7.2.1.173.gc9b40
    file_attributes:
      -
        description: t0000-basic.sh
        end_time: 1280437324.61398
        start_time: 1280437324.22186
      -
        description: t0001-init.sh
        end_time: 1280437325.12346
        start_time: 1280437324.62393
      -
        description: t0002-gitfile.sh
        end_time: 1280437325.29428
        start_time: 1280437325.13646
      -
        description: t0003-attributes.sh
        end_time: 1280437325.59678
        start_time: 1280437325.30565
      -
        description: t0004-unwritable.sh
        end_time: 1280437325.77376
        start_time: 1280437325.61003
      -
        description: t0005-signals.sh
        end_time: 1280437325.85426
        start_time: 1280437325.78727
      -
        description: t0006-date.sh
        end_time: 1280437326.2362
        start_time: 1280437325.86768
    file_order:
      - t0000-basic.sh
      - t0001-init.sh
      - t0002-gitfile.sh
      - t0003-attributes.sh
      - t0004-unwritable.sh
      - t0005-signals.sh
      - t0006-date.sh
    start_time: 1280437324
    stop_time: 1280437326

The "extra_properties" hash is where we'd stick Git-specific info,
like whether Git was compiled with gettext or the fallback regex
engine.

The entire tarball is then submitted to a central smoke
aggregator. Currently this is done with curl over HTTP, but could also
be done e.g. via E-Mail:

    curl \
        -F architecture=amd64 \
        -F platform=linux \
        -F revision="$(GIT_VERSION)" \
        -F report_file=@test-results/git-smoke.tar.gz \
        http://git.smoke.example.net/submit_report

The aggregator would make both the raw test reports available, and
might format them with a web interface. Smolder (on CPAN) is one
example of the latter. Here are smoke reports for the Rakudo project
on Smolder:

    http://smolder.plusthree.com/app/projects/smoke_reports/18

TODO:

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

  - How do I get things like PERL_PATH and uname_M from the top-level
    Makefile? Currently I just hardcode e.g. "perl" and
    "architecture=amd64". Maybe this needs to be a top-level target
    instead?

  - Set up the smoke aggregator. I was running into some issues with
    smolder, but those are solvable given some time. I can run and
    maintain the smoke aggregator if this gets accepted.

Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
 t/Makefile |   20 ++++++++++++++++++++
 t/harness  |   36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 0 deletions(-)
 create mode 100755 t/harness

diff --git a/t/Makefile b/t/Makefile
index 819b936..459a667 100644
--- a/t/Makefile
+++ b/t/Makefile

@@ -49,4 +49,24 @@ full-svn-test:
 valgrind:
 	GIT_TEST_OPTS=--valgrind $(MAKE)
 
+test-results:
+	mkdir -p test-results
+
+-include ../GIT-VERSION-FILE
+test-results/git-smoke.tar.gz:
+	perl ./harness --git-version="$(GIT_VERSION)" \
+		--no-verbose \
+		--archive="test-results/git-smoke.tar.gz" \
+		$(wildcard t000[0-9]-*.sh)
+
+smoke: test-results/git-smoke.tar.gz
+
+smoke_report: smoke
+	curl \
+		-F architecture=amd64 \
+		-F platform=linux \
+		-F revision="$(GIT_VERSION)" \
+		-F report_file=@test-results/git-smoke.tar.gz \
+		http://git.smoke.example.net/submit_report
+
 .PHONY: pre-clean $(T) aggregate-results clean valgrind

diff --git a/t/harness b/t/harness
new file mode 100755
index 0000000..72b4af6
--- /dev/null
+++ b/t/harness

@@ -0,0 +1,36 @@
+#!/usr/bin/perl
+use 5.010;
+use strict;
+use warnings;
+use Pod::Usage ();
+use Getopt::Long ();
+use TAP::Harness::Archive;
+
+Getopt::Long::Parser->new(
+    config => [ qw/ pass_through / ],
+)->getoptions(
+    'h|help'        => \my $help,
+    'jobs:1'        => \my $jobs,
+    'verbose!'      => \(my $verbose = 1),
+    'git-version=s' => \my $git_version,
+    'archive=s'     => \my $archive,
+) or die "$0: Couldn't getoptions()";
+
+my @tests = @ARGV;
+
+my $harness = TAP::Harness::Archive->new({
+    verbosity => 0,
+    jobs      => $ENV{TEST_JOBS} || $jobs || 1,
+    test_args => [
+        ($verbose
+         ? qw/ --verbose /
+         : ())
+    ],
+    archive   => $archive,
+    extra_properties => {
+        git => {
+            version => $git_version,
+        },
+    }
+});
+$harness->runtests(@tests);

-- 
1.7.1

Ævar Arnfjörð Bjarmason wrote:

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

I'm all for it!

I think I could put a cronjob on a RHEL5.4 machine, and regularly run
it on my own openSuSE 11.3 install.

Tarball contents:

    $ tar xzvf git-smoke.tar.gz
    t0004-unwritable.sh
    t0001-init.sh
    t0002-gitfile.sh
    t0005-signals.sh
    t0000-basic.sh
    t0003-attributes.sh
    t0006-date.sh
    meta.yml

That's the only thing I found confusing about this: why are they named
as if they were shellscripts?

Also, installing TAP::Harness::Archive was a bit hairy but that's
CPAN's fault...

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Ævar Arnfjörð Bjarmason wrote:

Currently we only notice bugs in the test suite when it's run
manually. Bugs in Git that only occur on obscure platforms or setups
that the core developers aren't using may thus go unnoticed until the
bug makes it into a release.

BTW, below is the script I have devised to automatically run the tests
under valgrind and then bisect any offenders.  (It keeps exposing odd
test failures and stealing my time tracking them down...)

It gives fairly good results, but since it's also rather kludgy I want
to test and improve it a bit more before shooting for contrib (if
anything).

Note that running this will take on the order of hours at least.

---- 8< ----
#!/bin/sh

MAILTO=tr@thomasrast.ch

branch="${1:-next}"
logfile="$(pwd)/bisection-log-$$"

export DIFF=diff # stupid bug

die () {
	(echo "Error message: $*"; echo '-----'; cat "$logfile") |
	mail -s "Testing $branch: die() called!" "$MAILTO"
	echo "fatal: $*" >&2
	exit 1
}

git fetch origin 2>&1 | tee "$logfile" || die "fetch failed"

# test if we've already done this
test "$(git rev-parse last_known_good_$branch)" = "$(git rev-parse origin/$branch)" && exit

git checkout -f origin/"$branch" 2>&1 | tee "$logfile" || die "checkout failed (can't happen)"

results="test-results.$(git describe)"
mkdir "$results"

make -j12 2>&1 | tee -a "$logfile" || die "initial compilation failed?"
make -k -j8 test 2>&1 | tee -a "$logfile"

if [ ! -d t/"test-results" ]; then
    mail -s "Valgrind-tested $branch: all good!" "$MAILTO" < "$logfile"
    git tag -f last_known_good_$branch origin/$branch
    rm "$logfile"
    exit
fi

cd t/
cp -a test-results "$results"
( cd $results && perl -i -ne 'print unless /^==.*execve/i' *.out )
failing_normally=$(cd "$results" && grep -L '^0$' *.exit)
failing_valgrind=$(cd "$results" && grep -lE '^==[0-9]+==' *.out)
cd ..

echo "failed (normal):" $failing_normally | tee -a "$logfile"
echo "failed (valgrind):" $failing_valgrind | tee -a "$logfile"

if [ -z "$failing_normally" -a -z "$failing_valgrind" ]; then
    mail -s "Valgrind-tested $branch: all good!" "$MAILTO" < "$logfile"
    git tag -f last_known_good_$branch origin/$branch
    rm "$logfile"
    exit
fi

mail -s "Valgrind-testing $branch: starting test bisection" "$MAILTO" < "$logfile"

for test_out in $failing_valgrind; do
    test="${test_out%.out}"
    echo "bisecting valgrind-failing test $test"
    git bisect start > t/"$results"/$test.bisect 2>&1
    git bisect good last_known_good_pu last_known_good_next 2>&1 | tee t/"$results"/$test.bisect
    git bisect bad origin/$branch 2>&1 | tee -a t/"$results"/$test.bisect
    git bisect run sh -c "test ! -f t/$test.sh || { make -j12 && cd t && ./$test.sh --valgrind --tee -i && ! grep -E '^==[0-9]+==' test-results/$test.out | grep -vi execve; }" 2>&1 | tee -a t/"$results"/$test.bisect
    git bisect log 2>&1 | tee -a t/"$results"/$test.bisect
    git bisect reset
    mail -s "Bisection results for $test (valgrind)" "$MAILTO" < t/"$results"/$test.bisect
done

for test_out in $failing_normally; do
    test="${test_out%.exit}"
    case "$failing_valgrind" in
	*$test.out*)
	    continue
	    ;;
    esac
    echo "bisecting failing test $test"
    git bisect start 2>&1 | tee t/"$results"/$test.bisect 2>&1
    git bisect good last_known_good_pu last_known_good_pu 2>&1 | tee -a t/"$results"/$test.bisect 2>&1
    git bisect bad origin/$branch 2>&1 | tee -a t/"$results"/$test.bisect 2>&1
    git bisect run sh -c "test ! -f t/$test.sh || { make -j12 && cd t && ./$test.sh --tee -i; }" 2>&1 | tee -a t/"$results"/$test.bisect 2>&1
    git bisect log | tee -a t/"$results"/$test.bisect
    git bisect reset
    mail -s "Bisection results for $test" "$MAILTO" < t/"$results"/$test.bisect
done

rm "$logfile"
---- >8 ----

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

On Thu, Jul 29, 2010 at 22:11, Thomas Rast [off-list ref] wrote:

Ævar Arnfjörð Bjarmason wrote:

quoted

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

I'm all for it!

I think I could put a cronjob on a RHEL5.4 machine, and regularly run
it on my own openSuSE 11.3 install.

That's at least three smokers there. And since your RHEL5.4 machine
already turned up a gettext bug..

quoted

Tarball contents:

    $ tar xzvf git-smoke.tar.gz
    t0004-unwritable.sh
    t0001-init.sh
    t0002-gitfile.sh
    t0005-signals.sh
    t0000-basic.sh
    t0003-attributes.sh
    t0006-date.sh
    meta.yml

That's the only thing I found confusing about this: why are they named
as if they were shellscripts?

The output file names just match the test names, which in this case
are shellscripts.

Also, installing TAP::Harness::Archive was a bit hairy but that's
CPAN's fault...

This is something we can probably make really easy if we write some
supporting scripts with cpanminus + local::lib. I.e. just have a
config file like:

  ~/.gitsmoker:
    repository = ~/g/git
    data       = ~/g/git-smoker

Where the git-smoker would maintain the state, dependencies etc. It
needs.

On Thu, Jul 29, 2010 at 09:20:55PM +0000, Ævar Arnfjörð Bjarmason wrote:

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

A few of us were running automated build/tests for a while. Check out:

  http://repo.or.cz/w/git/gitbuild.git

especially:

  http://repo.or.cz/w/git/gitbuild.git/tree/platform

Unfortunately the SunOS and FreeBSD machines I tested on went away, so I
am not actually running anything automated anymore. Mike Ralphson was
testing on AIX, but seems to have stopped, as well.

I think these days Junio runs the test suite through a couple of VMs,
but I don't know exactly which platforms, or how often he does so.

-Peff

On Fri, Jul 30, 2010 at 12:11:50AM +0200, Thomas Rast wrote:

Ævar Arnfjörð Bjarmason wrote:

quoted

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

I'm all for it!

Me too! I should be able to setup a cronjob for tests under a Windows XP
machine. Does that count as an obscure platform ? ;)

cheers Heiko

On Sun, Aug 8, 2010 at 13:42, Heiko Voigt [off-list ref] wrote:

On Fri, Jul 30, 2010 at 12:11:50AM +0200, Thomas Rast wrote:

quoted

Ćvar Arnfjörđ Bjarmason wrote:

quoted

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

I'm all for it!

Me too! I should be able to setup a cronjob for tests under a Windows XP
machine. Does that count as an obscure platform ? ;)

I think it does :)

I just sent another version of the series to the list. It'd be very
useful if you could try sending a smoke report from Windows XP.

Thanks.

On Sun, Aug 08, 2010 at 02:54:21PM +0000, Ævar Arnfjörð Bjarmason wrote:

On Sun, Aug 8, 2010 at 13:42, Heiko Voigt [off-list ref] wrote:

quoted

On Fri, Jul 30, 2010 at 12:11:50AM +0200, Thomas Rast wrote:

quoted

Ćvar Arnfjörđ Bjarmason wrote:

quoted

  - Is this worthwhile. Are there developers / packagers / other
    interested parties here who'd be interested in actually running
    smoke testers? It should be really easy to set one up.

I'm all for it!

Me too! I should be able to setup a cronjob for tests under a Windows XP
machine. Does that count as an obscure platform ? ;)

I think it does :)

I just sent another version of the series to the list. It'd be very
useful if you could try sending a smoke report from Windows XP.

I tried to do it yesterday but unfortunately some of the tests seem to
hang on Windows so I could not complete the report. I will try to
generate a skip list and run the smoke_report again later.

cheers Heiko