[PATCH v1 0/3] travis-ci: run Git bisect on failed tests

From: Lars Schneider <redacted>

This patch series run Git bisect on failing tests on Travis CI.

Example output on Travis CI:
https://travis-ci.org/larsxschneider/git/jobs/132049871

Plaintext example output:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/132049871/log.txt

Please scroll all the way down to see the bisect output.

Cheers,
Lars


Lars Schneider (3):
  travis-ci: move "after_failure" code to dedicated file in /ci
  travis-ci: disable verbose test output
  travis-ci: run Git bisect on failed tests

 .travis.yml       | 14 ++--------
 ci/test-report.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 12 deletions(-)
 create mode 100755 ci/test-report.sh

--
2.5.1

From: Lars Schneider <redacted>

The verbose output clutters the Travis CI webview and is not really
useful since test debugging usually happens on a local machine.

Signed-off-by: Lars Schneider <redacted>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index a93ecb3..81d2027 100644
--- a/.travis.yml
+++ b/.travis.yml

@@ -29,7 +29,7 @@ env:
     - LINUX_GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
-    - GIT_TEST_OPTS="--verbose --tee"
+    - GIT_TEST_OPTS="--tee"
     - GIT_TEST_CLONE_2GB=YesPlease
     # t9810 occasionally fails on Travis CI OS X
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X

-- 
2.5.1

From: Lars Schneider <redacted>

Move the code and adjust it to the Git shell script coding guidelines.

Signed-off-by: Lars Schneider <redacted>
---
 .travis.yml       | 12 +-----------
 ci/test-report.sh | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 11 deletions(-)
 create mode 100755 ci/test-report.sh

diff --git a/.travis.yml b/.travis.yml
index adab5b8..a93ecb3 100644
--- a/.travis.yml
+++ b/.travis.yml

@@ -101,17 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    for TEST_EXIT in t/test-results/*.exit;
-    do
-      if [ "$(cat "$TEST_EXIT")" != "0" ];
-      then
-        TEST_OUT="${TEST_EXIT%exit}out";
-        echo "------------------------------------------------------------------------";
-        echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)";
-        echo "------------------------------------------------------------------------";
-        cat "${TEST_OUT}";
-      fi;
-    done;
+    ./ci/test-report.sh
 
 notifications:
   email: false

diff --git a/ci/test-report.sh b/ci/test-report.sh
new file mode 100755
index 0000000..d08a999
--- /dev/null
+++ b/ci/test-report.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+#
+# Print test results
+#
+for TEST_EXIT in t/test-results/*.exit
+do
+	if test "$(cat "$TEST_EXIT")" != "0"
+	then
+		TEST="${TEST_EXIT%.exit}"
+		TEST_OUT="${TEST}.out"
+		echo "------------------------------------------------------------------------"
+		echo "  $(tput setaf 1)${TEST} Output$(tput sgr0)"
+		echo "------------------------------------------------------------------------"
+		cat "$TEST_OUT"
+		echo ""
+		echo ""
+	fi
+done
+

-- 
2.5.1

From: Lars Schneider <redacted>

Junio usually pushes many commits at once to the public "pu"/"next"/
"master" branches. If a test fails then it is not obvious what commit
caused the failure. Therefore we run Git bisect with the merge base
between the failing rev and its more stable branch ("next" for "pu",
"master" for "next", and "maint" for "master") as good ref to find the
offending commit. This is only enabled on "github.com/git/git" because
there we can assume that all relevant branches are up to date.

Signed-off-by: Lars Schneider <redacted>
---
 .travis.yml       |  2 +-
 ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 81d2027..922807b 100644
--- a/.travis.yml
+++ b/.travis.yml

@@ -101,7 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    ./ci/test-report.sh
+    ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH;
 
 notifications:
   email: false

diff --git a/ci/test-report.sh b/ci/test-report.sh
index d08a999..8f7adad 100755
--- a/ci/test-report.sh
+++ b/ci/test-report.sh

@@ -1,5 +1,12 @@
 #!/bin/sh
 #
+# Print test results and run Git bisect on failed tests.
+#
+REPO_ORG_NAME=$1
+CURRENT_BRANCH_NAME=$2
+
+
+#
 # Print test results
 #
 for TEST_EXIT in t/test-results/*.exit

@@ -17,3 +24,61 @@ do
 	fi
 done
 
+
+#
+# Run Git bisect
+#
+run_bisect () {
+	TEST_SCRIPT=$1
+	BAD_REV=$2
+	GOOD_RV=$3
+	TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+	cat > "$TMPDIR/bisect-run.sh" <<EOF
+
+EOF
+	chmod +x "$TMPDIR/bisect-run.sh"
+	git bisect start $BAD_REV $GOOD_RV
+	git bisect run "$TMPDIR/bisect-run.sh"
+	if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+	then
+		cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+	else
+		# If the test file does not exist or the build fails then tell
+		# Git bisect to skip the commit.
+		exit 125
+	fi
+	git bisect reset >/dev/null 2>&1
+}
+
+case "$CURRENT_BRANCH_NAME" in
+	master) STABLE_BRANCH="maint";;
+	next)   STABLE_BRANCH="master";;
+	pu)     STABLE_BRANCH="next";;
+esac
+
+if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
+then
+	BAD_REV=$(git rev-parse HEAD)
+
+	# Travis CI clones are shallow. It is possible that the last good revision
+	# was not fetched, yet. Therefore we need to fetch all commits on the
+	# stable branch.
+	git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
+	git fetch --unshallow --quiet
+	LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
+
+	for TEST_EXIT in t/test-results/*.exit
+	do
+		if test "$(cat "$TEST_EXIT")" != "0"
+		then
+			TEST="${TEST_EXIT%.exit}"
+			TEST_SCRIPT=${TEST#t/test-results/}
+			echo "------------------------------------------------------------------------"
+			echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
+			echo "------------------------------------------------------------------------"
+			run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
+			echo ""
+			echo ""
+		fi
+	done
+fi

-- 
2.5.1

On Sun, May 22, 2016 at 1:00 PM,  [off-list ref] wrote:

[...]

+#
+# Run Git bisect
+#
+run_bisect () {
+       TEST_SCRIPT=$1
+       BAD_REV=$2
+       GOOD_RV=$3
+       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+       cat > "$TMPDIR/bisect-run.sh" <<EOF
+
+EOF
+       chmod +x "$TMPDIR/bisect-run.sh"
+       git bisect start $BAD_REV $GOOD_RV
+       git bisect run "$TMPDIR/bisect-run.sh"
+       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+       then
+               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+       else
+               # If the test file does not exist or the build fails then tell
+               # Git bisect to skip the commit.
+               exit 125
+       fi

Shouldn't all the above "if ... fi" be in the here document creating
"$TMPDIR/bisect-run.sh"?

+       git bisect reset >/dev/null 2>&1
+}

On 22 May 2016, at 17:35, Christian Couder [off-list ref] wrote:

On Sun, May 22, 2016 at 1:00 PM,  [off-list ref] wrote:

[...]

quoted

+#
+# Run Git bisect
+#
+run_bisect () {
+       TEST_SCRIPT=$1
+       BAD_REV=$2
+       GOOD_RV=$3
+       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+       cat > "$TMPDIR/bisect-run.sh" <<EOF
+
+EOF
+       chmod +x "$TMPDIR/bisect-run.sh"
+       git bisect start $BAD_REV $GOOD_RV
+       git bisect run "$TMPDIR/bisect-run.sh"
+       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+       then
+               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+       else
+               # If the test file does not exist or the build fails then tell
+               # Git bisect to skip the commit.
+               exit 125
+       fi

Shouldn't all the above "if ... fi" be in the here document creating
"$TMPDIR/bisect-run.sh"?

Ohh. Absolutely... I wonder what happened. I'll post a v2.

Thank you,
Lars

quoted

+       git bisect reset >/dev/null 2>&1
+}

Hey Lars,

On Sun, May 22, 2016 at 4:30 PM,  [off-list ref] wrote:

If you are doing a re-roll, then you could probably fix the style issue.
' cat >"$TMPDIR..."  '

+
+EOF
+       chmod +x "$TMPDIR/bisect-run.sh"
+       git bisect start $BAD_REV $GOOD_RV
+       git bisect run "$TMPDIR/bisect-run.sh"
+       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+       then
+               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+       else
+               # If the test file does not exist or the build fails then tell
+               # Git bisect to skip the commit.
+               exit 125
+       fi
+       git bisect reset >/dev/null 2>&1
+}
+
+case "$CURRENT_BRANCH_NAME" in
+       master) STABLE_BRANCH="maint";;
+       next)   STABLE_BRANCH="master";;
+       pu)     STABLE_BRANCH="next";;
+esac
+
+if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
+then
+       BAD_REV=$(git rev-parse HEAD)
+
+       # Travis CI clones are shallow. It is possible that the last good revision
+       # was not fetched, yet. Therefore we need to fetch all commits on the
+       # stable branch.
+       git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
+       git fetch --unshallow --quiet
+       LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
+
+       for TEST_EXIT in t/test-results/*.exit
+       do
+               if test "$(cat "$TEST_EXIT")" != "0"
+               then
+                       TEST="${TEST_EXIT%.exit}"
+                       TEST_SCRIPT=${TEST#t/test-results/}
+                       echo "------------------------------------------------------------------------"
+                       echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
+                       echo "------------------------------------------------------------------------"
+                       run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
+                       echo ""
+                       echo ""
+               fi
+       done
+fi

Regards,
Pranit Bauva

On 22 May 2016, at 19:21, Pranit Bauva [off-list ref] wrote:

Hey Lars,

On Sun, May 22, 2016 at 4:30 PM,  [off-list ref] wrote:

quoted

From: Lars Schneider <redacted>

...

quoted

+
+#
+# Run Git bisect
+#
+run_bisect () {
+       TEST_SCRIPT=$1
+       BAD_REV=$2
+       GOOD_RV=$3
+       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+       cat > "$TMPDIR/bisect-run.sh" <<EOF

If you are doing a re-roll, then you could probably fix the style issue.
' cat >"$TMPDIR..."  '

Good eye :-) I will fix this, too.

Thank you,
Lars

On Sun, May 22, 2016 at 01:00:55PM +0200, larsxschneider@gmail.com wrote:

From: Lars Schneider <redacted>

The verbose output clutters the Travis CI webview and is not really
useful since test debugging usually happens on a local machine.

I have not really been using the Travis CI results, so perhaps my
opinion does not count. But in other systems, I have found that the more
verbose the CI output, the better, simply because you will inevitably be
faced with a test that breaks on CI and not your local machine, and you
will have no way to get more details.

I don't know if Travis provides a better way to hide the output in the
non-failing cases.

-Peff