Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

On 19 Jan 2016, at 21:00, Junio C Hamano [off-list ref] wrote:

larsxschneider@gmail.com writes:

quoted

From: Lars Schneider <redacted>

Use the Travis-CI cache feature to store prove test results and make them
available in subsequent builds. This allows to run previously failed tests
first and run remaining tests in slowest to fastest order. As a result it
is less likely that Travis-CI needs to wait for a single test at the end
which speeds up the test suite execution by ~2 min.

Unfortunately the cache feature is only available (for free) on the
Travis-CI Linux environment.

Suggested-by: Jeff King <redacted>
Signed-off-by: Lars Schneider <redacted>
---
.travis.yml | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context?  I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?

IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature".  This
improvement looks to me like using the feature of "prove" that
allows us to run slower tests first, and does not have much to do
with Travis.

You are relying on the assumption that things under $HOME/ is stable
while things under t/ (or in our source tree in general) are not,
and I think that is a sensible thing to take advantage of, but are
we sure that they are running in an environment where "ln -s" would
work?  Otherwise, it may be more robust to copy $HOME/.prove to
t/.prove before starting to test and then copy it back once the
tests are done.

OK, looks like my wording was not ideal. One important thing to know is that 
$HOME is *not* stable. These TravisCI machines start *always* in a completely 
clean state. That's why prove cannot store and use it's cache. With the following 
statement I instruct Travis to cache my ".prove-cache" directory. As a consequence
Travis CI will automatically restore this directory whenever it starts a new instance
for the git job. It will also save the content of this directory when the job is done.

quoted

+cache:
+  directories:
+    - $HOME/.prove-cache

The Travis CI cache works only on a directory basis. Since I don't want to cache
the entire /t directory I came up with the $HOME/.prove-cache directory. I also used
a file link to leverage the automated save/restore feature for the $HOME/.prove-cache
directory. If I would not use a link then I would need to copy the updated .prove file 
from t/ to .prove-cache after the test run.

Would the following first sentence for the commit message be less ambiguous?

"Use the Travis-CI cache feature to make the prove test results cache persistent 
across subsequent build jobs. This allows to run previously..."

Thanks,
Lars

quoted

diff --git a/.travis.yml b/.travis.yml
index c3bf9c6..f34726b 100644
--- a/.travis.yml
+++ b/.travis.yml

@@ -1,5 +1,9 @@

language: c

+cache:
+  directories:
+    - $HOME/.prove-cache
+
os:
  - linux
  - osx

@@ -18,7 +22,7 @@ env:
    - P4_VERSION="15.2"
    - GIT_LFS_VERSION="1.1.0"
    - DEFAULT_TEST_TARGET=prove
-    - GIT_PROVE_OPTS="--timer --jobs 3"
+    - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
    - GIT_TEST_OPTS="--verbose --tee"
    - CFLAGS="-g -O2 -Wall -Werror"
    - GIT_TEST_CLONE_2GB=YesPlease

@@ -67,6 +71,8 @@ before_install:
    p4 -V | grep Rev.;
    echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)";
    git-lfs version;
+    mkdir -p $HOME/.prove-cache;
+    ln -s $HOME/.prove-cache/.prove t/.prove;

before_script: make --jobs=2

--
2.5.1