Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

larsxschneider@gmail.com writes:

From: Lars Schneider <redacted>

Use the Travis-CI cache feature to store prove test results and make them
available in subsequent builds. This allows to run previously failed tests
first and run remaining tests in slowest to fastest order. As a result it
is less likely that Travis-CI needs to wait for a single test at the end
which speeds up the test suite execution by ~2 min.

Unfortunately the cache feature is only available (for free) on the
Travis-CI Linux environment.

Suggested-by: Jeff King <redacted>
Signed-off-by: Lars Schneider <redacted>
---
 .travis.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context?  I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?

IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature".  This
improvement looks to me like using the feature of "prove" that
allows us to run slower tests first, and does not have much to do
with Travis.

You are relying on the assumption that things under $HOME/ is stable
while things under t/ (or in our source tree in general) are not,
and I think that is a sensible thing to take advantage of, but are
we sure that they are running in an environment where "ln -s" would
work?  Otherwise, it may be more robust to copy $HOME/.prove to
t/.prove before starting to test and then copy it back once the
tests are done.

Thanks.

Hi Junio,

On Tue, 19 Jan 2016, Junio C Hamano wrote:

larsxschneider@gmail.com writes:

quoted

From: Lars Schneider <redacted>

Use the Travis-CI cache feature to store prove test results and make
them available in subsequent builds. This allows to run previously
failed tests first and run remaining tests in slowest to fastest
order. As a result it is less likely that Travis-CI needs to wait for
a single test at the end which speeds up the test suite execution by
~2 min.

Unfortunately the cache feature is only available (for free) on the
Travis-CI Linux environment.

Suggested-by: Jeff King <redacted>
Signed-off-by: Lars Schneider <redacted>
---
 .travis.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context?  I am
not suggesting to touch anything other than .travis.yml file in this
patch, but if I wanted to get the benefit from the idea in this patch
when I run my tests manually, I can just tell prove to use the cached
states, no?

You are basically talking about prove's support to modify behavior based
on previous runs. This patch does not introduce that support, it was
introduced long ago: 5099b99 (test-lib: Adjust output to be valid TAP
format, 2010-06-24). This commit also advertises that feature in t/README.

IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature".

Lars' patch is really about Travis and its useful feature to retain state
from previous runs. AFAICT this is the feature Lars is talking about, and
it is absolutely necessary to make use of this prove feature (don't try
this with BuildHive, for example, its workspaces are typically
garbage-collected before the next CI run).

As such, I think it makes sense to talk about the Travis-CI feature in the
commit message (and not about the prove feature because we introduced
support for prove much, much earlier in Git's history, and advertised its
benefits at that time, as I stated above).

Ciao,
Dscho

On 19 Jan 2016, at 21:00, Junio C Hamano [off-list ref] wrote:

larsxschneider@gmail.com writes:

quoted

From: Lars Schneider <redacted>

Use the Travis-CI cache feature to store prove test results and make them
available in subsequent builds. This allows to run previously failed tests
first and run remaining tests in slowest to fastest order. As a result it
is less likely that Travis-CI needs to wait for a single test at the end
which speeds up the test suite execution by ~2 min.

Unfortunately the cache feature is only available (for free) on the
Travis-CI Linux environment.

Suggested-by: Jeff King <redacted>
Signed-off-by: Lars Schneider <redacted>
---
.travis.yml | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context?  I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?

IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature".  This
improvement looks to me like using the feature of "prove" that
allows us to run slower tests first, and does not have much to do
with Travis.

You are relying on the assumption that things under $HOME/ is stable
while things under t/ (or in our source tree in general) are not,
and I think that is a sensible thing to take advantage of, but are
we sure that they are running in an environment where "ln -s" would
work?  Otherwise, it may be more robust to copy $HOME/.prove to
t/.prove before starting to test and then copy it back once the
tests are done.

OK, looks like my wording was not ideal. One important thing to know is that 
$HOME is *not* stable. These TravisCI machines start *always* in a completely 
clean state. That's why prove cannot store and use it's cache. With the following 
statement I instruct Travis to cache my ".prove-cache" directory. As a consequence
Travis CI will automatically restore this directory whenever it starts a new instance
for the git job. It will also save the content of this directory when the job is done.

quoted

+cache:
+  directories:
+    - $HOME/.prove-cache

The Travis CI cache works only on a directory basis. Since I don't want to cache
the entire /t directory I came up with the $HOME/.prove-cache directory. I also used
a file link to leverage the automated save/restore feature for the $HOME/.prove-cache
directory. If I would not use a link then I would need to copy the updated .prove file 
from t/ to .prove-cache after the test run.

Would the following first sentence for the commit message be less ambiguous?

"Use the Travis-CI cache feature to make the prove test results cache persistent 
across subsequent build jobs. This allows to run previously..."

Thanks,
Lars

quoted

diff --git a/.travis.yml b/.travis.yml
index c3bf9c6..f34726b 100644
--- a/.travis.yml
+++ b/.travis.yml

@@ -1,5 +1,9 @@

language: c

+cache:
+  directories:
+    - $HOME/.prove-cache
+
os:
  - linux
  - osx

@@ -18,7 +22,7 @@ env:
    - P4_VERSION="15.2"
    - GIT_LFS_VERSION="1.1.0"
    - DEFAULT_TEST_TARGET=prove
-    - GIT_PROVE_OPTS="--timer --jobs 3"
+    - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
    - GIT_TEST_OPTS="--verbose --tee"
    - CFLAGS="-g -O2 -Wall -Werror"
    - GIT_TEST_CLONE_2GB=YesPlease

@@ -67,6 +71,8 @@ before_install:
    p4 -V | grep Rev.;
    echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)";
    git-lfs version;
+    mkdir -p $HOME/.prove-cache;
+    ln -s $HOME/.prove-cache/.prove t/.prove;

before_script: make --jobs=2

--
2.5.1