Junio C Hamano [off-list ref] writes:
This is cute, but isn't it useful even outside Travis's context? I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?
It seems that exporting something like
GIT_PROVE_OPTS="--timer --state=slow,save -j8"
when running "make DEFAULT_TEST_TARGET=prove test" does give me the
same benefit by leaving the stats from the previous run in t/.prove
when making the test scheduling decisions.
One thing I noticed but didn't dig further to fix was that this
"prove --state" business did not seem to work well together with
make T="...list of tests..." test
that limits the set of tests to perform. For example:
$ rm -f t/.prove
$ make -j4 GIT_PROVE_OPTS="--timer --state=slow,save -j8" \
T="$( cd t && echo t0???-*.sh)" \
DEFAULT_TEST_TARGET=prove test
runs all test in 0xxx series and populates t/.prove with them. And
immediately after that, with t/.prove still there:
$ make -j4 GIT_PROVE_OPTS="--timer --state=slow,save -j8" \
DEFAULT_TEST_TARGET=prove \
T=t0000-basic.sh test
does not limit the test to only 0000, but ends up running all the
others recorded in t/.prove file, it seems.
I would imagine that this would not affect your use case negatively,
as it is unlikely that your automated tests are skipping different
set of tests in each run.
On Tue, Jan 19, 2016 at 02:53:00PM -0800, Junio C Hamano wrote:
Junio C Hamano [off-list ref] writes:
quoted
This is cute, but isn't it useful even outside Travis's context? I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?
It seems that exporting something like
GIT_PROVE_OPTS="--timer --state=slow,save -j8"
when running "make DEFAULT_TEST_TARGET=prove test" does give me the
same benefit by leaving the stats from the previous run in t/.prove
when making the test scheduling decisions.
Yes, I've been using this on my local machine for years (which is why I
suggested it to Lars for the Travis build). I have also noticed that my
test runs take about as much time as the longest-running test, and do
not fully utilize all of my processors. I suspect we could drop the
run-time of the test suite substantially by splitting a few of the
longer tests.
You also wrote earlier:
IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature". This
improvement looks to me like using the feature of "prove" that
allows us to run slower tests first, and does not have much to do
with Travis.
The interesting Travis feature we are using is that we are allowed to
store some data from run-to-run. So we use the Travis feature that lets
us use the prove feature. :)
One thing I noticed but didn't dig further to fix was that this
"prove --state" business did not seem to work well together with
make T="...list of tests..." test
that limits the set of tests to perform.
Right, it does not do what you want. The "prove --state" feature is not
just about ordering, but also about selecting. When run via "make", we
always give prove the full list of tests. But you can also do:
prove --state=failed
manually to just run whatever failed on the last run.
I don't know if there is a way to tell prove "use the state for
ordering, but don't otherwise select from it", which would do what you
want above.
You can also note that if we ever delete a test script, it will still be
mentioned in prove's state file. I think prove is smart enough to
realize it went away and not bother you.
-Peff
On Tue, Jan 19, 2016 at 06:06:33PM -0500, Jeff King wrote:
quoted
It seems that exporting something like
GIT_PROVE_OPTS="--timer --state=slow,save -j8"
when running "make DEFAULT_TEST_TARGET=prove test" does give me the
same benefit by leaving the stats from the previous run in t/.prove
when making the test scheduling decisions.
Yes, I've been using this on my local machine for years (which is why I
suggested it to Lars for the Travis build). I have also noticed that my
test runs take about as much time as the longest-running test, and do
not fully utilize all of my processors. I suspect we could drop the
run-time of the test suite substantially by splitting a few of the
longer tests.
Here are the numbers for that:
$ time make ;# configured to use prove --state=slow,save -j16
[...]
real 0m47.035s
user 1m6.884s
sys 0m19.892s
$ grep -v '^\.\.\.' .prove |
perl -MYAML -e '
local $/;
$x = YAML::Load(<>)->{tests};
print int($x->{$_}->{elapsed}), " $_\n" for keys(%$x)
' |
sort -rn |
head
39 t3404-rebase-interactive.sh
29 t3421-rebase-topology-linear.sh
27 t9001-send-email.sh
16 t9500-gitweb-standalone-no-errors.sh
15 t3425-rebase-topology-merges.sh
14 t6030-bisect-porcelain.sh
13 t7610-mergetool.sh
13 t5572-pull-submodule.sh
13 t3426-rebase-submodule.sh
12 t3415-rebase-autosquash.sh
So we're running t3404 for the majority of the time. I guess that
doesn't tell us how full our pipelines are for the rest of the time,
though. It could be worth splitting some of those long tests and seeing
if that improves run-time, though.
-Peff