Re: Prove "Tests out of sequence" Error

On Thu, Oct 20, 2016 at 11:10:39PM -0700, Stefan Beller wrote:

quoted

on TravisCI I see these weird "Tests out of sequence" errors with prove
and they seem to not go away. I assume the reason that they not go away
is that the ".prove" file is carried over from on build to another (but I can't
look into this file on TravisCI).

Has anyone an idea where these errors might come from?

------------------------------------------------------------------------
t5547-push-quarantine.sh                         (Wstat: 0 Tests: 5 Failed: 0)

push quarantine is a new thing made by Jeff 2 weeks ago, IIRC.

Yes, but I do not see how it can trigger this:

quoted

  Parse errors: Tests out of sequence.  Found (2) but expected (3)
                Tests out of sequence.  Found (3) but expected (4)
                Tests out of sequence.  Found (4) but expected (5)
                Bad plan.  You planned 4 tests but ran 5.

The TAP output from one of our tests should look something like:

  ok 1 - subject one
  ok 2 - subject two
  ok 3 - subject three
  ok 4 - subject four
  # passed all 4 test(s)
  1..4

the "plan" is the bit at the end. That looks like $test_count
accidentally got incremented by one and we generated something like:

  ok 1 - subject one
  ok 3 - subject two
  ok 4 - subject three
  ok 5 - subject four
  1..4

which would explain the "out of sequence" errors as well as the "planned
4 but ran 5".

But I do not see how the test script could screw that up. The counting
is handled entirely by the harness in test-lib.sh.

Nor do I see how a stale .prove file could matter. It does not store
information about the test plan at all. E.g., here is the entry from
mine for t5547:

  t5547-push-quarantine.sh:
    elapsed: 0.0762169361114502
    gen: 1
    last_pass_time: 1477037708.741
    last_result: 0
    last_run_time: 1477037708.741
    last_todo: 0
    seq: 437
    total_passes: 1

Puzzling.

-Peff