How do I run tests under Valgrind?

Hi,

I tried running `make valgind` inside t/ and got:

    bug in test framework: multiple prerequisite tags do not work reliably

which means that even the basic tests don't pass.  Am I doing something wrong?

Ram

On Mon, Sep 17, 2012 at 10:31:07PM +0530, Ramkumar Ramachandra wrote:

I tried running `make valgind` inside t/ and got:

    bug in test framework: multiple prerequisite tags do not work reliably

which means that even the basic tests don't pass.  Am I doing something wrong?

No, that should work (and it does work here). I assume you can pass
t0000 without --valgrind?

It seems very odd that valgrind would have an impact here, since those
tests are not even running git.

-Peff

Hi Peff,

Jeff King wrote:

No, that should work (and it does work here). I assume you can pass
t0000 without --valgrind?

Yes.  Here's the output from running it with --valgrind.  I'm digging
deeper to see what went wrong.

make_valgrind_symlink:6: permission denied:
/home/artagnon/src/git/t/../git-instaweb
./test-lib.sh:487: no matches found:
/home/artagnon/svn/prefix/svn-trunk/bin:/home/artagnon/bin:/home/artagnon/bin/depot_tools:/home/artagnon/.ruby/bin:/home/artagnon/.python/bin:/home/artagnon/.cabal/bin:/home/artagnon/svn/prefix/svn-trunk/bin:/home/artagnon/bin:/home/artagnon/bin/depot_tools:/home/artagnon/.ruby/bin:/home/artagnon/.python/bin:/home/artagnon/.cabal/bin:/home/artagnon/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/home/artagnon/.ec2/bin:/home/artagnon/.ec2/bin/git-*
Initialized empty Git repository in /home/artagnon/src/git/t/trash
directory.test-lib/.git/
expecting success:
        find .git/objects -type f -print >should-be-empty &&
        test_line_count = 0 should-be-empty

ok 1 - .git/objects should be empty after git init in an empty repo

expecting success:
        find .git/objects -type d -print >full-of-directories &&
        test_line_count = 3 full-of-directories

ok 2 - .git/objects should have 3 subdirectories

expecting success:
        :

ok 3 - success is reported like this

checking known breakage:
        false

not ok 4 - pretend we have a known breakage # TODO known breakage

expecting success:
        mkdir passing-todo &&
        (cd passing-todo &&
        cat >passing-todo.sh <<-EOF &&
        #!/bin/sh

        test_description='A passing TODO test

        This is run in a sub test-lib so that we do not get incorrect
        passing metrics
        '

        # Point to the t/test-lib.sh, which isn't in ../ as usual
        TEST_DIRECTORY="/home/artagnon/src/git/t"
        . "$TEST_DIRECTORY"/test-lib.sh

        test_expect_failure 'pretend we have fixed a known breakage' '
                :
        '

        test_done
        EOF
        chmod +x passing-todo.sh &&
        ./passing-todo.sh >out 2>err &&
        ! test -s err &&
        sed -e 's/^> //' >expect <<-\EOF &&
        > ok 1 - pretend we have fixed a known breakage # TODO known breakage
        > # fixed 1 known breakage(s)
        > # passed all 1 test(s)
        > 1..1
        EOF
        test_cmp expect out)

test_cmp:1: command not found: diff -u
not ok - 5 pretend we have fixed a known breakage (run in sub test-lib)
#
#               mkdir passing-todo &&
#               (cd passing-todo &&
#               cat >passing-todo.sh <<-EOF &&
#               #!/bin/sh
#
#               test_description='A passing TODO test
#
#               This is run in a sub test-lib so that we do not get incorrect
#               passing metrics
#               '
#
#               # Point to the t/test-lib.sh, which isn't in ../ as usual
#               TEST_DIRECTORY="/home/artagnon/src/git/t"
#               . "$TEST_DIRECTORY"/test-lib.sh
#
#               test_expect_failure 'pretend we have fixed a known breakage' '
#                       :
#               '
#
#               test_done
#               EOF
#               chmod +x passing-todo.sh &&
#               ./passing-todo.sh >out 2>err &&
#               ! test -s err &&
#               sed -e 's/^> //' >expect <<-\EOF &&
#               > ok 1 - pretend we have fixed a known breakage # TODO
known breakage
#               > # fixed 1 known breakage(s)
#               > # passed all 1 test(s)
#               > 1..1
#               EOF
#               test_cmp expect out)
#

expecting success:
        test_have_prereq HAVEIT &&
        haveit=yes

ok 6 - test runs if prerequisite is satisfied

skipping test: unmet prerequisite causes test to be skipped
        donthaveit=no

ok 7 # skip unmet prerequisite causes test to be skipped (missing DONTHAVEIT)

skipping test: test runs if prerequisites are satisfied
        test_have_prereq HAVEIT &&
        test_have_prereq HAVETHIS &&
        haveit=yes

ok 8 # skip test runs if prerequisites are satisfied (missing HAVETHIS,HAVEIT)

skipping test: unmet prerequisites causes test to be skipped
        donthaveit=no

ok 9 # skip unmet prerequisites causes test to be skipped (missing
HAVEIT,DONTHAVEIT)

skipping test: unmet prerequisites causes test to be skipped
        donthaveiteither=no

ok 10 # skip unmet prerequisites causes test to be skipped (missing
DONTHAVEIT,HAVEIT)

bug in test framework: multiple prerequisite tags do not work reliably
FATAL: Unexpected exit with code 1

It seems very odd that valgrind would have an impact here, since those
tests are not even running git.

Yeah, I know!

Ram

On Mon, Sep 17, 2012 at 10:53:18PM +0530, Ramkumar Ramachandra wrote:

Hi Peff,

Jeff King wrote:

quoted

No, that should work (and it does work here). I assume you can pass
t0000 without --valgrind?

Yes.  Here's the output from running it with --valgrind.  I'm digging
deeper to see what went wrong.

make_valgrind_symlink:6: permission denied:
/home/artagnon/src/git/t/../git-instaweb
./test-lib.sh:487: no matches found:
/home/artagnon/svn/prefix/svn-trunk/bin:/home/artagnon/bin:/home/artagnon/bin/depot_tools:/home/artagnon/.ruby/bin:/home/artagnon/.python/bin:/home/artagnon/.cabal/bin:/home/artagnon/svn/prefix/svn-trunk/bin:/home/artagnon/bin:/home/artagnon/bin/depot_tools:/home/artagnon/.ruby/bin:/home/artagnon/.python/bin:/home/artagnon/.cabal/bin:/home/artagnon/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/home/artagnon/.ec2/bin:/home/artagnon/.ec2/bin/git-*

That's certainly odd. It sounds like the valgrind setup is broken for
you. Can you run:

  sh -x t0000-basic.sh --valgrind

and see what's happening near those weird errors?

test_cmp:1: command not found: diff -u

Lack of diff is going to be a problem. What OS is this? Do you really
not have diff? Or is there something funny going on with your PATH?

-Peff

Hi again,

Jeff King wrote:

That's certainly odd. It sounds like the valgrind setup is broken for
you. Can you run:

  sh -x t0000-basic.sh --valgrind

and see what's happening near those weird errors?

Not helpful:

+ . ./test-lib.sh
+ mkdir -p test-results
+ basename t0000-basic.sh .sh
+ BASE=test-results/t0000-basic
+ GIT_TEST_TEE_STARTED=done /usr/bin/zsh t0000-basic.sh --valgrind
+ tee test-results/t0000-basic.out

quoted

test_cmp:1: command not found: diff -u

Lack of diff is going to be a problem. What OS is this? Do you really
not have diff? Or is there something funny going on with your PATH?

It's plain Ubuntu.  Ofcourse I have `diff`- I don't know what's going on.

Ram

On Mon, Sep 17, 2012 at 11:09:27PM +0530, Ramkumar Ramachandra wrote:

Hi again,

Jeff King wrote:

quoted

That's certainly odd. It sounds like the valgrind setup is broken for
you. Can you run:

  sh -x t0000-basic.sh --valgrind

and see what's happening near those weird errors?

Not helpful:

+ . ./test-lib.sh
+ mkdir -p test-results
+ basename t0000-basic.sh .sh
+ BASE=test-results/t0000-basic
+ GIT_TEST_TEE_STARTED=done /usr/bin/zsh t0000-basic.sh --valgrind
+ tee test-results/t0000-basic.out

Oh, bleh. Stupid automatic --tee for valgrind. Try this:

  SHELL="/usr/bin/zsh -x" ./t0000-basic.sh --valgrind

I am also doing my tests with "dash" as my shell. You might try setting
your SHELL to /bin/sh to see if it makes a difference.

quoted

quoted

test_cmp:1: command not found: diff -u

Lack of diff is going to be a problem. What OS is this? Do you really
not have diff? Or is there something funny going on with your PATH?

It's plain Ubuntu.  Ofcourse I have `diff`- I don't know what's going on.

Maybe the PATH-munging in test-lib.sh is screwing up your PATH somehow.
But it all looks pretty simple. The "sh -x" output can maybe tell us
more.

-Peff

Am 17.09.2012 19:44, schrieb Jeff King:

Oh, bleh. Stupid automatic --tee for valgrind. Try this:

  SHELL="/usr/bin/zsh -x" ./t0000-basic.sh --valgrind

I am also doing my tests with "dash" as my shell. You might try setting
your SHELL to /bin/sh to see if it makes a difference.

Shouldn't -v be used as well? Or is --valgrind different?

-- Hannes

On Mon, Sep 17, 2012 at 07:55:24PM +0200, Johannes Sixt wrote:

Am 17.09.2012 19:44, schrieb Jeff King:

quoted

Oh, bleh. Stupid automatic --tee for valgrind. Try this:

  SHELL="/usr/bin/zsh -x" ./t0000-basic.sh --valgrind

I am also doing my tests with "dash" as my shell. You might try setting
your SHELL to /bin/sh to see if it makes a difference.

Shouldn't -v be used as well? Or is --valgrind different?

It turns "-v" on automatically.

-Peff

Hi,

Jeff King wrote:

Oh, bleh. Stupid automatic --tee for valgrind. Try this:

  SHELL="/usr/bin/zsh -x" ./t0000-basic.sh --valgrind

I got tons of output, but found it unhelpful.  Any other ideas?

Ram