Re: t5800-*.sh: Intermittent test failures

Sverre Rabbelier [off-list ref] writes:

Heya,

On Tue, Aug 9, 2011 at 20:30, Ramsay Jones [off-list ref] wrote:

quoted

The git-fast-import is hung in the read() syscall waiting for data which will
never arrive. This is because the git(fast-export) process, started by the above
git(push), executes (producing it's data on stdout) and completes successfully
and exits *before* the above git-fast-import process starts.

I haven't looked to see how the git(fast-export)/git-fast-import processes are
plumbed together, but there seems to be a synchronization problem somewhere ...

This seems odd, before the fast-export process is even started it's
stdout are wired to the stdin of the helper (and thus the fast-import
process). What indication do you have that fast-import hasn't started
and that fast-export has finished?

Also, you say git remote-test everywhere, but it should be git
remote-testgit, typo?

FWIW, I have been seeing this every once in a while.

Junio C Hamano wrote:

Sverre Rabbelier [off-list ref] writes:

quoted

On Tue, Aug 9, 2011 at 20:30, Ramsay Jones [off-list ref] wrote:

quoted

The git-fast-import is hung in the read() syscall waiting for data which will
never arrive. This is because the git(fast-export) process, started by the above
git(push), executes (producing it's data on stdout) and completes successfully
and exits *before* the above git-fast-import process starts.

I haven't looked to see how the git(fast-export)/git-fast-import processes are
plumbed together, but there seems to be a synchronization problem somewhere ...

This seems odd, before the fast-export process is even started it's
stdout are wired to the stdin of the helper (and thus the fast-import
process). What indication do you have that fast-import hasn't started
and that fast-export has finished?

Also, you say git remote-test everywhere, but it should be git
remote-testgit, typo?

FWIW, I have been seeing this every once in a while.

Good to know I'm not alone ;-P

Unfortunately, I haven't had the time to debug this further than I've
already reported ...

As I said, it's obviously a process plumbing/synchronization problem; the reading
end of the fast-export output pipe must be open for read by someone (probably by
it's parent), otherwise it would receive SIGPIPE (also, the output is small enough
not to fill the pipe) rather than exiting with success.

When I run the tests with "make test >test-out", I see a failure rate of about
1 in 10. If I then set the debug environment variables (GIT_TRANSPORT_HELPER_DEBUG,
GIT_TRANSLOOP_DEBUG and GIT_DEBUG_TESTGIT) and run the test script directly (-v),
then the failure rate goes up to about 1 in 3.

Well, ... I added debug code to git-fast-{im,ex}port which writes the debug info
to a file (can't write to stdout/stderr obviously), so that may well be affecting
the timing enough to increase the chance of a failure. Having said that, If I'm
listening to music (rhythmbox) at the same time, then the failure rate seems to
increase ...

ATB,
Ramsay Jones

On Thu, Sep 08, 2011 at 06:42:11PM +0100, Ramsay Jones wrote:

When I run the tests with "make test >test-out", I see a failure rate of about
1 in 10. If I then set the debug environment variables (GIT_TRANSPORT_HELPER_DEBUG,
GIT_TRANSLOOP_DEBUG and GIT_DEBUG_TESTGIT) and run the test script directly (-v),
then the failure rate goes up to about 1 in 3.

Hmm. I can't reproduce a failure here, but I do get some weirdness. My
recipe is:

-- >8 --
cat >foo.sh <<\EOF
#!/bin/sh

exec >$1.out 2>&1

n=0
while test $n -lt 100; do
	n=$(($n+1))
	GIT_TRANSPORT_HELPER_DEBUG=1 \
	GIT_TRANSLOOP_DEBUG=1 \
	GIT_DEBUG_TESTGIT=1 \
	./t5800-remote-helpers.sh --root=/run/shm/git-tests-$1 -v || {
		echo FAIL $n
		exit 1
	}
	echo OK $n
done
EOF

# try to keep an 8-core machine busy
for i in `seq 1 16`; do
  sh foo.sh $i &
done
-- 8< --

I never see a test failure, but a few of the 16 end up hanging. The
process tree for the hanged tests look like:

  t5800-remote-helper
    git push
      git-remote-testgit
        git fast-import
          git-fast-import

All of them are blocked on wait(), except for the final fast-import,
which is blocked trying to read() from stdin.

-Peff

Jeff King wrote:

On Thu, Sep 08, 2011 at 06:42:11PM +0100, Ramsay Jones wrote:

quoted

When I run the tests with "make test >test-out", I see a failure rate of about
1 in 10. If I then set the debug environment variables (GIT_TRANSPORT_HELPER_DEBUG,
GIT_TRANSLOOP_DEBUG and GIT_DEBUG_TESTGIT) and run the test script directly (-v),
then the failure rate goes up to about 1 in 3.

Hmm. I can't reproduce a failure here, but I do get some weirdness. My
recipe is:

Ah, sorry, ... I didn't make myself clear then, because ...

-- >8 --
cat >foo.sh <<\EOF
#!/bin/sh

exec >$1.out 2>&1

n=0
while test $n -lt 100; do
	n=$(($n+1))
	GIT_TRANSPORT_HELPER_DEBUG=1 \
	GIT_TRANSLOOP_DEBUG=1 \
	GIT_DEBUG_TESTGIT=1 \
	./t5800-remote-helpers.sh --root=/run/shm/git-tests-$1 -v || {
		echo FAIL $n
		exit 1
	}
	echo OK $n
done
EOF

# try to keep an 8-core machine busy
for i in `seq 1 16`; do
  sh foo.sh $i &
done
-- 8< --

I never see a test failure, but a few of the 16 end up hanging. The
process tree for the hanged tests look like:

  t5800-remote-helper
    git push
      git-remote-testgit
        git fast-import
          git-fast-import

All of them are blocked on wait(), except for the final fast-import,
which is blocked trying to read() from stdin.

... these hangs *are* the failures of which I speak!  Yes, the script
doesn't get to declare a failure, but AFAIAC a hanging test (and it
isn't the same test # each time) is a failing test. :-D

ATB,
Ramsay Jones

On Sun, Sep 11, 2011 at 21:14, Ramsay Jones [off-list ref] wrote:

... these hangs *are* the failures of which I speak!  Yes, the script
doesn't get to declare a failure, but AFAIAC a hanging test (and it
isn't the same test # each time) is a failing test. :-D

Was there any outcome of this discussion? I'm asking because I
can reproduce this very reliably on a little server here.