Re: [PATCHv2] write_index: optionally allow broken null sha1s

Jeff King [off-list ref] writes:

I'd be very surprised if this works in practice on most of our current
test scripts. There are often subtle dependencies on the state left over
from previous tests. Running the script below up through t3800 (at which
point I lost patience) reveals 37 test scripts that are broken. Which is
only about 17%, but we're clearly not quite there yet.
...

Yes, I agree that in an ideal world with infinite amount of time, we
should strive to make sure that each test is independent from
previous steps, and if we do not have infinite amount of time, we
should make sure we find some (say 5%) time to various code clean-up
effort, in both writing and reviewing such, including tests.

The attached is cute.  Thanks for a food for thought.

-- >8 --
#!/usr/bin/perl
#
# Run as "perl foo.pl t0000-basic.sh" (or whatever script you want to
# check).

my $script = shift;
my ($script_num) = $script =~ /^(t\d+)/;

# run once to get the list of tests
my @tests = run_tests($script);

# mark some tests as "setup" tests that will always be run
foreach my $test (@tests) {
	if ($test->{desc} =~ /set.?up/i) {
		print STDERR "marking $test->{num} - $test->{desc} as setup\n";
		$test->{setup} = 1;
	}
}

# now try each test in isolation, but including setup tests
foreach my $test (@tests) {
	$ENV{GIT_SKIP_TESTS} = skip_all_except($script_num, $test, @tests);
	run_tests($script) or die "failed $test->{num} ($test->{desc})\n";
}

sub run_tests {
	my @r;
	open(my $fh, '-|', qw(sh), $script);
	while (<$fh>) {
		/^ok (\d+) - (.*)/ and push @r, { num => $1, desc => $2 };
	}
	$? and return ();
	return @r;
}

sub skip_all_except {
	my $prefix = shift;
	my $want = shift;

	return join(' ',
		map { "$prefix.$_->{num}" }
		grep { !$_->{setup} && $_->{num} != $want->{num} }
		@_);
}