Re: [PATCH v2] t/t1517: mark tests that fail with GIT_TEST_INSTALLED

Adam Dinwoodie [off-list ref] writes:

The changes added by 39fc408562 (t/t1517: automate `git subcmd -h` tests
outside a repository, 2025-08-08) to automatically loop over all "main"
Git commands will, when run against an installed build using
GIT_TEST_INSTALLED rather than the build in the build directory, include
some extra git-gui commands that are installed by `make install`, or
credential helpers that might be installed manually from the contrib
directories.  These fail the test, so record them as such.

Signed-off-by: Adam Dinwoodie <redacted>
---

This re-roll adds a few more commands to those marked as known failures,
notably credential helpers I see installed in various builds for the
Nixpkgs packaging of Git.

 t/t1517-outside-repo.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

I'd appreciate these efforts, but I am not sure if this is a losing
battle.  Your ~/libexec/git-core/ directory, when GIT_TEST_INSTALLED
is in effect, likely has old commands that are retired, commands
that are added by third-parties (so that their users can say "git
frotz" and run their "frotz" software), and/or commands from the
future that the running t1517 has not seen yet (while bisecting and
running t1517 from an older commit, say).  For example, I have these
differences...

	archimport.perl
	        citool
	cvsexportcommit.perl
	cvsimport.perl
	cvsserver.perl
	difftool--helper.sh
	filter-branch.sh
	        gui
	        gui--askpass
	instaweb.sh
	last-modified
	merge-octopus.sh
	merge-one-file.sh
	merge-resolve.sh
	mergetool.sh
	p4.py
	quiltimport.sh
	request-pull.sh
	send-email.perl
	submodule.sh
	svn.perl
	web--browse.sh

... in what t1517 $(git --list-cmds=main) sees between 'master' in
normal test mode and with GIT_TEST_INSTALLED set to ~/git/jch/bin
(i.e. the version I run for my everyday use).  "last-modified" is an
example of a new-ish command that the t1517 test being run is not
yet aware of but included in GIT_TEST_INSTALLED.

I am wondering if we are better off skipping this test, or at least
limiting to some known subset (e.g. "git --list-cmds=builtins") to
skip the files on disk when GIT_TEST_INSTALLED is in effect, instead
of "git --list-cmds=main" that is quite broad)?

In any case, this is a strict improvement over the previous one, so
I'll replace and queue this for now, but we may want to rethink the
approach this test uses.  Even without GIT_TEST_INSTALLED, the fake
GIT_EXEC_PATH we use during test has somewhat different from the
real thing, I suspect.  

Thanks.