From: Junio C Hamano <hidden> Date: 2021-11-05 19:11:29
Adam Dinwoodie [off-list ref] writes:
This is probably a much broader conversation. I remember when I first
started packaging Git for Cygwin, I produced a release that didn't
have support for HTTPS URLs due to a missing dependency in my build
environment. The build and test suite all passed -- it assumed I just
wanted to build a release that didn't have HTTPS support -- so some
relatively critical function was silently skipped. I don't know how to
avoid that sort of issue other than relying on (a) user bug (or at
least missing function) reports and (b) folk building Git for
themselves/others periodically going through the output of the
configure scripts and the skipped subtests to make sure only expected
things get missed; neither of those options seem great to me.
I agree with you that there needs a good way to enumerate what the
unsatisfied prerequisites for a particular build are. That would
have helped in your HTTPS situation.
But that is a separate issue how we should determine a lazy
prerequisite for any feature is satisified.
"We have this feature that our code utilizes. If it is not working
correctly, then we can expect our code that depends on it would not
work, and it is no use testing" is what the test prerequisite system
tries to achieve. That is quite different from "the frotz feature
could work here as we see a binary /usr/bin/frotz installed, so
let's go test our code that depends on it---we'll find out if the
installed frotz is not what we expect, or way too old to help our
code, as the test will break and let us notice."
From: Adam Dinwoodie <hidden> Date: 2021-11-05 19:25:35
On Fri, 5 Nov 2021 at 19:11, Junio C Hamano [off-list ref] wrote:
Adam Dinwoodie [off-list ref] writes:
quoted
This is probably a much broader conversation. I remember when I first
started packaging Git for Cygwin, I produced a release that didn't
have support for HTTPS URLs due to a missing dependency in my build
environment. The build and test suite all passed -- it assumed I just
wanted to build a release that didn't have HTTPS support -- so some
relatively critical function was silently skipped. I don't know how to
avoid that sort of issue other than relying on (a) user bug (or at
least missing function) reports and (b) folk building Git for
themselves/others periodically going through the output of the
configure scripts and the skipped subtests to make sure only expected
things get missed; neither of those options seem great to me.
I agree with you that there needs a good way to enumerate what the
unsatisfied prerequisites for a particular build are. That would
have helped in your HTTPS situation.
But that is a separate issue how we should determine a lazy
prerequisite for any feature is satisified.
"We have this feature that our code utilizes. If it is not working
correctly, then we can expect our code that depends on it would not
work, and it is no use testing" is what the test prerequisite system
tries to achieve. That is quite different from "the frotz feature
could work here as we see a binary /usr/bin/frotz installed, so
let's go test our code that depends on it---we'll find out if the
installed frotz is not what we expect, or way too old to help our
code, as the test will break and let us notice."
I can see how they're separate problems, but they seem related to me.
If OpenSSH were not installed on my system, Git would be compiled
without this function and the tests would be skipped. If OpenSSH is
installed but the prerequisite check fails, Git will be compiled with
the function, but the tests will be skipped. In the first case,
function some users might depend on will be missing; in the second,
the function will be nominally present but we won't be sure it's
actually working as expected. Both issues would be avoided if the
tests were always run, because suddenly both sorts of silent failure
become noisy.
I'm not actually advocating that -- running all tests all the time
would clearly cause far more problems than it would solve! -- but
that's why I'm seeing these as two sides of the same coin, and
problems that might have a single shared solution.
From: Carlo Arenas <hidden> Date: 2021-11-05 21:00:46
On Fri, Nov 5, 2021 at 1:16 PM Adam Dinwoodie [off-list ref] wrote:
If OpenSSH were not installed on my system, Git would be compiled
without this function and the tests would be skipped.
that is correct for the http dependency (because it is a library that
gets linked in), but not for the OpenSSH dependency, which is just
invoking the binary at runtime.
Regardless of what you have in your build environment the code will be
compiled in (and tested or not), and will fail instead at runtime if
OpenSSH is not installed.
Carlo
This is probably a much broader conversation. I remember when I first
started packaging Git for Cygwin, I produced a release that didn't
have support for HTTPS URLs due to a missing dependency in my build
environment. The build and test suite all passed -- it assumed I just
wanted to build a release that didn't have HTTPS support -- so some
relatively critical function was silently skipped. I don't know how to
avoid that sort of issue other than relying on (a) user bug (or at
least missing function) reports and (b) folk building Git for
themselves/others periodically going through the output of the
configure scripts and the skipped subtests to make sure only expected
things get missed; neither of those options seem great to me.
I agree with you that there needs a good way to enumerate what the
unsatisfied prerequisites for a particular build are. That would
have helped in your HTTPS situation.
Sorry for not replying earlier. I've been sick the last couple of days
and only slowly getting up to speed again. I will improve the prereq
tests in a new commit in the other patch series still in progress that
i'll shortly reroll.
As for the general prereq issue i ran into that as well during
development. When you depend on other patches / a specific version of
ssh-keygen for git I always have to remember to set the path correctly
or the tests might silently be ignored by the missing prereq. Usually
not a problem for single test runs, but when i run the full suite before
sending something.
So, here's a simple rfc patch to maybe start with addressing this issue.