From: Lars Schneider <redacted>
Apple removed the OpenSSL header files in macOS and therefore Git does
not build out of the box on macOS anymore. See previous discussion with
Torsten here: http://public-inbox.org/git/565B3036.8000604@web.de/
This mini series makes Git build out of the box on macOS, again, and
disables the HTTPD tests on macOS TravisCI as they don't work anymore
with the new macOS TravisCI default image:
https://blog.travis-ci.com/2016-10-04-osx-73-default-image-live/
Thanks,
Lars
Lars Schneider (2):
config.mak.in: set NO_OPENSSL and APPLE_COMMON_CRYPTO for macOS >10.11
travis-ci: disable GIT_TEST_HTTPD for macOS
.travis.yml | 3 ++-
config.mak.uname | 6 ++++++
2 files changed, 8 insertions(+), 1 deletion(-)
--
2.10.0
From: Lars Schneider <redacted>
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.
Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. Make it possible to override this and use OpenSSL by defining
`DARWIN_OPENSSL`.
Original-patch-by: Torsten Bögershausen [off-list ref]
Signed-off-by: Lars Schneider <redacted>
---
config.mak.uname | 6 ++++++
1 file changed, 6 insertions(+)
From: Lars Schneider <redacted>
TravisCI changed their default macOS image from 10.10 to 10.11 [1].
Unfortunately the HTTPD tests do not run out of the box using the
pre-installed Apache web server anymore. Therefore we enable these
tests only for Linux and disable them for macOS.
[1] https://blog.travis-ci.com/2016-10-04-osx-73-default-image-live/
Signed-off-by: Lars Schneider <redacted>
---
.travis.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
@@ -32,7 +32,6 @@ env:-DEFAULT_TEST_TARGET=prove-GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"-GIT_TEST_OPTS="--verbose --tee"--GIT_TEST_HTTPD=true-GIT_TEST_CLONE_2GB=YesPlease# t9810 occasionally fails on Travis CI OS X# t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
From: Jeff King <hidden> Date: 2016-10-17 09:57:17
On Sun, Oct 16, 2016 at 05:25:49PM -0700, larsxschneider@gmail.com wrote:
From: Lars Schneider <redacted>
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.
Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. Make it possible to override this and use OpenSSL by defining
`DARWIN_OPENSSL`.
I like that you gave an override, but I don't think it works in all
cases:
This is in config.mak.uname, which gets sourced before config.mak (and
ifndef is evaluated at the time of parsing). So it would work to do:
make DARWIN_OPENSSL=Yep
but not:
echo DARWIN_OPENSSL=Yep >>config.mak
make
I think you'd have to set a flag in config.mak.uname, and then resolve
it in the Makefile proper like:
ifdef DARWIN_OPENSSL
# Overrides AUTO_AVOID_OPENSSL, do nothing.
else ifdef AUTO_AVOID_OPENSSL
NO_OPENSSL = YesPlease
APPLE_COMMON_CRYPTO = YesPlease
endif
but that's totally untested.
-Peff
From: Lars Schneider <hidden> Date: 2016-11-06 19:50:21
On 17 Oct 2016, at 11:50, Jeff King [off-list ref] wrote:
On Sun, Oct 16, 2016 at 05:25:49PM -0700, larsxschneider@gmail.com wrote:
quoted
From: Lars Schneider <redacted>
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.
Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. Make it possible to override this and use OpenSSL by defining
`DARWIN_OPENSSL`.
I like that you gave an override, but I don't think it works in all
cases:
This is in config.mak.uname, which gets sourced before config.mak (and
ifndef is evaluated at the time of parsing). So it would work to do:
make DARWIN_OPENSSL=Yep
but not:
echo DARWIN_OPENSSL=Yep >>config.mak
make
I think you'd have to set a flag in config.mak.uname, and then resolve
it in the Makefile proper like:
ifdef DARWIN_OPENSSL
# Overrides AUTO_AVOID_OPENSSL, do nothing.
else ifdef AUTO_AVOID_OPENSSL
NO_OPENSSL = YesPlease
APPLE_COMMON_CRYPTO = YesPlease
endif
but that's totally untested.
Good point. I think I found an even easier way to achieve the same.
What do you think about the patch below?
Thanks,
Lars
-- >8 --
Subject: Makefile: set NO_OPENSSL on macOS by default
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.
Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. It is possible to override this and use OpenSSL by defining
`NO_APPLE_COMMON_CRYPTO`.
Original-patch-by: Torsten Bögershausen [off-list ref]
Signed-off-by: Lars Schneider <redacted>
---
Makefile | 1 +
1 file changed, 1 insertion(+)
From: Lars Schneider <hidden> Date: 2016-11-06 21:43:26
quoted hunk
On 17 Oct 2016, at 02:25, larsxschneider@gmail.com wrote:
From: Lars Schneider <redacted>
TravisCI changed their default macOS image from 10.10 to 10.11 [1].
Unfortunately the HTTPD tests do not run out of the box using the
pre-installed Apache web server anymore. Therefore we enable these
tests only for Linux and disable them for macOS.
[1] https://blog.travis-ci.com/2016-10-04-osx-73-default-image-live/
Signed-off-by: Lars Schneider <redacted>
---
.travis.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
@@ -32,7 +32,6 @@ env:-DEFAULT_TEST_TARGET=prove-GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"-GIT_TEST_OPTS="--verbose --tee"--GIT_TEST_HTTPD=true-GIT_TEST_CLONE_2GB=YesPlease# t9810 occasionally fails on Travis CI OS X# t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
That is much simpler.
I have in the back of my mind a fear that it is harder to unset a make
variable than it is to override it with a new value (which is what you'd
want to do here to turn openssl back on), but I can't seem to come up
with a case that doesn't work. So I am probably misremembering, or just
thinking of something that used to be a problem long ago.
-Peff
From: Paul Smith <hidden> Date: 2016-11-07 17:38:52
On Mon, 2016-11-07 at 12:26 -0500, Jeff King wrote:
I have in the back of my mind a fear that it is harder to unset a
make variable than it is to override it with a new value (which is
what you'd want to do here to turn openssl back on),
It depends on what you mean by "unset".
If you mean it as per the shell "unset" command, where the variable is
completely forgotten as if it never was set at all, that's tricky. You
have to use the "undefine" special command which was introduced in GNU
make 3.82 (released in 2010).
But if you just want to set the variable to the empty string, using
"FOO=" works fine for that in all versions of make (GNU and otherwise)
and using all the normal rules (command line override, etc.)
It's not easy to distinguish between a variable that is empty and one
that is actually not defined, in make, so it's a difference without a
distinction in almost all situations.
From: Jeff King <hidden> Date: 2016-11-07 17:46:08
On Mon, Nov 07, 2016 at 12:36:34PM -0500, Paul Smith wrote:
On Mon, 2016-11-07 at 12:26 -0500, Jeff King wrote:
quoted
I have in the back of my mind a fear that it is harder to unset a
make variable than it is to override it with a new value (which is
what you'd want to do here to turn openssl back on),
It depends on what you mean by "unset".
If you mean it as per the shell "unset" command, where the variable is
completely forgotten as if it never was set at all, that's tricky. You
have to use the "undefine" special command which was introduced in GNU
make 3.82 (released in 2010).
But if you just want to set the variable to the empty string, using
"FOO=" works fine for that in all versions of make (GNU and otherwise)
and using all the normal rules (command line override, etc.)
Specifically I wanted to make sure that
FOO = bar
FOO =
ifdef FOO
... something ...
endif
works as if FOO had never been set in the first place. Which it seems
to, at least in GNU make (and that is the only one we support, for other
reasons).
-Peff
From: Paul Smith <hidden> Date: 2016-11-07 17:49:47
On Mon, 2016-11-07 at 12:46 -0500, Jeff King wrote:
Specifically I wanted to make sure that
FOO = bar
FOO =
ifdef FOO
... something ...
endif
works as if FOO had never been set in the first place. Which it seems
to, at least in GNU make (and that is the only one we support, for
other reasons).
Yes, it will work. Confusingly, "ifdef" actually tests whether the
variable has a non-empty value, not whether it's defined:
The 'ifdef' form takes the _name_ of a variable as its argument, not
a reference to a variable. The value of that variable has a non-
empty value, the TEXT-IF-TRUE is effective; otherwise, the TEXT-IF-
FALSE, if any, is effective
From: Jeff King <hidden> Date: 2016-11-07 21:20:10
On Sun, Nov 06, 2016 at 10:42:36PM +0100, Lars Schneider wrote:
quoted
From: Lars Schneider <redacted>
TravisCI changed their default macOS image from 10.10 to 10.11 [1].
Unfortunately the HTTPD tests do not run out of the box using the
pre-installed Apache web server anymore. Therefore we enable these
tests only for Linux and disable them for macOS.
[...]
Hi Junio,
the patch above is one of two patches to make TravisCI pass, again.
Could you queue it?
I don't really mind disabling tests if they don't run on a platform. But
the more interesting question to me is: why don't they run any more? Is
there some config tweak needed, or is it an insurmountable problem?
Using Apache in the tests has been the source of frequent portability
problems and configuration headaches. I do wonder if we'd be better off
using some small special-purpose web server (even a short perl script
written around HTTP::Server::Simple or something).
On the other hand, testing against Apache approximates a more real-world
case, which has value. It might be nice if our tests supported multiple
web servers, but that would mean duplicating the config for each
manually.
-Peff
[]
I don't know if that is a correct solution.
If I have Mac OS 10.12 and Mac Ports installed, I may want to use
OPENSSL from Mac Ports.
How about this:
[]
I don't know if that is a correct solution.
If I have Mac OS 10.12 and Mac Ports installed, I may want to use
OPENSSL from Mac Ports.
Can't you define `NO_APPLE_COMMON_CRYPTO` in that case?
I think if you use OpenSSL then you don't need the Apple crypto lib, right?
After re-reading the Makefile: that makes sense :-)
Do you want to send a new patch ?
Feel free to omit
"Original-patch-by: Torsten Bögershausen [off-list ref]"
From: Lars Schneider <hidden> Date: 2016-11-10 11:07:22
On 07 Nov 2016, at 22:20, Jeff King [off-list ref] wrote:
On Sun, Nov 06, 2016 at 10:42:36PM +0100, Lars Schneider wrote:
quoted
quoted
From: Lars Schneider <redacted>
TravisCI changed their default macOS image from 10.10 to 10.11 [1].
Unfortunately the HTTPD tests do not run out of the box using the
pre-installed Apache web server anymore. Therefore we enable these
tests only for Linux and disable them for macOS.
[...]
Hi Junio,
the patch above is one of two patches to make TravisCI pass, again.
Could you queue it?
I don't really mind disabling tests if they don't run on a platform. But
the more interesting question to me is: why don't they run any more? Is
there some config tweak needed, or is it an insurmountable problem?
I can't really remember what the problem was. I think some apache config
required some module that was not present and I wasn't able to get this
working quickly.
Using Apache in the tests has been the source of frequent portability
problems and configuration headaches. I do wonder if we'd be better off
using some small special-purpose web server (even a short perl script
written around HTTP::Server::Simple or something).
On the other hand, testing against Apache approximates a more real-world
case, which has value. It might be nice if our tests supported multiple
web servers, but that would mean duplicating the config for each
manually.
I agree that the real-world Apache test is more valuable and I really want
to keep the Linux Apache test running. However, I don't think many people
use macOS as Git web server and therefore I thought it is not worth the
effort to investigate this problem further.
- Lars
From: Lars Schneider <redacted>
Apple removed the OpenSSL header files in macOS 10.11 and above. OpenSSL
was deprecated since macOS 10.7.
Set `NO_OPENSSL` and `APPLE_COMMON_CRYPTO` to `YesPlease` as default for
macOS. It is possible to override this and use OpenSSL by defining
`NO_APPLE_COMMON_CRYPTO`.
Signed-off-by: Lars Schneider <redacted>
---
Makefile | 1 +
1 file changed, 1 insertion(+)
From: Jeff King <hidden> Date: 2016-11-10 16:10:21
On Thu, Nov 10, 2016 at 12:07:14PM +0100, Lars Schneider wrote:
quoted
Using Apache in the tests has been the source of frequent portability
problems and configuration headaches. I do wonder if we'd be better off
using some small special-purpose web server (even a short perl script
written around HTTP::Server::Simple or something).
On the other hand, testing against Apache approximates a more real-world
case, which has value. It might be nice if our tests supported multiple
web servers, but that would mean duplicating the config for each
manually.
I agree that the real-world Apache test is more valuable and I really want
to keep the Linux Apache test running. However, I don't think many people
use macOS as Git web server and therefore I thought it is not worth the
effort to investigate this problem further.
IMHO, the value in the http tests is not testing the server side, but
the client side. Without being able to set up a dummy HTTP server, we do
not have any way to exercise the client side of git-over-http at all.
And people on macOS _do_ use that. :)
-Peff
From: Lars Schneider <hidden> Date: 2016-11-11 08:22:59
On 10 Nov 2016, at 17:10, Jeff King [off-list ref] wrote:
On Thu, Nov 10, 2016 at 12:07:14PM +0100, Lars Schneider wrote:
quoted
quoted
Using Apache in the tests has been the source of frequent portability
problems and configuration headaches. I do wonder if we'd be better off
using some small special-purpose web server (even a short perl script
written around HTTP::Server::Simple or something).
On the other hand, testing against Apache approximates a more real-world
case, which has value. It might be nice if our tests supported multiple
web servers, but that would mean duplicating the config for each
manually.
I agree that the real-world Apache test is more valuable and I really want
to keep the Linux Apache test running. However, I don't think many people
use macOS as Git web server and therefore I thought it is not worth the
effort to investigate this problem further.
IMHO, the value in the http tests is not testing the server side, but
the client side. Without being able to set up a dummy HTTP server, we do
not have any way to exercise the client side of git-over-http at all.
And people on macOS _do_ use that. :)
Well, I haven't seen it from that perspective, yet, but I agree :-)
To all macOS users on the list:
Does anyone execute the tests with GIT_TEST_HTTPD enabled successfully?
There would be an alternative way to approach the problem:
Someone (GitHub?, BitBucket?, GitLab?, ...) could setup a bunch of webservers
with popular configurations and a way to reset a clean test environment. Then
the TravisCI client tests could go against these servers.
I realize that this idea is probably unrealistic because too much setup and
maintenance work would be required.
Cheers,
Lars
From: Jeff King <hidden> Date: 2016-11-11 08:47:32
On Fri, Nov 11, 2016 at 09:22:51AM +0100, Lars Schneider wrote:
There would be an alternative way to approach the problem:
Someone (GitHub?, BitBucket?, GitLab?, ...) could setup a bunch of webservers
with popular configurations and a way to reset a clean test environment. Then
the TravisCI client tests could go against these servers.
I realize that this idea is probably unrealistic because too much setup and
maintenance work would be required.
Yeah, it seems like it adds a lot of complexity for little gain. Plus it
creates a network dependency on running the tests. I know you care
mostly about Travis, but I am much more interested in all of the people
(developers and not) who run "make test" on their own platforms.
If you did want to have a more real-world network-based test, I think
the right solution is not for GitHub to set up a bunch of mock servers,
but to design client-side tests that hit the _real_ GitHub (or GitLab,
or whatever) and perform some basic operations. OTOH, people running
"master" (or "next", etc) are doing that implicitly every day.
-Peff
From: Lars Schneider <hidden> Date: 2016-11-11 09:13:56
On 11 Nov 2016, at 09:47, Jeff King [off-list ref] wrote:
On Fri, Nov 11, 2016 at 09:22:51AM +0100, Lars Schneider wrote:
quoted
There would be an alternative way to approach the problem:
Someone (GitHub?, BitBucket?, GitLab?, ...) could setup a bunch of webservers
with popular configurations and a way to reset a clean test environment. Then
the TravisCI client tests could go against these servers.
I realize that this idea is probably unrealistic because too much setup and
maintenance work would be required.
Yeah, it seems like it adds a lot of complexity for little gain. Plus it
creates a network dependency on running the tests. I know you care
mostly about Travis, but I am much more interested in all of the people
(developers and not) who run "make test" on their own platforms.
If you did want to have a more real-world network-based test, I think
the right solution is not for GitHub to set up a bunch of mock servers,
but to design client-side tests that hit the _real_ GitHub (or GitLab,
or whatever) and perform some basic operations. OTOH, people running
"master" (or "next", etc) are doing that implicitly every day.
That is actually a neat idea. We could setup a test repo on each of the major
Git hosting sites and then the TravisCI run could clone a repo and push
changes to it. That shouldn't take long and would probably be a good real
world test.
The credentials of these repos could be stored encrypted in Travis CI [1].
Where would such a test repo live on github.com? On github.com/git or would
you prefer a separate organization? (no worries, I am not going to tackle this
anytime soon -- too many things in my backlog already).
BTW: I just noticed https://github.com/git/hello-world ... is this legitimate
or did someone hack github.com/git? :)
Cheers,
Lars
[1] https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml
From: Jeff King <hidden> Date: 2016-11-11 09:28:36
On Fri, Nov 11, 2016 at 10:13:44AM +0100, Lars Schneider wrote:
quoted
If you did want to have a more real-world network-based test, I think
the right solution is not for GitHub to set up a bunch of mock servers,
but to design client-side tests that hit the _real_ GitHub (or GitLab,
or whatever) and perform some basic operations. OTOH, people running
"master" (or "next", etc) are doing that implicitly every day.
That is actually a neat idea. We could setup a test repo on each of the major
Git hosting sites and then the TravisCI run could clone a repo and push
changes to it. That shouldn't take long and would probably be a good real
world test.
The credentials of these repos could be stored encrypted in Travis CI [1].
Where would such a test repo live on github.com? On github.com/git or would
you prefer a separate organization? (no worries, I am not going to tackle this
anytime soon -- too many things in my backlog already).
I think I'd prefer for it to live outside of the "git" organization
entirely, if only because it's credentials will be necessarily less
secure. It would be nice if people could point the suite at their own
user/repo, too, so they can run it outside of travis.
Hmm. I wondered myself. There's no audit record of who created it, but
the creation timestamp is from 2008-07-23, which predates a lot of the
logging. So offhand, I'd say the likely explanation is "Scott Chacon
experimenting".
It's probably worth cleaning out now, though.
-Peff
From: Jeff King <hidden> Date: 2016-11-15 15:32:55
On Tue, Nov 15, 2016 at 01:07:18PM +0100, Heiko Voigt wrote:
On Fri, Nov 11, 2016 at 09:22:51AM +0100, Lars Schneider wrote:
quoted
To all macOS users on the list:
Does anyone execute the tests with GIT_TEST_HTTPD enabled successfully?
Nope. The following tests fail for me on master: 5539, 5540, 5541, 5542,
5550, 5551, 5561, 5812.
Failing how? Does apache fail to start up? Do tests fails? What does
"-v" say? Is there anything interesting in httpd/error.log in the trash
directory?
-Peff
On Tue, Nov 15, 2016 at 10:31:59AM -0500, Jeff King wrote:
On Tue, Nov 15, 2016 at 01:07:18PM +0100, Heiko Voigt wrote:
quoted
On Fri, Nov 11, 2016 at 09:22:51AM +0100, Lars Schneider wrote:
quoted
To all macOS users on the list:
Does anyone execute the tests with GIT_TEST_HTTPD enabled successfully?
Nope. The following tests fail for me on master: 5539, 5540, 5541, 5542,
5550, 5551, 5561, 5812.
Failing how? Does apache fail to start up? Do tests fails? What does
"-v" say? Is there anything interesting in httpd/error.log in the trash
directory?
This is what I see for 5539:
$ GIT_TEST_HTTPD=1 ./t5539-fetch-http-shallow.sh -v
Initialized empty Git repository in /Users/hvoigt/Repository/git4/t/trash directory.t5539-fetch-http-shallow/.git/
checking prerequisite: NOT_ROOT
mkdir -p "$TRASH_DIRECTORY/prereq-test-dir" &&
(
cd "$TRASH_DIRECTORY/prereq-test-dir" &&
uid=$(id -u) &&
test "$uid" != 0
)
prerequisite NOT_ROOT ok
httpd: Syntax error on line 65 of /Users/hvoigt/Repository/git4/t/lib-httpd/apache.conf: Cannot load modules/mod_mpm_prefork.so into server: dlopen(/Users/hvoigt/Repository/git4/t/trash directory.t5539-fetch-http-shallow/httpd/modules/mod_mpm_prefork.so, 10): image not found
error: web server setup failed
It seems the other failures have the same cause.
Cheers Heiko
On Tue, Nov 15, 2016 at 10:31:59AM -0500, Jeff King wrote:
quoted
On Tue, Nov 15, 2016 at 01:07:18PM +0100, Heiko Voigt wrote:
quoted
On Fri, Nov 11, 2016 at 09:22:51AM +0100, Lars Schneider wrote:
quoted
To all macOS users on the list:
Does anyone execute the tests with GIT_TEST_HTTPD enabled successfully?
Nope. The following tests fail for me on master: 5539, 5540, 5541, 5542,
5550, 5551, 5561, 5812.
Failing how? Does apache fail to start up? Do tests fails? What does
"-v" say? Is there anything interesting in httpd/error.log in the trash
directory?
This is what I see for 5539:
$ GIT_TEST_HTTPD=1 ./t5539-fetch-http-shallow.sh -v
Initialized empty Git repository in /Users/hvoigt/Repository/git4/t/trash directory.t5539-fetch-http-shallow/.git/
checking prerequisite: NOT_ROOT
mkdir -p "$TRASH_DIRECTORY/prereq-test-dir" &&
(
cd "$TRASH_DIRECTORY/prereq-test-dir" &&
uid=$(id -u) &&
test "$uid" != 0
)
prerequisite NOT_ROOT ok
httpd: Syntax error on line 65 of /Users/hvoigt/Repository/git4/t/lib-httpd/apache.conf: Cannot load modules/mod_mpm_prefork.so into server: dlopen(/Users/hvoigt/Repository/git4/t/trash directory.t5539-fetch-http-shallow/httpd/modules/mod_mpm_prefork.so, 10): image not found
error: web server setup failed
I run into other issues:
[core:emerg] [pid 2502] (2)No such file or directory: AH00023: Couldn't create the rewrite-map mutex (file /private/var/run/rewrite-map.2502)
AH00016: Configuration Failed
(apache2 comes via MacPorts)