Re: [PATCH v2] t: use user-specific utf-8 locale for testing
From: Torsten Bögershausen <hidden>
Date: 2021-06-06 20:06:28
This all looks good. Some suggestions about the commit message are inline. On Sun, Jun 06, 2021 at 11:33:16PM +0700, Đoàn Trần Công Danh wrote:
In some test-cases, utf-8 locale is required. To find such locale, we're using the first available UTF-8 locale that returned by "locale -a".
Good explanation. I think that in generaral "utf-8" as a specification/specifier is better written as "UTF-8", with uppercase. "utf-8" or utf8 may be used inside the code, depending on the language.
Despite being required by POSIX, locale(1) is unavailable in some systems, e.g. Linux with musl libc. Some of those systems support utf-8 locale out of the box.
This reads a little bit harsh (the first sentence) and it is not fully clear which systems do what (the second sentence). Or are 2 things mentioned - the locale(1) utility and the support of one UTF-8 locale "out of the box" ? Does Linux with musl libs support an UTF-8 locale, but not the locale(1) untility ? Git itself supports many systems, that are not POSIX compliant, strictly speaking. But if avaliable, the functions defined in POSIX are used, whenever available. Could we write: However, the locale(1) utility is unavailable on some systems, e.g. Linux with musl libc.
However, without "locale -a", we can't guess provided UTF-8 locale. Let's give users of those systems an option to have better test coverage.
Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl
This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale, since we no longer prepare the variable named "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE instead. The fallback will be LC_ALL, LANG environment variable, or the first utf-8 locale from output of "locale -a", in that order.
rename -> renames, may be drop "This change", like this ? Rename t/lib-git-svn.sh:prepare_a_utf8_locale into prepare_utf8_locale, since we no longer prepare the variable named "a_utf8_locale", but set up a fallback value for GIT_TEST_UTF8_LOCALE instead. The fallback will be LC_ALL, LANG environment variable, or the first UTF-8 locale from output of "locale -a", in that order.
quoted hunk ↗ jump to hunk
Signed-off-by: Đoàn Trần Công Danh <redacted> --- Range-diff against v1: 1: d242ce64c4 ! 1: f299ae2239 t: use user-specific utf-8 locale for testing @@ Commit message Let's give users of those systems an option to have better test coverage. + This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to + prepare_utf8_locale, since we no longer prepare the variable named + "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE + instead. The fallback will be LC_ALL, LANG environment variable, + or the first utf-8 locale from output of "locale -a", in that order. + Signed-off-by: Đoàn Trần Công Danh [off-list ref] ## Makefile ## @@ Makefile: all:: # with a different indexfile format version. If it isn't set the index # file format used is index-v[23]. # -+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing. -+# If it isn't set, use the first utf-8 locale returned by "locale -a". ++# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing. ++# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8 ++# locale returned by "locale -a". +# # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime. # @@ Makefile: ifdef GIT_TEST_CMP @echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+ ifdef GIT_PERF_REPEAT_COUNT + ## ci/lib.sh ## +@@ ci/lib.sh: linux-musl) + CC=gcc + MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes" + MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes" ++ MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8" + ;; + esac + + ## t/lib-git-svn.sh ## @@ t/lib-git-svn.sh: start_svnserve () { --listen-host 127.0.0.1 & @@ t/lib-git-svn.sh: start_svnserve () { -}') - if test -n "$a_utf8_locale" +prepare_utf8_locale () { -+ if test -z "$GIT_TEST_UTF8_LOCALE" ++ if test -n "$GIT_TEST_UTF8_LOCALE" ++ then ++ : test_set_prereq UTF8 ++ elif test -n "${LC_ALL:-$LANG}" + then ++ case "${LC_ALL:-$LANG}" in ++ *.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8) ++ GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}" ++ ;; ++ esac ++ else + GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{ + p + q Makefile | 7 +++++++ ci/lib.sh | 1 + t/lib-git-svn.sh | 24 ++++++++++++++++++------ t/t9100-git-svn-basic.sh | 14 +++----------- t/t9115-git-svn-dcommit-funky-renames.sh | 6 +++--- t/t9129-git-svn-i18n-commitencoding.sh | 4 ++-- 6 files changed, 34 insertions(+), 22 deletions(-)diff --git a/Makefile b/Makefile index c3565fc0f8..502e0c9a81 100644 --- a/Makefile +++ b/Makefile@@ -398,6 +398,10 @@ all:: # with a different indexfile format version. If it isn't set the index # file format used is index-v[23]. # +# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing. +# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8 +# locale returned by "locale -a". +# # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime. # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.@@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP endif ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT @echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+ +endif +ifdef GIT_TEST_UTF8_LOCALE + @echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+ endif @echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+ ifdef GIT_PERF_REPEAT_COUNTdiff --git a/ci/lib.sh b/ci/lib.sh index d848c036c5..476c3f369f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh@@ -229,6 +229,7 @@ linux-musl) CC=gcc MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes" MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes" + MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8" ;; esacdiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh index 547eb3c31a..83efc17661 100644 --- a/t/lib-git-svn.sh +++ b/t/lib-git-svn.sh@@ -121,12 +121,24 @@ start_svnserve () { --listen-host 127.0.0.1 & } -prepare_a_utf8_locale () { - a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{ - p - q -}') - if test -n "$a_utf8_locale" +prepare_utf8_locale () { + if test -n "$GIT_TEST_UTF8_LOCALE" + then + : test_set_prereq UTF8 + elif test -n "${LC_ALL:-$LANG}" + then + case "${LC_ALL:-$LANG}" in + *.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8) + GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}" + ;; + esac + else + GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{ + p + q + }') + fi + if test -n "$GIT_TEST_UTF8_LOCALE" then test_set_prereq UTF8 elsediff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh index 1d3fdcc997..d5563ec35f 100755 --- a/t/t9100-git-svn-basic.sh +++ b/t/t9100-git-svn-basic.sh@@ -4,21 +4,13 @@ # test_description='git svn basic tests' -GIT_SVN_LC_ALL=${LC_ALL:-$LANG} GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./lib-git-svn.sh -case "$GIT_SVN_LC_ALL" in -*.UTF-8) - test_set_prereq UTF8 - ;; -*) - say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)" - ;; -esac +prepare_utf8_locale test_expect_success 'git svn --version works anywhere' ' nongit git svn --version@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" ' test ! -h "$SVN_TREE"/exec-2.sh && test_cmp help "$SVN_TREE"/exec-2.sh' -name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL" -LC_ALL="$GIT_SVN_LC_ALL" +name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE" +LC_ALL="$GIT_TEST_UTF8_LOCALE" export LC_ALL # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh index 9b44a44bc1..743fbe1fe4 100755 --- a/t/t9115-git-svn-dcommit-funky-renames.sh +++ b/t/t9115-git-svn-dcommit-funky-renames.sh@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' ' # > ... All of the above characters, except for the backslash, are converted # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the # > "Private use area") when creating or accessing files. -prepare_a_utf8_locale +prepare_utf8_locale test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' ' - LC_ALL=$a_utf8_locale && + LC_ALL=$GIT_TEST_UTF8_LOCALE && export LC_ALL && neq=$(printf "\201\202") && git config svn.pathnameencoding cp932 &&@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new # See the comment on the above test for setting of LC_ALL. test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' ' - LC_ALL=$a_utf8_locale && + LC_ALL=$GIT_TEST_UTF8_LOCALE && export LC_ALL && inf=$(printf "\201\207") && git config svn.pathnameencoding cp932 &&diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh index 2c213ae654..01e1e8a8f7 100755 --- a/t/t9129-git-svn-i18n-commitencoding.sh +++ b/t/t9129-git-svn-i18n-commitencoding.sh@@ -14,12 +14,12 @@ compare_git_head_with () { test_cmp current "$1" } -prepare_a_utf8_locale +prepare_utf8_locale compare_svn_head_with () { # extract just the log message and strip out committer info. # don't use --limit here since svn 1.1.x doesn't have it, - LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e ' + LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e ' use bytes; $/ = ("-"x72) . "\n"; my @x = <STDIN>; --2.32.0.rc3.5.gf3d78db977