[Buildroot] [PATCH 0/2] package/tvheadend: needs an UTF8 locale

Hello All!

This short series is a band-aid to fix the build of tvheadend on systems
where the current locale is not UTF8.


Regards,
Yann E. MORIN.


The following changes since commit d02655cd2157636d6150daf3a1344b25141780c1

  package/ser2net: disable threads for toolchains without threads support (2016-11-06 15:11:21 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 6f09b8c02099896660df1f6cfe177c92d946f01c

  package/tvheadend: needs a host with an UTF8 locale (2016-11-06 21:21:34 +0100)


----------------------------------------------------------------
Yann E. MORIN (2):
      core: find an UFT8 locale on the host
      package/tvheadend: needs a host with an UTF8 locale

 Makefile                         |  3 +++
 package/tvheadend/tvheadend.mk   |  5 +++--
 support/scripts/find-utf8-locale | 48 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 2 deletions(-)
 create mode 100755 support/scripts/find-utf8-locale

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Some packages need an UTF8 locale to build sucessfully, so find one for
them.

We try hard to have a locale as close as possible to the one currently
used by the user, so as to output messages in his language. We first try
any of the current locale settings to see if they are fit, then we look
if said settigns have an UTF8 variant, then look for the same language
for other countries, and finally fallback to some well-known locales.

Signed-off-by: "Yann E. MORIN" <redacted>
Cc: Arnout Vandecappelle <redacted>
Cc: Romain Naour <redacted>
---
 Makefile                         |  3 +++
 support/scripts/find-utf8-locale | 48 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)
 create mode 100755 support/scripts/find-utf8-locale

diff --git a/Makefile b/Makefile
index 4ea64b3..4832160 100644
--- a/Makefile
+++ b/Makefile

@@ -244,6 +244,9 @@ export LANG=C
 export LC_ALL=C
 endif
 
+# Some packages want an UTF8 locale, so find one.
+UTF8_LOCALE := $(shell support/scripts/find-utf8-locale)
+
 # To put more focus on warnings, be less verbose as default
 # Use 'make V=1' to see the full commands
 ifeq ("$(origin V)", "command line")

diff --git a/support/scripts/find-utf8-locale b/support/scripts/find-utf8-locale
new file mode 100755
index 0000000..278247c
--- /dev/null
+++ b/support/scripts/find-utf8-locale

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Try to find an UTF8 locale.
+#
+# We try to find a locale that is as close as possible to the one that
+# the user currently uses. For that, we look at the same environment
+# variables that glibc uses: $LANGUAGE, then $LC_ALL, then $LC_XXX and
+# finally $LANG. $LANGUAGE can be a colon-separated list.
+# https://www.gnu.org/software/libc/manual/html_node/Using-gettextized-software.html#Using-gettextized-software
+
+# First, see if any of those variables is already an UTF8 locale.
+for l in ${LANGUAGE//:/ } ${LC_ALL} ${LC_MESSAGES} ${LANG}; do
+    case "${l,,}" in
+    (*.utf8|*.utf-8)    printf "%s" "${l}"; exit 0;;
+    esac
+done
+
+# No luck. Try to see if any of those have an UTF8 variant.
+for l in ${LANGUAGE//:/ } ${LC_ALL} ${LC_MESSAGES} ${LANG}; do
+    l="$(locale -a |grep -i -E "^${l}\.utf.?8" |head -n 1)"
+    if [ -n "${l}" ]; then
+        printf "%s" "${l}"
+        exit 0
+    fi
+done
+
+# Still no luck. Try to see if we have a locale for the same language,
+# but for another country.
+for l in ${LANGUAGE//:/ } ${LC_ALL} ${LC_MESSAGES} ${LANG}; do
+    l="$(locale -a |grep -i -E "^${l%_*}_.+\.utf.?8" |head -n 1)"
+    if [ -n "${l}" ]; then
+        printf "%s" "${l}"
+        exit 0
+    fi
+done
+
+# We're really out of luck. Try well-known UTF8 locales.
+for l in en_US.utf8 en_GB.utf8 C.UTF-8; do
+    l="$(locale -a |grep -i -E "^${l}" |head -n 1)"
+    if [ -n "${l}" ]; then
+        printf "%s" "${l}"
+        exit 0
+    fi
+done
+
+# We don't have an easy life, no we don't. And we don't have an
+# UTF8 locale either... :-(
+exit 1

-- 
2.7.4

The mkbundle python script requires an UTF8 locale to encode non-ASCII
characters.

Force the locale on the configure, build and install command lines.

Fixes:
    http://autobuild.buildroot.org/results/6fe/6fe3331f1f1d54cb977bb61a09d7707f19f7eafa/
    http://autobuild.buildroot.org/results/d1d/d1dda2c1d5af8fbaa09ecbf4a87f3892369b9e53/
    [...]

Signed-off-by: "Yann E. MORIN" <redacted>
Cc: Romain Naour <redacted>
Cc: Arnout Vandecappelle <redacted>

---
There is still a corner case, where the user has a build machine that
has no UTF-8 locale. This is deemed a corner case which we do not want
to address yet for master. Changes in how we handle the UTF8 locale will
come in the -next branch.
---
 package/tvheadend/tvheadend.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/package/tvheadend/tvheadend.mk b/package/tvheadend/tvheadend.mk
index a3afec8..d5b58b4 100644
--- a/package/tvheadend/tvheadend.mk
+++ b/package/tvheadend/tvheadend.mk

@@ -70,6 +70,7 @@ TVHEADEND_PRE_CONFIGURE_HOOKS += TVHEADEND_INSTALL_DTV_SCAN_TABLES
 
 define TVHEADEND_CONFIGURE_CMDS
 	(cd $(@D);						\
+		LC_ALL=$(UTF8_LOCALE)					\
 		$(TARGET_CONFIGURE_OPTS)			\
 		$(TARGET_CONFIGURE_ARGS)			\
 		CFLAGS="$(TVHEADEND_CFLAGS)"			\

@@ -88,11 +89,11 @@ define TVHEADEND_CONFIGURE_CMDS
 endef
 
 define TVHEADEND_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) LC_ALL=$(UTF8_LOCALE) $(MAKE) -C $(@D)
 endef
 
 define TVHEADEND_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
+	$(TARGET_MAKE_ENV) LC_ALL=$(UTF8_LOCALE) $(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
 endef
 
 # Remove documentation and source files that are not needed because we

-- 
2.7.4

Hi Yann, all,

Le 06/11/2016 ? 21:23, Yann E. MORIN a ?crit :

Some packages need an UTF8 locale to build sucessfully, so find one for

s/sucessfully/successfully/

them.

We try hard to have a locale as close as possible to the one currently
used by the user, so as to output messages in his language. We first try
any of the current locale settings to see if they are fit, then we look
if said settigns have an UTF8 variant, then look for the same language

s/settigns/settings/

for other countries, and finally fallback to some well-known locales.

Tested with:
LC_ALL=fr_FR.utf8

Tested-by: Romain Naour <redacted>

Best regards,
Romain

Hi Yann,

Le 06/11/2016 ? 21:23, Yann E. MORIN a ?crit :

The mkbundle python script requires an UTF8 locale to encode non-ASCII
characters.

Force the locale on the configure, build and install command lines.

Fixes:
    http://autobuild.buildroot.org/results/6fe/6fe3331f1f1d54cb977bb61a09d7707f19f7eafa/
    http://autobuild.buildroot.org/results/d1d/d1dda2c1d5af8fbaa09ecbf4a87f3892369b9e53/
    [...]

Signed-off-by: "Yann E. MORIN" <redacted>
Cc: Romain Naour <redacted>
Cc: Arnout Vandecappelle <redacted>

Tested with:
LC_ALL=fr_FR.utf8

Reviewed-by: Romain Naour <redacted>
Tested-by: Romain Naour <redacted>

Best regards,
Romain