gitweb - encoding problems

I use ISO-8859-1 as my locale, so my blobs, commits and tags are in
this encoding.

On perl v5.8.6, decode_utf8 of any non utf-8 value returns undefined:

$cat xy
#!/usr/bin/perl
use strict;
use warnings;
use CGI qw(:standard :escapeHTML -nosticky);
use CGI::Util qw(unescape);
use CGI::Carp qw(fatalsToBrowser);
use Encode;
use Fcntl ':mode';
use File::Find qw();
use File::Basename qw(basename);

binmode STDOUT, ':utf8';

print decode_utf8('äöü');
$perl xy
[Mon May 21 22:00:00 2007] xy: Use of uninitialized value in print at xy line 14.

If gitweb encounters, eg. an "Umlaut" (äöü) in a commit/tag, use of
uninitialized value message are generated. In one case,
decode_utf8($long) in format_subject_html is undefined, which results
in a invalid link (a tag contains only title without any value
assignment) and a browser message, that the html is not valid.

The previous installed version of git/gitweb (1.5.0rc3) showed only
small black rhombuses, but didn't generate "uninitialized value"
messages or invalid html.

So there is regression between git-1.5.0 and git-1.5.2.

Adding $var = encode_utf8($var) if (!defined decode_utf8($var)) for
each "uninitialized value" message results in a correct result for me.

I wanted to post a patch with these changes, as it solved my locale problem.
But then I tried the same a different computer with a newer perl (v5.8.8).
$ cat x
#!/usr/bin/perl
use strict;
use warnings;
use CGI qw(:standard :escapeHTML -nosticky);
use CGI::Util qw(unescape);
use CGI::Carp qw(fatalsToBrowser);
use Encode;
use Fcntl ':mode';
use File::Find qw();
use File::Basename qw(basename);

binmode STDOUT, ':utf8';

print decode_utf8('äöü');
$ perl x
ￜￜï¿

Here perl decodes the ISO-8859-1 text to something differnent:
00000000  ef bf bd ef bf bd ef bf  bd                       |ￜￜï¿

The result is, that all "Umlaute" are shown as a small black rhombus
in gitweb (and no invalid html).

mfg Martin Kögler

cat x |hexdump -C
00000000  23 21 2f 75 73 72 2f 62  69 6e 2f 70 65 72 6c 0a  |#!/usr/bin/perl.|
00000010  75 73 65 20 73 74 72 69  63 74 3b 0a 75 73 65 20  |use strict;.use |
00000020  77 61 72 6e 69 6e 67 73  3b 0a 75 73 65 20 43 47  |warnings;.use CG|
00000030  49 20 71 77 28 3a 73 74  61 6e 64 61 72 64 20 3a  |I qw(:standard :|
00000040  65 73 63 61 70 65 48 54  4d 4c 20 2d 6e 6f 73 74  |escapeHTML -nost|
00000050  69 63 6b 79 29 3b 0a 75  73 65 20 43 47 49 3a 3a  |icky);.use CGI::|
00000060  55 74 69 6c 20 71 77 28  75 6e 65 73 63 61 70 65  |Util qw(unescape|
00000070  29 3b 0a 75 73 65 20 43  47 49 3a 3a 43 61 72 70  |);.use CGI::Carp|
00000080  20 71 77 28 66 61 74 61  6c 73 54 6f 42 72 6f 77  | qw(fatalsToBrow|
00000090  73 65 72 29 3b 0a 75 73  65 20 45 6e 63 6f 64 65  |ser);.use Encode|
000000a0  3b 0a 75 73 65 20 46 63  6e 74 6c 20 27 3a 6d 6f  |;.use Fcntl ':mo|
000000b0  64 65 27 3b 0a 75 73 65  20 46 69 6c 65 3a 3a 46  |de';.use File::F|
000000c0  69 6e 64 20 71 77 28 29  3b 0a 75 73 65 20 46 69  |ind qw();.use Fi|
000000d0  6c 65 3a 3a 42 61 73 65  6e 61 6d 65 20 71 77 28  |le::Basename qw(|
000000e0  62 61 73 65 6e 61 6d 65  29 3b 0a 0a 62 69 6e 6d  |basename);..binm|
000000f0  6f 64 65 20 53 54 44 4f  55 54 2c 20 27 3a 75 74  |ode STDOUT, ':ut|
00000100  66 38 27 3b 0a 0a 70 72  69 6e 74 20 64 65 63 6f  |f8';..print deco|
00000110  64 65 5f 75 74 66 38 28  27 e4 f6 fc 27 29 3b 0a  |de_utf8('äöü');.|
00000120

On Monday 21 May 2007 23:57:21 you wrote:

binmode STDOUT, ':utf8';

print decode_utf8('äöü');

[~]> perl test.pl
äöü

[~]> cat test.pl
use Encode;
binmode STDOUT, ':utf8';

print decode_utf8('äöü'),"\n";
[cartman@southpark][00:08:15]
[~]> perl --version

This is perl, v5.8.8 built for i686-linux

Copyright 1987-2006, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

You got an old Encode.

-- 
Perfect is the enemy of good

On Mon, 2007-05-21 at 22:57 +0200, Martin Koegler wrote:

I use ISO-8859-1 as my locale, so my blobs, commits and tags are in
this encoding. 

That's a very strange thing for anyone to do in the 21st century.
Did you configure this archaic thing correctly in .git/config?

Otherwise, gitweb will assume that you're using utf-8 like any normal
person would, and of course you'll have problems when it tries to deal
with your legacy character set as if it were something sensible.

-- 
dwmw2

On Thu, 22 May 2007, David Woodhouse wrote:

On Mon, 2007-05-21 at 22:57 +0200, Martin Koegler wrote:

quoted

I use ISO-8859-1 as my locale, so my blobs, commits and tags are in
this encoding. 

That's a very strange thing for anyone to do in the 21st century.
Did you configure this archaic thing correctly in .git/config?

Otherwise, gitweb will assume that you're using utf-8 like any normal
person would, and of course you'll have problems when it tries to deal
with your legacy character set as if it were something sensible.

Actually gitweb does not respect i18n.* configuration variables and
happily assumes that everything is in utf-8, with the exception of 
*_plain views, which are send :raw.

If you decide to implement supporting encodings other that utf-8 in 
gitweb, please remember that some (like git-show, git-log) but not all 
parts (like git-rev-list or --pretty=raw) do the decoding/encoding. And 
that git can be compiled without iconv support. And that comits might 
be in different encodings, which should be given by 'encoding' header, 
but there is no way to guess encoding for a blob, or for a file names.


git-commit(1):
 i18n.commitEncoding::
     Character encoding the commit messages are stored in; git itself
     does not care per se, but this information is necessary e.g. when
     importing commits from emails or in the gitk graphical history
     browser (and possibly at other places in the future or in other
     porcelains). See e.g. gitlink:git-mailinfo[1]. Defaults to 'utf-8'.

 i18n.logOutputEncoding::
     Character encoding the commit messages are converted to when
     running `git-log` and friends.

-- 
Jakub Narebski
Poland