gitweb on kernel.org and UTF-8

Is it possible that the UTF-8 check in gitweb running on
kernel.org machine is somehow too strict?

The following two commits in git.git repository are not showing
properly.

I have a track record of getting peoples' names wrong, so I
double checked my commit objects, and as far as I can tell, all
of them are encoded in UTF-8 properly (or at least I can view
what I expect if I throw raw bytes from the commit objects at my
Firefox):

        c3df8568424684bbcc7df7722eb3ec34bdae8b2d

        This is from Yoshifuji-san; the third character in
        author name field is mangled.

	bb931cf9d73d94d9940b6d0ee56b6c13ad42f1a0

	This is from Lukas Sandstr*m; o with Umlaut on top is
	showing a ?.  Incidentally, the blob that records recent
	version of Documentation/git-pack-redundant.txt has his
	name in it, which has the same ? problem, but "plain"
	option shows his name correctly in UTF-8.

Interestingly enough, my name spelled in Japanese
(Documentatino/git-lost-found.txt) is intact.  Am I getting a
VIP treatment somehow?

Junio C Hamano wrote:

Is it possible that the UTF-8 check in gitweb running on
kernel.org machine is somehow too strict?

The following two commits in git.git repository are not showing
properly.

I have a track record of getting peoples' names wrong, so I
double checked my commit objects, and as far as I can tell, all
of them are encoded in UTF-8 properly (or at least I can view
what I expect if I throw raw bytes from the commit objects at my
Firefox):

        c3df8568424684bbcc7df7722eb3ec34bdae8b2d

        This is from Yoshifuji-san; the third character in
        author name field is mangled.

	bb931cf9d73d94d9940b6d0ee56b6c13ad42f1a0

	This is from Lukas Sandstr*m; o with Umlaut on top is
	showing a ?.  Incidentally, the blob that records recent
	version of Documentation/git-pack-redundant.txt has his
	name in it, which has the same ? problem, but "plain"
	option shows his name correctly in UTF-8.

Interestingly enough, my name spelled in Japanese
(Documentatino/git-lost-found.txt) is intact.  Am I getting a
VIP treatment somehow?

I think it's missing a "binmode STDOUT, ':utf8';" somewhere...

For what it's worth, I looked at both the above examples and the binary 
encoding in the git repository is undoubtedly correct; the two 
characters are U+82F1/E8 8B B1 (英) and U+00F6/C3 B6 (ö) respectively, 
both of which are 100% valid UTF-8.

	-hpa

On Tue, Nov 22, 2005 at 04:59:16PM -0800, H. Peter Anvin wrote:

Junio C Hamano wrote:

quoted

Is it possible that the UTF-8 check in gitweb running on
kernel.org machine is somehow too strict?

The following two commits in git.git repository are not showing
properly.

I have a track record of getting peoples' names wrong, so I
double checked my commit objects, and as far as I can tell, all
of them are encoded in UTF-8 properly (or at least I can view
what I expect if I throw raw bytes from the commit objects at my
Firefox):

       c3df8568424684bbcc7df7722eb3ec34bdae8b2d

       This is from Yoshifuji-san; the third character in
       author name field is mangled.

bb931cf9d73d94d9940b6d0ee56b6c13ad42f1a0

This is from Lukas Sandstr*m; o with Umlaut on top is
showing a ?.  Incidentally, the blob that records recent
version of Documentation/git-pack-redundant.txt has his
name in it, which has the same ? problem, but "plain"
option shows his name correctly in UTF-8.

Interestingly enough, my name spelled in Japanese
(Documentatino/git-lost-found.txt) is intact.  Am I getting a
VIP treatment somehow?

I think it's missing a "binmode STDOUT, ':utf8';" somewhere...

For what it's worth, I looked at both the above examples and the binary 
encoding in the git repository is undoubtedly correct; the two 
characters are U+82F1/E8 8B B1 (英) and U+00F6/C3 B6 (ö) respectively, 
both of which are 100% valid UTF-8.

Should be fine now. The escapeHTML() garbled the utf8 "ö", and the
decode() failed that.

Thanks,
Kay

Kay Sievers wrote:

Should be fine now. The escapeHTML() garbled the utf8 "ö", and the
decode() failed that.

Indeed, looks much better.

Now if I could only figure out why both Konsole and Firefox seems to use 
a standalone cedilla to represent U+FFFD, instead of something more 
logical like an inverted question mark or empty box.

	-hpa