From: Junio C Hamano <hidden> Date: 2016-06-15 22:43:55
Benjamin Close [off-list ref] writes:
quoted hunk
quoted
From 83042abf3967b455953cddeab43e33c1d59c6f03 Mon Sep 17 00:00:00 2001
From: Benjamin Close <redacted>
Date: Sun, 2 Dec 2007 15:09:00 -0800
Subject: [PATCH] Gitweb: Fix encoding to always translate rather than
sometimes fail
When performing the utf translation don't test if $res is defined.
It appears that it is defined even when the conversion fails. This causes
failures on the writing of the output stream which is expecting UTF.
@@ -696,12 +696,8 @@ sub validate_refname { sub to_utf8 { my $str = shift; my $res;- eval { $res = decode_utf8($str, Encode::FB_CROAK); };- if (defined $res) {- return $res;- } else {- return decode($fallback_encoding, $str, Encode::FB_DEFAULT);- }+ eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };+ return decode($fallback_encoding, $str, Encode::FB_DEFAULT); }
This is funny.
I thought the standard catch ... throw idiom in Perl was to do the above
like this:
my $res;
eval { $res = decode_utf8($str, Encode::FB_CROAK); };
if ($@) {
return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
}
return $res;
(alternatively, you can assign return value of eval {} to $res).
Monday 03 December 2007 Tarihinde 12:14:43 yazmıştı:
Benjamin Close [off-list ref] writes:
quoted
quoted
From 83042abf3967b455953cddeab43e33c1d59c6f03 Mon Sep 17 00:00:00 2001
From: Benjamin Close <redacted>
Date: Sun, 2 Dec 2007 15:09:00 -0800
Subject: [PATCH] Gitweb: Fix encoding to always translate rather than
sometimes fail
When performing the utf translation don't test if $res is defined.
It appears that it is defined even when the conversion fails. This causes
failures on the writing of the output stream which is expecting UTF.
@@ -696,12 +696,8 @@ sub validate_refname { sub to_utf8 { my $str = shift; my $res;- eval { $res = decode_utf8($str, Encode::FB_CROAK); };- if (defined $res) {- return $res;- } else {- return decode($fallback_encoding, $str, Encode::FB_DEFAULT);- }+ eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };+ return decode($fallback_encoding, $str, Encode::FB_DEFAULT); }
This is funny.
I thought the standard catch ... throw idiom in Perl was to do the above
like this:
my $res;
eval { $res = decode_utf8($str, Encode::FB_CROAK); };
if ($@) {
return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
}
return $res;
I think this is correct, but the current code in gitweb doesn't look correct
since it checks for $res and not $@.
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
I thought the standard catch ... throw idiom in Perl was to do the above
like this:
my $res;
eval { $res = decode_utf8($str, Encode::FB_CROAK); };
if ($@) {
return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
}
return $res;
I think this is correct, but the current code in gitweb doesn't look correct
since it checks for $res and not $@.
First version of the patch was created by Martin Koegler. I have
participated in creating the version which is now in gitweb, but I
have to say that I wrote it based on decode_utf8
documentation... which doesn't necessarily agree with facts :-(
I'm all for the "throw idion" version. Ack.
--
Jakub Narebski
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1
encoding of the same character sequence yields to different results.
[...]
eval { $res = decode_utf8(...); }
if ($@)
return decode(...);
return $res
or
eval { $res = decode_utf8(...); }
if (defined $res)
return $res;
else
return decode(...);
show the same (wrong) behaviour on Debian sarge. They do not always
decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work
On Debian etch, both versions are working.
I don't know enough Perl to decide if it is a bug in gitweb usage
of decode_utf8, if it is a bug in your version of Encode, or if it
is bug in Encode.
Send copy of this mail to maintainers of Encode perl module.
--
Jakub Narebski
Poland
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1
encoding of the same character sequence yields to different results.
quoted
quoted
I thought the standard catch ... throw idiom in Perl was to do the above
like this:
my $res;
eval { $res = decode_utf8($str, Encode::FB_CROAK); };
if ($@) {
return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
}
return $res;
I think this is correct, but the current code in gitweb doesn't look correct
since it checks for $res and not $@.
First version of the patch was created by Martin Koegler. I have
participated in creating the version which is now in gitweb, but I
have to say that I wrote it based on decode_utf8
documentation... which doesn't necessarily agree with facts :-(
eval { $res = decode_utf8(...); }
if ($@)
return decode(...);
return $res
or
eval { $res = decode_utf8(...); }
if (defined $res)
return $res;
else
return decode(...);
show the same (wrong) behaviour on Debian sarge. They do not always
decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work
On Debian etch, both versions are working.
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1
encoding of the same character sequence yields to different results.
For the record, this was on a debian sid machine.
#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
and the result of not using the original patch was:
<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at /usr/lib/perl/5.8/Encode.pm line 166.
</pre>
I haven't tried the other solutions tested here.
quoted
eval { $res = decode_utf8(...); }
if ($@)
return decode(...);
return $res
or
eval { $res = decode_utf8(...); }
if (defined $res)
return $res;
else
return decode(...);
show the same (wrong) behaviour on Debian sarge. They do not always
decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work
On Debian etch, both versions are working.
I don't know enough Perl to decide if it is a bug in gitweb usage
of decode_utf8, if it is a bug in your version of Encode, or if it
is bug in Encode.
Send copy of this mail to maintainers of Encode perl module.
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a
latin1 encoding of the same character sequence yields to different
results.
For the record, this was on a debian sid machine.
#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
and the result of not using the original patch was:
<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166. </pre>
Can you try the attached patch?
--
Never learn by your mistakes, if you do you may never dare to try again.
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a
latin1 encoding of the same character sequence yields to different
results.
For the record, this was on a debian sid machine.
#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
and the result of not using the original patch was:
<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166. </pre>
Can you try the attached patch?
I confirm that the patch corrects the problem.
Without it I get the Cannot decode string error. With it gitweb displays
correctly.
Cheers,
Benjamin
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:56
On Tue, 4 Dec 2007, Benjamin Close wrote:
On Tue, Dec 04, 2007 at 12:20:26AM +0200, Ismail Donmez wrote:
quoted
Can you try the attached patch?
I confirm that the patch corrects the problem.
Without it I get the Cannot decode string error. With it gitweb
displays correctly.
But the patch _avoids_ issue (des not convert owner to utf8), rather
than solving it, if I understand it correctly. What if gecos is in
utf-8?
--
Jakub Narebski
Poland
Tuesday 04 December 2007 Tarihinde 01:37:35 yazmıştı:
On Tue, 4 Dec 2007, Benjamin Close wrote:
quoted
On Tue, Dec 04, 2007 at 12:20:26AM +0200, Ismail Donmez wrote:
quoted
Can you try the attached patch?
I confirm that the patch corrects the problem.
Without it I get the Cannot decode string error. With it gitweb
displays correctly.
But the patch _avoids_ issue (des not convert owner to utf8), rather
than solving it, if I understand it correctly. What if gecos is in
utf-8?
Indeed its a workaround but UTF-8 username is correctly displayed in gitweb so
my understanding was gecos field is already UTF-8.
--
Never learn by your mistakes, if you do you may never dare to try again.
From: Martin Koegler <hidden> Date: 2016-06-15 22:43:56
On Mon, Dec 03, 2007 at 06:02:54PM +0100, Jakub Narebski wrote:
On Mon, 3 Dec 2007, Martin Koegler wrote:
quoted
eval { $res = decode_utf8(...); }
if ($@)
return decode(...);
return $res
or
eval { $res = decode_utf8(...); }
if (defined $res)
return $res;
else
return decode(...);
show the same (wrong) behaviour on Debian sarge. They do not always
decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work
On Debian etch, both versions are working.
I don't know enough Perl to decide if it is a bug in gitweb usage
of decode_utf8, if it is a bug in your version of Encode, or if it
is bug in Encode.
Send copy of this mail to maintainers of Encode perl module.
The bug affects old versions of perl (Debian sarge = oldstable).
As it works on the newer Debian etch, do you really think, that it is
a good idea to report issue?
How would you handle a bug report, which reports a bug for gitweb in GIT
1.4, and tells you, that a newer versions works?
As Debian sarge has reached its end of life, the distribution will
probable also issue no update.
mfg Martin Kögler
Tuesday 04 December 2007 Tarihinde 09:50:28 yazmıştı:
The bug affects old versions of perl (Debian sarge = oldstable).
As it works on the newer Debian etch, do you really think, that it is
a good idea to report issue?
Same problem here with v5.8.8 which is latest stable perl5 release.
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a
latin1
encoding of the same character sequence yields to different results.
For the record, this was on a debian sid machine.
#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
and the result of not using the original patch was:
<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166.
</pre>
I haven't tried the other solutions tested here.
Debian etch also has v5.8.8.
My main question is, why is the error not catched?
I'm not a perl programmer, but in your patch the first line is a
NOP. The return in eval seems to only returns from the eval block, so
any text is decoded as latin1 with the second statement.
In the original version, decode($fallback_encoding, $str,
Encode::FB_DEFAULT) can not emit an error, else it would in your
version too.
In your version, eval is able to surpress the error of
decode_utf8($str, Encode::FB_CROAK);, but not in the original version.
Strange.
mfg Martin Kögler
This version is broken on Debian sarge and etch. Feeding a UTF-8 and a
latin1
encoding of the same character sequence yields to different results.
For the record, this was on a debian sid machine.
#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi
and the result of not using the original patch was:
<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166.
</pre>
I haven't tried the other solutions tested here.
Debian etch also has v5.8.8.
My main question is, why is the error not catched?
I'm not a perl programmer, but in your patch the first line is a
NOP. The return in eval seems to only returns from the eval block, so
any text is decoded as latin1 with the second statement.
In the original version, decode($fallback_encoding, $str,
Encode::FB_DEFAULT) can not emit an error, else it would in your
version too.
In your version, eval is able to surpress the error of
decode_utf8($str, Encode::FB_CROAK);, but not in the original version.
I think just a better method is to use (not tested):
if( is_utf8($str) )
{
return decode_utf8($str);
}
else {
return decode($str);
}
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
From: Martin Koegler <hidden> Date: 2016-06-15 22:43:56
On Tue, Dec 04, 2007 at 09:55:04AM +0200, Ismail Dönmez wrote:
Tuesday 04 December 2007 Tarihinde 09:50:28 yazm????t??:
quoted
The bug affects old versions of perl (Debian sarge = oldstable).
As it works on the newer Debian etch, do you really think, that it is
a good idea to report issue?
Same problem here with v5.8.8 which is latest stable perl5 release.
I have put together a small perl script, which tests the various ways
of decoding, which have been posted on the list. The first test is
wrong by design. A working decoding method should result in
"#öäü#äöü".
Debian sarge:
#öäü#ÀöÌ
##äöü
##äöü
##äöü
Debian etch, OpenSuSE 10.2, Fedora 7:
#öäü#ÀöÌ
#öäü#äöü
#öäü#äöü
#öäü#äöü
mfg Martin Kögler
#!/usr/bin/perl
use Encode;
sub t {
my $str = shift;
my ($res);
eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
return decode("latin1", $str, Encode::FB_DEFAULT);
}
sub t1 {
my $str = shift;
my ($res);
eval { ($res = decode_utf8($str, Encode::FB_CROAK)); };
if ($@) {
return decode("latin1", $str, Encode::FB_DEFAULT); }
else
{ return $res; }
}
sub t2 {
my $str = shift;
my ($res);
eval { $res = decode_utf8($str, Encode::FB_CROAK); };
if (defined $res) {
return $res;
} else {
return decode("latin1", $str, Encode::FB_DEFAULT);
}
}
sub t3 {
my $str = shift;
my $res;
eval { $res = decode_utf8 ($str, 1); };
return $res || decode('latin1', $str);
}
print t("#öäü");
print t("#ÀöÌ");
print "\n";
print t1("#öäü");
print t1("#ÀöÌ");
print "\n";
print t2("#öäü");
print t2("#ÀöÌ");
print "\n";
print t3("#öäü");
print t3("#ÀöÌ");
print "\n";
From: Martin Koegler <hidden> Date: 2016-06-15 22:43:56
On Tue, Dec 04, 2007 at 10:12:50AM +0200, Ismail Dönmez wrote:
I think just a better method is to use (not tested):
if( is_utf8($str) )
{
return decode_utf8($str);
}
else {
return decode($str);
}
I already tried this function. It does not test, if a string is
really UTF-8. It seems to be to intended to check, if perl stores
the string internally in a multi byte encoding.
mfg Martin Kögler.
Tuesday 04 December 2007 10:16:34 Martin Koegler yazmıştı:
[...]
print t("#öäü");
print t("#ÀöÌ");
print "\n";
How about this one, doesn't even use Encode, uses just built-in utf8
function :
[~]> cat test.pl
binmode STDOUT, ':utf8';
my $str = "#öäü";
if (utf8::valid($str))
{
utf8::decode($str);
}
print $str."\n";
[~]> perl test.pl
#öäü
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
Tuesday 04 December 2007 10:28:59 Ismail Dönmez yazmıştı:
Tuesday 04 December 2007 10:16:34 Martin Koegler yazmıştı:
[...]
quoted
print t("#öäü");
print t("#ÀöÌ");
print "\n";
How about this one, doesn't even use Encode, uses just built-in utf8
function :
[~]> cat test.pl
binmode STDOUT, ':utf8';
my $str = "#öäü";
if (utf8::valid($str))
{
utf8::decode($str);
}
print $str."\n";
[~]> perl test.pl
#öäü
Following to_utf8 function works for me :
sub to_utf8 {
· my $str = shift;
if(utf8::valid($str))
{
utf8::decode($str);
}
·
return $str;
}
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
From: Martin Koegler <hidden> Date: 2016-06-15 22:43:56
On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail Dönmez wrote:
Following to_utf8 function works for me :
For me too (Debian sarge+etch).
sub to_utf8 {
· my $str = shift;
if(utf8::valid($str))
{
utf8::decode($str);
}
·
return $str;
In the original thread, there was some discussion, that some people
might want a different fallback endcoding. So mayme you should
keep the second call to decode for the fallback encoding.
Tuesday 04 December 2007 10:44:12 Martin Koegler yazmıştı:
On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail Dönmez wrote:
quoted
Following to_utf8 function works for me :
For me too (Debian sarge+etch).
Thanks for testing.
quoted
sub to_utf8 {
· my $str = shift;
if(utf8::valid($str))
{
utf8::decode($str);
}
·
return $str;
In the original thread, there was some discussion, that some people
might want a different fallback endcoding. So mayme you should
keep the second call to decode for the fallback encoding.
Probably, I just wanted to fix this damn UTF-8 bug surfacing over and over =)
Regards,
ismail
--
Never learn by your mistakes, if you do you may never dare to try again.
@@ -695,10 +695,9 @@ sub validate_refname {# in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginningsubto_utf8{my$str=shift;-my$res;-eval{$res=decode_utf8($str,Encode::FB_CROAK);};-if(defined$res){-return$res;+if(utf8::valid($str)){+utf8::decode($str);+return$str;}else{returndecode($fallback_encoding,$str,Encode::FB_DEFAULT);}
--
Never learn by your mistakes, if you do you may never dare to try again.
# in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning
sub to_utf8 {
my $str = shift;
- my $res;
- eval { $res = decode_utf8($str, Encode::FB_CROAK); };
- if (defined $res) {
- return $res;
+ if (utf8::valid($str)) {
+ utf8::decode($str);
+ return $str;
This is good as it fixes another problem which some may have
encountered. On at least one distro that I use (Red Hat Enterprise
Linux 3) the Encode module is very old (it's 1.83; the latest release
is 2.23), and so gitweb won't even run, dying during compilation with
this:
Too many arguments for Encode::decode_utf8 at gitweb.cgi line 686,
near "Encode::FB_CROAK)"
Of course, the workaround is to install a newer version of the module,
but this patch eliminates that dependency which is IMO a good thing.
Cheers,
Wincent