[RFC PATCH] gitweb: use CGI with -utf8

I noticed that gitweb tries a lot to properly process UTF-8 data, for
example it prints my name correctly in log and commit information, but
it echos junk in the search field. It looks like:

	Michał Kiedrowicz

I don't know CGI well and I never touched gitewb code, but I found this
on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI '-utf8';
	my $value = params ('input');

I tried it and that fixed my problem. I'm not sure about the
consequences, maybe someone more experienced in CGI might help?
---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index abb5a79..74d45b1 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -10,7 +10,7 @@
 use 5.008;
 use strict;
 use warnings;
-use CGI qw(:standard :escapeHTML -nosticky);
+use CGI qw(:standard :escapeHTML -nosticky -utf8);
 use CGI::Util qw(unescape);
 use CGI::Carp qw(fatalsToBrowser set_message);
 use Encode;

-- 
1.7.3.4

Michał Kiedrowicz [off-list ref] writes:

I noticed that gitweb tries a lot to properly process UTF-8 data, for
example it prints my name correctly in log and commit information, but
it echos junk in the search field. It looks like:

	Michał Kiedrowicz

I don't know CGI well and I never touched gitewb code, but I found this
on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI '-utf8';
	my $value = params ('input');

I tried it and that fixed my problem. I'm not sure about the
consequences, maybe someone more experienced in CGI might help?

I have reworded this to form a proper commit message (see
Documentation/SubmittingPatches) and I'll resend this as a reply to
this email.

Does this actually work for you?  Because it doesn't work for me
(perhaps I have too old CGI module: what CGI.pm and what Perl version
do you use?).

See other solution to this in other reply to this email.

-- 
Jakub Narebski

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

According to "Using Unicode in a Perl CGI script" article on
http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
solution is to just import '-utf8' pragma for CGI module:

	use CGI '-utf8';
	my $value = params('input');

According to CGI module documentation, the '-utf8' pragma may cause
problems with POST requests containing binary files... but gitweb
currently do not use POST requests at all, so this should be not a
problem now.

Alternate solution would be to explicity decode query parameters when
storing them in %input_params (and perhaps also path_info).

[jn: reworded / rewritten commit message]

Signed-off-by: Michał Kiedrowicz <redacted>
Signed-off-by: Jakub Narębski <redacted>
---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..a7441ef 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -10,7 +10,7 @@
 use 5.008;
 use strict;
 use warnings;
-use CGI qw(:standard :escapeHTML -nosticky);
+use CGI qw(:standard :escapeHTML -nosticky -utf8);
 use CGI::Util qw(unescape);
 use CGI::Carp qw(fatalsToBrowser set_message);
 use Encode;

-- 
1.7.6

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.

Noticed-by: Michał Kiedrowicz [off-list ref]
Signed-off-by: Jakub Narębski <redacted>
---
 gitweb/gitweb.perl |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..55b2c24 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -52,7 +52,7 @@ sub evaluate_uri {
 	# as base URL.
 	# Therefore, if we needed to strip PATH_INFO, then we know that we have
 	# to build the base URL ourselves:
-	our $path_info = $ENV{"PATH_INFO"};
+	our $path_info = decode_utf8($ENV{"PATH_INFO"});
 	if ($path_info) {
 		if ($my_url =~ s,\Q$path_info\E$,, &&
 		    $my_uri =~ s,\Q$path_info\E$,, &&

@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }

@@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
 	}
 
 	my $cloud;
-	my $matched = $cgi->param('by_tag');
+	my $matched = $input_params{'ctag'};
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
 		foreach my $ctag (sort keys %ctags_lc) {

@@ -5282,7 +5282,7 @@ sub git_project_list_body {
 
 	my $check_forks = gitweb_check_feature('forks');
 	my $show_ctags  = gitweb_check_feature('ctags');
-	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
+	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
 	$check_forks = undef
 		if ($tagfilter || $searchtext);
 

@@ -6197,7 +6197,7 @@ sub git_tag {
 
 sub git_blame_common {
 	my $format = shift || 'porcelain';
-	if ($format eq 'porcelain' && $cgi->param('js')) {
+	if ($format eq 'porcelain' && $input_params{'javascript'}) {
 		$format = 'incremental';
 		$action = 'blame_incremental'; # for page title etc
 	}

-- 
1.7.6

According to "Using Unicode in a Perl CGI script" article on
http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
solution is to just import '-utf8' pragma for CGI module:

	use CGI '-utf8';
	my $value = params('input');

[...]

---

Except it doesn't work for me...

-- 
Jakub Narebski
Poland

Jakub Narebski [off-list ref] wrote:

Michał Kiedrowicz [off-list ref] writes:

quoted

I noticed that gitweb tries a lot to properly process UTF-8 data, for
example it prints my name correctly in log and commit information, but
it echos junk in the search field. It looks like:

	Michał Kiedrowicz

I don't know CGI well and I never touched gitewb code, but I found this
on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI '-utf8';
	my $value = params ('input');

I tried it and that fixed my problem. I'm not sure about the
consequences, maybe someone more experienced in CGI might help?

I have reworded this to form a proper commit message (see
Documentation/SubmittingPatches) and I'll resend this as a reply to
this email.

Thanks, your message is much better.

quoted

---
 gitweb/gitweb.perl |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index abb5a79..74d45b1 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -10,7 +10,7 @@
 use 5.008;
 use strict;
 use warnings;
-use CGI qw(:standard :escapeHTML -nosticky);
+use CGI qw(:standard :escapeHTML -nosticky -utf8);
 use CGI::Util qw(unescape);
 use CGI::Carp qw(fatalsToBrowser set_message);
 use Encode;

-- 

Does this actually work for you? 

Yes. It correctly displays "ł" in the search form.

 Because it doesn't work for me
(perhaps I have too old CGI module: what CGI.pm and what Perl version
do you use?).

$ perl --version

This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux
(with 12 registered patches, see perl -V for more detail)

$ eix -e CGI -c
[I] perl-core/CGI (3.510@01.02.2012): Simple Common Gateway Interface Class

See other solution to this in other reply to this email.

Jakub Narebski [off-list ref] wrote:

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

According to "Using Unicode in a Perl CGI script" article on
http://www.lemoda.net/cgi/perl-unicode/index.html the simplest
solution is to just import '-utf8' pragma for CGI module:

	use CGI '-utf8';
	my $value = params('input');

According to CGI module documentation, the '-utf8' pragma may cause
problems with POST requests containing binary files... but gitweb
currently do not use POST requests at all, so this should be not a
problem now.

This was exactly my feeling  when I sent this patch :).

Alternate solution would be to explicity decode query parameters when
storing them in %input_params (and perhaps also path_info).

[jn: reworded / rewritten commit message]

Signed-off-by: Michał Kiedrowicz <redacted>

Thanks, I forgot about that.

Jakub Narebski [off-list ref] wrote:

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

I'm sorry but this doesn't work for me. I would be happy to help if you
have some questions about it.

On Thu, 2 Jan 2012, Michał Kiedrowicz wrote:

Jakub Narebski [off-list ref] wrote:

quoted

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

I'm sorry but this doesn't work for me. I would be happy to help if you
have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
those two approaches should be equivalent.  The -utf8 pragma version
doesn't work for me at all, while this one works in that if finds what
it is supposed to, but shows garbage in search form.

Will investigate.
 

quoted

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it doesn't work with old CGI.pm version 3.10 from Perl v5.8.6.

[...]

quoted

@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }

-- 
Jakub Narebski
Poland

On Thu, 2 Feb 2012, Jakub Narebski wrote:

On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:

quoted

Jakub Narebski [off-list ref] wrote:

quoted

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

I'm sorry but this doesn't work for me. I would be happy to help if you
have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
those two approaches should be equivalent.  The -utf8 pragma version
doesn't work for me at all, while this one works in that if finds what
it is supposed to, but shows garbage in search form.

Is it what you mean by "this doesn't work for me", i.e. working search,
garbage in search field?
 

Will investigate.

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext) like
in gitweb, CGI.pm would read $cgi->param("s") by itself - without decoding.
To skip this we need to pass -force=>1  or  -override=>1 (i.e. further
changes to gitweb).

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-- 
Jakub Narebski
Poland

Jakub Narebski [off-list ref] wrote:

On Thu, 2 Feb 2012, Jakub Narebski wrote:

quoted

On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:

quoted

Jakub Narebski [off-list ref] wrote:

quoted

Gitweb tries hard to properly process UTF-8 data, by marking
output from git commands and contents of files as UTF-8 with
to_utf8() subroutine.  This ensures that gitweb would print
correctly UTF-8 e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode
input, namely query parameters.  The result is that one cannot
search for a string containing characters outside US-ASCII.
For example searching for "Michał Kiedrowicz" (containing
letter 'ł' - LATIN SMALL LETTER L WITH STROKE, with Unicode
codepoint U+0142, represented with 0xc5 0x82 bytes in UTF-8 and
percent-encoded as %C5%81) result in the following incorrect
data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as
two characters in Perl legacy encoding latin-1 (iso-8859-1),
because 's' query parameter is not processed explicitly as
UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI
script" article on
http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to
move from explicit $cgi->param(<label>) to
$input_params{<name>} in a few places.

I'm sorry but this doesn't work for me. I would be happy to help
if you have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says
that those two approaches should be equivalent.  The -utf8 pragma
version doesn't work for me at all, while this one works in that if
finds what it is supposed to, but shows garbage in search form.

Is it what you mean by "this doesn't work for me", i.e. working
search, garbage in search field?

I mean "garbage in search field". Search works even without the patch
(at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
don't have my notebook nearby at the moment to check).

quoted

Will investigate.

Thanks for your time spending on this. I wouldn't call this problem
"production critial" but it seems wrong to support UTF-8 everywhere
properly except for one place.

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
like in gitweb, CGI.pm would read $cgi->param("s") by itself -
without decoding. 

Makes sense. When I tried calling to_utf8() in the line that defines
textfield (this was my first approach to this problem), it haven't
changed anything.

To skip this we need to pass -force=>1  or
-override=>1 (i.e. further changes to gitweb).

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

Decoding UTF-8 is done when filling %input_params hash and $path_info
variable; the former required to move from explicit $cgi->param(<label>)
to $input_params{<name>} in a few places, which is a good idea anyway.

Another required change was to add -override=>1 parameter to
$cgi->textfield() invocation (in search form).  Otherwise CGI would
use values from query string if it is present, filling value from
$cgi->param... without decode_utf8().  As we are using value of
appropriate parameter anyway, -override=>1 doesn't change the
situation but makes gitweb fill search field correctly.

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it requires CGI 3.31 (I think), released with perl v5.8.9.

Noticed-by: Michał Kiedrowicz [off-list ref]
Signed-off-by: Jakub Narębski <redacted>
---
On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:

Jakub Narebski [off-list ref] wrote:

quoted

Is it what you mean by "this doesn't work for me", i.e. working
search, garbage in search field?

I mean "garbage in search field". Search works even without the patch
(at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
don't have my notebook nearby at the moment to check).

[...]

quoted

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
like in gitweb, CGI.pm would read $cgi->param("s") by itself -
without decoding. 

Makes sense. When I tried calling to_utf8() in the line that defines
textfield (this was my first approach to this problem), it haven't
changed anything.

Yes, and it doesn't makes sense in gitweb case - we use value of 
$cgi->param("s") as default value of text field anyway, but in
Unicode-aware way.
 

quoted

To skip this we need to pass -force=>1  or
-override=>1 (i.e. further changes to gitweb).

This patch does this.  

Does it make work for you?

quoted

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-utf8 pragma was added in CVS revision 1.238 of CGI.pm, which I think
is present in CGI 3.31, released with perl v5.8.9.  Theoretically gitweb
maintains backward compatibility with perl v5.8.3 or something 
("use 5.008;" but IIRC 5.8.3 is needed for correct Unicde handling anyway).

 gitweb/gitweb.perl |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..bd5fff9 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -52,7 +52,7 @@ sub evaluate_uri {
 	# as base URL.
 	# Therefore, if we needed to strip PATH_INFO, then we know that we have
 	# to build the base URL ourselves:
-	our $path_info = $ENV{"PATH_INFO"};
+	our $path_info = decode_utf8($ENV{"PATH_INFO"});
 	if ($path_info) {
 		if ($my_url =~ s,\Q$path_info\E$,, &&
 		    $my_uri =~ s,\Q$path_info\E$,, &&

@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }

@@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
 	}
 
 	my $cloud;
-	my $matched = $cgi->param('by_tag');
+	my $matched = $input_params{'ctag'};
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
 		foreach my $ctag (sort keys %ctags_lc) {

@@ -3873,7 +3873,7 @@ sub print_search_form {
 	                       -values => ['commit', 'grep', 'author', 'committer', 'pickaxe']) .
 	      $cgi->sup($cgi->a({-href => href(action=>"search_help")}, "?")) .
 	      " search:\n",
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "<span title=\"Extended regular expression\">" .
 	      $cgi->checkbox(-name => 'sr', -value => 1, -label => 're',
 	                     -checked => $search_use_regexp) .

@@ -5282,7 +5282,7 @@ sub git_project_list_body {
 
 	my $check_forks = gitweb_check_feature('forks');
 	my $show_ctags  = gitweb_check_feature('ctags');
-	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
+	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
 	$check_forks = undef
 		if ($tagfilter || $searchtext);
 

@@ -5994,7 +5994,7 @@ sub git_project_list {
 	}
 	print $cgi->startform(-method => "get") .
 	      "<p class=\"projsearch\">Search:\n" .
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "</p>" .
 	      $cgi->end_form() . "\n";
 	git_project_list_body(\@list, $order);

@@ -6197,7 +6197,7 @@ sub git_tag {
 
 sub git_blame_common {
 	my $format = shift || 'porcelain';
-	if ($format eq 'porcelain' && $cgi->param('js')) {
+	if ($format eq 'porcelain' && $input_params{'javascript'}) {
 		$format = 'incremental';
 		$action = 'blame_incremental'; # for page title etc
 	}

-- 
1.7.6

Jakub Narebski [off-list ref] wrote:

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

Decoding UTF-8 is done when filling %input_params hash and $path_info
variable; the former required to move from explicit $cgi->param(<label>)
to $input_params{<name>} in a few places, which is a good idea anyway.

Another required change was to add -override=>1 parameter to
$cgi->textfield() invocation (in search form).  Otherwise CGI would
use values from query string if it is present, filling value from
$cgi->param... without decode_utf8().  As we are using value of
appropriate parameter anyway, -override=>1 doesn't change the
situation but makes gitweb fill search field correctly.

Alternate solution would be to simply use the '-utf8' pragma (via
"use CGI '-utf8';"), but according to CGI.pm documentation it may
cause problems with POST requests containing binary files... and
it requires CGI 3.31 (I think), released with perl v5.8.9.

Noticed-by: Michał Kiedrowicz [off-list ref]
Signed-off-by: Jakub Narębski <redacted>
---
On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:

quoted

Jakub Narebski [off-list ref] wrote:

quoted

quoted

Is it what you mean by "this doesn't work for me", i.e. working
search, garbage in search field?

I mean "garbage in search field". Search works even without the patch
(at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
don't have my notebook nearby at the moment to check).

[...]

quoted

quoted

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
like in gitweb, CGI.pm would read $cgi->param("s") by itself -
without decoding. 

Makes sense. When I tried calling to_utf8() in the line that defines
textfield (this was my first approach to this problem), it haven't
changed anything.

Yes, and it doesn't makes sense in gitweb case - we use value of 
$cgi->param("s") as default value of text field anyway, but in
Unicode-aware way.
 

quoted

quoted

To skip this we need to pass -force=>1  or
-override=>1 (i.e. further changes to gitweb).

This patch does this.  

Does it make work for you?

Yes, it works for me. Search form properly displays "ł". Thanks!