Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

On Thu, 2 Feb 2012, Jakub Narebski wrote:

On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:

quoted

Jakub Narebski [off-list ref] wrote:

quoted

Gitweb tries hard to properly process UTF-8 data, by marking output
from git commands and contents of files as UTF-8 with to_utf8()
subroutine.  This ensures that gitweb would print correctly UTF-8
e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input,
namely query parameters.  The result is that one cannot search for a
string containing characters outside US-ASCII.  For example searching
for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
bytes in UTF-8 and percent-encoded as %C5%81) result in the following
incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two
characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script"
article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

This is done when filling %input_params hash; this required to move
from explicit $cgi->param(<label>) to $input_params{<name>} in a few
places.

I'm sorry but this doesn't work for me. I would be happy to help if you
have some questions about it.

Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says that
those two approaches should be equivalent.  The -utf8 pragma version
doesn't work for me at all, while this one works in that if finds what
it is supposed to, but shows garbage in search form.

Is it what you mean by "this doesn't work for me", i.e. working search,
garbage in search field?
 

Will investigate.

Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext) like
in gitweb, CGI.pm would read $cgi->param("s") by itself - without decoding.
To skip this we need to pass -force=>1  or  -override=>1 (i.e. further
changes to gitweb).

-utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-- 
Jakub Narebski
Poland