gitweb forgets to send utf8 header for raw blob views

Hi,


I have configured gitweb to use utf8, and that works for text blob views 
like on
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob;f=bin/git-forest;hb=HEAD
but it does not for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD


Jan

Jan Engelhardt wrote:

[utf8 does not work] for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD

Gitweb should probably not be recoding blobs, so the best I can think of 
is check for UTF-8 validity and add charset=utf-8 in that case (and in 
other cases leave the charset undeclared).

The drawback with that is that we cannot send plain blobs without 
reading them into memory (or reading them twice), since we have to check 
for UTF-8 validity of the whole blob before sending it.  (Gitweb is 
currently reading the whole blob into memory, but that's unnecessary and 
could be changed in the future.)

After my next refactoring, there *might* be some chance to easily 
implement something like "if it's smaller than x KB (e.g. 512), read it 
into memory, check for valid UTF-8 and optionally add charset=utf-8, 
otherwise don't read it into memory and send it without charset=utf-8 
[or perhaps check for BOM presence at the beginning]."  I'll remember 
if/when it comes up in my refactoring and get back to the mailing list 
about it.

-- Lea

Jan Engelhardt [off-list ref] writes:

I have configured gitweb to use utf8, and that works for text blob views 
like on
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob;f=bin/git-forest;hb=HEAD
but it does not for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD

This can depend on configuration, both on gitweb configuration (you
can for example define $default_blob_plain_mimetype to 'text/plain;
charset=utf-8', and define $default_text_plain_charset to 'utf-8'),
and on your /etc/mime.types; gitweb does not add charset info if
mimetype is acquired from mime.types, which I guess is a mistake.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Always add charset info from $default_text_plain_charset (if it is
defined) to "raw" (a=blob_plain) output for 'text/plain' blobs.
Adding charset info in a special case was removed from blob_mimetype().

Signed-off-by: Jakub Narebski <redacted>
---
Please note that to have utf-8 for 'text/plain' blobs in blob_plain
view ("raw" output) you still have to set $default_text_plain_charset
to 'utf-8' (in gitweb configuration file).

 gitweb/gitweb.perl |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 57a1905..dd0f0ac 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -2481,8 +2481,7 @@ sub blob_mimetype {
 	return $default_blob_plain_mimetype unless $fd;
 
 	if (-T $fd) {
-		return 'text/plain' .
-		       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');
+		return 'text/plain';
 	} elsif (! $filename) {
 		return 'application/octet-stream';
 	} elsif ($filename =~ m/\.png$/i) {

@@ -4397,6 +4396,9 @@ sub git_blob_plain {
 		or die_error(undef, "Couldn't cat $file_name, $hash");
 
 	$type ||= blob_mimetype($fd, $file_name);
+	if ($type eq 'text/plain' && defined $default_text_plain_charset) {
+		$type .= "; charset=$default_text_plain_charset";
+	}
 
 	# save as filename, even when no $file_name is given
 	my $save_as = "$hash";

On Friday 2008-05-30 10:18, Jakub Narebski wrote:

Jan Engelhardt [off-list ref] writes:

quoted

I have configured gitweb to use utf8, and that works for text blob views 
like on
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob;f=bin/git-forest;hb=HEAD
but it does not for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD

This can depend on configuration, both on gitweb configuration (you
can for example define $default_blob_plain_mimetype to 'text/plain;
charset=utf-8', and define $default_text_plain_charset to 'utf-8'),
and on your /etc/mime.types; gitweb does not add charset info if
mimetype is acquired from mime.types, which I guess is a mistake.

Thanks for the hint. Setting 
	our $default_text_plain_charset  = "utf-8";
was all that was needed. I only had $fallback_encoding set to utf-8
for whatever reason...

On Sat, 31 May 2008, Jan Engelhardt wrote:

On Friday 2008-05-30 10:18, Jakub Narebski wrote:

quoted

Jan Engelhardt [off-list ref] writes:

quoted

I have configured gitweb to use utf8, and that works for text blob views 
like on
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob;f=bin/git-forest;hb=HEAD
but it does not for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD

This can depend on configuration, both on gitweb configuration (you
can for example define $default_blob_plain_mimetype to 'text/plain;
charset=utf-8', and define $default_text_plain_charset to 'utf-8'),
and on your /etc/mime.types; gitweb does not add charset info if
mimetype is acquired from mime.types, which I guess is a mistake.

Thanks for the hint. Setting 
	our $default_text_plain_charset  = "utf-8";
was all that was needed.

By the way, do you think that this should be the default for gitweb?

-- 
Jakub Narebski
Poland

On Sunday 2008-06-01 00:39, Jakub Narebski wrote:

On Sat, 31 May 2008, Jan Engelhardt wrote:

quoted

On Friday 2008-05-30 10:18, Jakub Narebski wrote:

quoted

Jan Engelhardt [off-list ref] writes:

quoted

I have configured gitweb to use utf8, and that works for text blob views 
like on
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob;f=bin/git-forest;hb=HEAD
but it does not for raw blob views like
http://dev.medozas.de/gitweb.cgi?p=hxtools;a=blob_plain;f=bin/git-forest;hb=HEAD

This can depend on configuration, both on gitweb configuration (you
can for example define $default_blob_plain_mimetype to 'text/plain;
charset=utf-8', and define $default_text_plain_charset to 'utf-8'),
and on your /etc/mime.types; gitweb does not add charset info if
mimetype is acquired from mime.types, which I guess is a mistake.

Thanks for the hint. Setting 
	our $default_text_plain_charset  = "utf-8";
was all that was needed.

By the way, do you think that this should be the default for gitweb?

Definitely. I also made tbz2 the default for me over tgz, because that's
just how it is.