Re: Non-ASCII paths and git-cvsserver

sf [off-list ref] writes:

I want to access a git repository via git-cvsserver. The problem is
that the repository contains paths with umlauts. These paths come out
quoted and escaped when checked out with cvs.

I think this is because the cvsserver invokes diff-tree and
ls-tree without -z and the output from these command quote
non-ascii letters as unsafe.

Martin's sqlite may probably be needed as well, but regardless
of that something like this patch is needed -- otherwise what 
populates sqlite database will be quoted to begin with so it
would not help much.

I've tested with your reproduction recipe, but otherwise not
tested this patch.

-- >8 --

diff --git a/git-cvsserver.perl b/git-cvsserver.perl
index 8817f8b..ca519b7 100755
--- a/git-cvsserver.perl
+++ b/git-cvsserver.perl

@@ -2343,67 +2343,72 @@ sub update
 
         if ( defined ( $lastpicked ) )
         {
-            my $filepipe = open(FILELIST, '-|', 'git-diff-tree', '-r', $lastpicked, $commit->{hash}) or die("Cannot call git-diff-tree : $!");
+            my $filepipe = open(FILELIST, '-|', 'git-diff-tree', '-z', '-r', $lastpicked, $commit->{hash}) or die("Cannot call git-diff-tree : $!");
+	    local ($/) = "\0";
             while ( <FILELIST> )
             {
-                unless ( /^:\d{6}\s+\d{3}(\d)\d{2}\s+[a-zA-Z0-9]{40}\s+([a-zA-Z0-9]{40})\s+(\w)\s+(.*)$/o )
+		chomp;
+                unless ( /^:\d{6}\s+\d{3}(\d)\d{2}\s+[a-zA-Z0-9]{40}\s+([a-zA-Z0-9]{40})\s+(\w)$/o )
                 {
                     die("Couldn't process git-diff-tree line : $_");
                 }
+		my ($mode, $hash, $change) = ($1, $2, $3);
+		my $name = <FILELIST>;
+		chomp($name);
 
-                # $log->debug("File mode=$1, hash=$2, change=$3, name=$4");
+                # $log->debug("File mode=$mode, hash=$hash, change=$change, name=$name");
 
                 my $git_perms = "";
-                $git_perms .= "r" if ( $1 & 4 );
-                $git_perms .= "w" if ( $1 & 2 );
-                $git_perms .= "x" if ( $1 & 1 );
+                $git_perms .= "r" if ( $mode & 4 );
+                $git_perms .= "w" if ( $mode & 2 );
+                $git_perms .= "x" if ( $mode & 1 );
                 $git_perms = "rw" if ( $git_perms eq "" );
 
-                if ( $3 eq "D" )
+                if ( $change eq "D" )
                 {
-                    #$log->debug("DELETE   $4");
-                    $head->{$4} = {
-                        name => $4,
-                        revision => $head->{$4}{revision} + 1,
+                    #$log->debug("DELETE   $name");
+                    $head->{$name} = {
+                        name => $name,
+                        revision => $head->{$name}{revision} + 1,
                         filehash => "deleted",
                         commithash => $commit->{hash},
                         modified => $commit->{date},
                         author => $commit->{author},
                         mode => $git_perms,
                     };
-                    $self->insert_rev($4, $head->{$4}{revision}, $2, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
+                    $self->insert_rev($name, $head->{$name}{revision}, $hash, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
                 }
-                elsif ( $3 eq "M" )
+                elsif ( $change eq "M" )
                 {
-                    #$log->debug("MODIFIED $4");
-                    $head->{$4} = {
-                        name => $4,
-                        revision => $head->{$4}{revision} + 1,
-                        filehash => $2,
+                    #$log->debug("MODIFIED $name");
+                    $head->{$name} = {
+                        name => $name,
+                        revision => $head->{$name}{revision} + 1,
+                        filehash => $hash,
                         commithash => $commit->{hash},
                         modified => $commit->{date},
                         author => $commit->{author},
                         mode => $git_perms,
                     };
-                    $self->insert_rev($4, $head->{$4}{revision}, $2, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
+                    $self->insert_rev($name, $head->{$name}{revision}, $hash, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
                 }
-                elsif ( $3 eq "A" )
+                elsif ( $change eq "A" )
                 {
-                    #$log->debug("ADDED    $4");
-                    $head->{$4} = {
-                        name => $4,
+                    #$log->debug("ADDED    $name");
+                    $head->{$name} = {
+                        name => $name,
                         revision => 1,
-                        filehash => $2,
+                        filehash => $hash,
                         commithash => $commit->{hash},
                         modified => $commit->{date},
                         author => $commit->{author},
                         mode => $git_perms,
                     };
-                    $self->insert_rev($4, $head->{$4}{revision}, $2, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
+                    $self->insert_rev($name, $head->{$name}{revision}, $hash, $commit->{hash}, $commit->{date}, $commit->{author}, $git_perms);
                 }
                 else
                 {
-                    $log->warn("UNKNOWN FILE CHANGE mode=$1, hash=$2, change=$3, name=$4");
+                    $log->warn("UNKNOWN FILE CHANGE mode=$mode, hash=$hash, change=$change, name=$name");
                     die;
                 }
             }

@@ -2412,10 +2417,12 @@ sub update
             # this is used to detect files removed from the repo
             my $seen_files = {};
 
-            my $filepipe = open(FILELIST, '-|', 'git-ls-tree', '-r', $commit->{hash}) or die("Cannot call git-ls-tree : $!");
+            my $filepipe = open(FILELIST, '-|', 'git-ls-tree', '-z', '-r', $commit->{hash}) or die("Cannot call git-ls-tree : $!");
+	    local $/ = "\0";
             while ( <FILELIST> )
             {
-                unless ( /^(\d+)\s+(\w+)\s+([a-zA-Z0-9]+)\s+(.*)$/o )
+		chomp;
+                unless ( /^(\d+)\s+(\w+)\s+([a-zA-Z0-9]+)\t(.*)$/o )
                 {
                     die("Couldn't process git-ls-tree line : $_");
                 }

måndag 13 november 2006 19:57 skrev Jakub Narebski:

That was my idea, to have i18n.filesystemEncoding configuration variable
to convert between filesystem encoding (which is usually something you
don't have control over, and which depends from place to place, but not
from repository to repository) and UTF-8 encoding git would store
filenames.

Yes, I know.

Martin Langhoff wrote:

On 11/13/06, sf [off-list ref] wrote:

quoted

Martin, are you sure your patch is needed? (see below)

Not 100% sure. I was just making sure we crossed all the Ts and dotted
the Is. I gather you have tried my patch and it didn't make any
difference. What SQLite and Perl versions are you using?

Your patch did make a difference but the outcome is not good:

+ WORK=/tmp/gittest
+ FILE=$'\303\244'
+ mkdir /tmp/gittest
+ mkdir /tmp/gittest/git
+ cd /tmp/gittest/git
+ git init-db
defaulting to local storage area
+ git repo-config gitcvs.enabled 1
+ git repo-config gitcvs.logfile /tmp/gittest/git/.git/cvslog.txt
+ touch $'\303\244'
+ git add $'\303\244'
+ git commit -a -mx
Committing initial tree 23d6145738bba135994775c19d6e8ae707d399ee
+ cd /tmp/gittest
+ CVS_SERVER=git-cvsserver
+ export CVS_SERVER
+ cvs -d :fork:/tmp/gittest/git/.git co master
cvs checkout: Updating master
U master/ä
+ ls master
ä  CVS


The pathname has been UTF-8 encoded _twice_!

Perl's version is 5.8.8. How do I get the version of SQLite? Do you mean 
DBD-SQLite-1.11?

Regards

Stephan

måndag 13 november 2006 15:20 skrev Jakub Narebski:

sf wrote:

quoted

Thanks, Junio. Paths with umlauts are returned correctly now both in
UTF-8 and ISO-8859-1. I guess git-cvsserver is now as encoding agnostic
as git core.

By the way, now that git has per user config file, ~/.gitconfig, perhaps
it is time to add i18n.filesystemEncoding configuration variable, to
automatically convert between filesystem encoding (somthing you usually
don't have any control over) and UTF-8 encoding of paths in tree objects.

I'd prefer git to store filenames and comments in UTF-8 and convert on 
input/output when and if it is necessary rather than forcing everybody to 
take the hit. Most systems, but far from all, already use UTF-8 so it's a 
noop for them. The only reason I want conversion is for the years to come 
where we still live in two worlds of non-utf-8 and utf-8 and then forget 
about everything non-utf-8, rather than carry around the baggage forever.

sf wrote:

Thanks, Junio. Paths with umlauts are returned correctly now both in 
UTF-8 and ISO-8859-1. I guess git-cvsserver is now as encoding agnostic 
as git core.

By the way, now that git has per user config file, ~/.gitconfig, perhaps
it is time to add i18n.filesystemEncoding configuration variable, to
automatically convert between filesystem encoding (somthing you usually
don't have any control over) and UTF-8 encoding of paths in tree objects.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Junio C Hamano wrote:

sf [off-list ref] writes:

quoted

I want to access a git repository via git-cvsserver. The problem is
that the repository contains paths with umlauts. These paths come out
quoted and escaped when checked out with cvs.

I think this is because the cvsserver invokes diff-tree and
ls-tree without -z and the output from these command quote
non-ascii letters as unsafe.

I knew I had seen that kind of quoting before but right then I thought 
it was related to Perl or SQLite.

Martin's sqlite may probably be needed as well, but regardless
of that something like this patch is needed -- otherwise what 
populates sqlite database will be quoted to begin with so it
would not help much.

Martin, are you sure your patch is needed? (see below)

I've tested with your reproduction recipe, but otherwise not
tested this patch.

Thanks, Junio. Paths with umlauts are returned correctly now both in 
UTF-8 and ISO-8859-1. I guess git-cvsserver is now as encoding agnostic 
as git core.

Regards

Dnia poniedziałek 13. listopada 2006 19:30, Robin Rosenberg napisał:

måndag 13 november 2006 15:20 skrev Jakub Narebski:

quoted

sf wrote:

quoted

Thanks, Junio. Paths with umlauts are returned correctly now both in
UTF-8 and ISO-8859-1. I guess git-cvsserver is now as encoding agnostic
as git core.

By the way, now that git has per user config file, ~/.gitconfig, perhaps
it is time to add i18n.filesystemEncoding configuration variable, to
automatically convert between filesystem encoding (somthing you usually
don't have any control over) and UTF-8 encoding of paths in tree objects.

I'd prefer git to store filenames and comments in UTF-8 and convert on 
input/output when and if it is necessary rather than forcing everybody to 
take the hit. Most systems, but far from all, already use UTF-8 so it's a 
noop for them. The only reason I want conversion is for the years to come 
where we still live in two worlds of non-utf-8 and utf-8 and then forget 
about everything non-utf-8, rather than carry around the baggage forever.

That was my idea, to have i18n.filesystemEncoding configuration variable
to convert between filesystem encoding (which is usually something you don't
have control over, and which depends from place to place, but not from
repository to repository) and UTF-8 encoding git would store filenames.

-- 
Jakub Narebski

On 11/13/06, sf [off-list ref] wrote:

Martin, are you sure your patch is needed? (see below)

Not 100% sure. I was just making sure we crossed all the Ts and dotted
the Is. I gather you have tried my patch and it didn't make any
difference. What SQLite and Perl versions are you using?

cheers,