Re: Stupid quoting...

Johannes Schindelin [off-list ref] writes:

quoted

I don't see our discourse leading anywhere: the points have been made.

I would really, really, really like to see a solution. Alas, I cannot 
think of one, other than _forcing_ the developers to use ASCII-only 
filenames.

Note that there is no convention yet in Git to state which encoding your 
filenames are supposed to use. And in fact, we already had a fine example 
in git.git why this is particularly difficult. MacOSX is too clever to be 
true, in that it gladly takes filenames in one encoding, but reads those 
filenames out in _another_ encoding. Thus, a "git add <filename>" can well 
end up in git-status saying that a file was deleted, and another file 
(actually the same, but in a different encoding) is untracked.

By the way, the pathname quoting done by "diff" does not even
attempt to tackle that.  I already explained why in the thread
so I would not repeat myself.

Having said that, the absolute minimum that needs to be quoted
are double-quote (because it is used by quoting as agreed with
GNU diff/patch maintainer), backslash (used to introduce C-like
quoting), newline and horizontal tab (makes "patch" confused, as
it would make it ambiguous where the pathname ends), so I am not
opposed to a patch that introduces a new mode, probably on by
default _unless_ we are generating --format=email, that does not
quote high byte values.  That would solve "My UTF-8 filenames
are unreadable on my terminal" problem.

Junio C Hamano [off-list ref] writes:

Johannes Schindelin [off-list ref] writes:

quoted

quoted

I don't see our discourse leading anywhere: the points have been made.

I would really, really, really like to see a solution. Alas, I
cannot think of one, other than _forcing_ the developers to use
ASCII-only filenames.

And ASCII-only files.  Just eradicate that dreaded Bit 7 from the world.

quoted

Note that there is no convention yet in Git to state which encoding
your filenames are supposed to use. And in fact, we already had a
fine example in git.git why this is particularly difficult. MacOSX
is too clever to be true, in that it gladly takes filenames in one
encoding, but reads those filenames out in _another_
encoding. Thus, a "git add <filename>" can well end up in
git-status saying that a file was deleted, and another file
(actually the same, but in a different encoding) is untracked.

Having said that, the absolute minimum that needs to be quoted are
double-quote (because it is used by quoting as agreed with GNU
diff/patch maintainer), backslash (used to introduce C-like
quoting),
newline and horizontal tab (makes "patch" confused, as it would make
it ambiguous where the pathname ends), so I am not opposed to a
patch that introduces a new mode, probably on by default _unless_ we
are generating --format=email, that does not quote high byte values.

I think it would be ok to quote non-graphic characters with octal
escape sequences.  On ASCII-based systems, those are the characters
0x00 to 0x1f.  They don't have a visual representation of their own,
anyway.  _IF_ they appear in filenames, it is certainly a case
involved with excessive cleverness and/or garbage.  I'd leave the rest
alone.

That would solve "My UTF-8 filenames are unreadable on my terminal"
problem.

But there is no point if the most primitive of mail readers does a
better job than listing the directory will.

7-Bit terminals are the wrong thing to use for manipulating
8-bit-encoded files, period.  And the escape sequences for 8-bit
terminals are quite certain to start with characters in the 0x00 to
0x1f range.

-- 
David Kastrup

David Kastrup wrote:

Junio C Hamano [off-list ref] writes:

quoted

Johannes Schindelin [off-list ref] writes:

quoted

quoted

I don't see our discourse leading anywhere: the points have been made.

I would really, really, really like to see a solution. Alas, I
cannot think of one, other than _forcing_ the developers to use
ASCII-only filenames.
Note that there is no convention yet in Git to state which encoding
your filenames are supposed to use. And in fact, we already had a
fine example in git.git why this is particularly difficult. MacOSX
is too clever to be true, in that it gladly takes filenames in one
encoding, but reads those filenames out in _another_
encoding. Thus, a "git add <filename>" can well end up in
git-status saying that a file was deleted, and another file
(actually the same, but in a different encoding) is untracked.

Having said that, the absolute minimum that needs to be quoted are
double-quote (because it is used by quoting as agreed with GNU
diff/patch maintainer), backslash (used to introduce C-like
quoting),
newline and horizontal tab (makes "patch" confused, as it would make
it ambiguous where the pathname ends), so I am not opposed to a
patch that introduces a new mode, probably on by default _unless_ we
are generating --format=email, that does not quote high byte values.

I think it would be ok to quote non-graphic characters with octal
escape sequences.  On ASCII-based systems, those are the characters
0x00 to 0x1f.  They don't have a visual representation of their own,
anyway.  _IF_ they appear in filenames, it is certainly a case
involved with excessive cleverness and/or garbage.  I'd leave the rest
alone.

By the way, ls(1) has its --quoting-style=WORD option, why shouldn't
git-diff and friends (including git-format-patch) have the same? And we
could change the default later on...

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Jakub Narebski [off-list ref] writes:

By the way, ls(1) has its --quoting-style=WORD option, why shouldn't
git-diff and friends (including git-format-patch) have the same? And
we could change the default later on...

Because interpreting a diff means interpreting both file names as well
as contents.  It does not make much sense to use different forms of
escaping (\01a and similar) here, though in the diff command line,
some additional quoting might be called for.

It is also worth noting that bash's echo -e can interpret octal
escapes only when they start with \0, and the quoted 3-character forms
of 0x00-0x1f incidentally do start in this manner.  There is still
potential for misinterpretation if an escaped character is immediately
followed by a digit.  Since octal ASCII digits are in the range 060 to
067, one can get around this problem by continuing to escape
characters until one hits a non-octal-digit.  So there is at least a
reasonable builtin way for bash scripts to translate the three-digit
octal escapes for 0x00 to 0x1f uniquely into the proper corresponding
strings.

With regard to escaping: unless used unarmored in Email (a bad idea)
or on a terminal, it might be easiest (for post-processors) to
completely refrain from escaping (in effect ignoring the
non-printability of characters) and just apply a minimal level of
quoting on the file names.

-- 
David Kastrup

On Tue, Jun 19, 2007 at 23:19:39 -0700, Junio C Hamano wrote:

Johannes Schindelin [off-list ref] writes:

quoted

quoted

I don't see our discourse leading anywhere: the points have been made.

I would really, really, really like to see a solution. Alas, I cannot 
think of one, other than _forcing_ the developers to use ASCII-only 
filenames.

Note that there is no convention yet in Git to state which encoding your 
filenames are supposed to use. And in fact, we already had a fine example 
in git.git why this is particularly difficult. MacOSX is too clever to be 
true, in that it gladly takes filenames in one encoding, but reads those 
filenames out in _another_ encoding. Thus, a "git add <filename>" can well 
end up in git-status saying that a file was deleted, and another file 
(actually the same, but in a different encoding) is untracked.

I saw bazaar folks discussing this MacOSX issue. Basically in MacOSX
filenames are *unicode* strings (just as they are in Windows, btw). Unicode,
for compatibility reasons allows expressing many characters in multiple forms
-- composed and decomposed. For example 'á' can be expressed as '\u00e1'
('\xc3\xa1' in utf-8) or as 'a\u0301' ('a\xcc\x81' in utf-8).

MaxOSX opts to, in accord with unicode standard, treat such representations
as equal and it does so by normalizing all filenames to one form. I don't
know whether it uses compatibility normalization and I believe it uses the
decomposed form (which makes the issue immediately obvious, because most
programs work in composed form).

By the way, the pathname quoting done by "diff" does not even
attempt to tackle that.  I already explained why in the thread
so I would not repeat myself.

Having said that, the absolute minimum that needs to be quoted
are double-quote (because it is used by quoting as agreed with
GNU diff/patch maintainer), backslash (used to introduce C-like
quoting), newline and horizontal tab (makes "patch" confused, as
it would make it ambiguous where the pathname ends), so I am not
opposed to a patch that introduces a new mode, probably on by
default _unless_ we are generating --format=email, that does not
quote high byte values.  That would solve "My UTF-8 filenames
are unreadable on my terminal" problem.

IMHO it should be the default even for email format. Most projects that use
non-ascii filenames probably have all members using same locale. And for
such group, it will just work. Also usually the file names, content and
commit messages will usually be in the same (though project-specific)
encoding, so if charset in content-type is set to that, people with different
locale able to represent the same characters will still see the names
correctly. For other people, the MUA will probably print some escape anyway
(it will not screw up the terminal -- it usually knows what it can safely
pass to it).

-- 
						 Jan 'Bulb' Hudec [off-list ref]

söndag 24 juni 2007 skrev Jan Hudec:

IMHO it should be the default even for email format. Most projects that use
non-ascii filenames probably have all members using same locale. And for
such group, it will just work. Also usually the file names, content and
commit messages will usually be in the same (though project-specific)
encoding, so if charset in content-type is set to that, people with 

different

locale able to represent the same characters will still see the names
correctly. For other people, the MUA will probably print some escape anyway
(it will not screw up the terminal -- it usually knows what it can safely
pass to it).

I can't talk about "most" here, only local conditions, i.e. northern Europe 
where both the legacy ISO encodings are very common with a steady increase in 
UTF-8 usage, in the Linux community. People using OSS in windows almost 
exclusively get the windows-1252 (for most practical purposes the same as 
ISO-8859-1).

Even a *very* small set of random people you will wind up with people having 
different locales.

-- robin

On Sun, Jun 24, 2007 at 13:14:45 +0200, Robin Rosenberg wrote:

söndag 24 juni 2007 skrev Jan Hudec:

quoted

IMHO it should be the default even for email format. Most projects that use
non-ascii filenames probably have all members using same locale. And for
such group, it will just work. Also usually the file names, content and
commit messages will usually be in the same (though project-specific)
encoding, so if charset in content-type is set to that, people with 

different

quoted

locale able to represent the same characters will still see the names
correctly. For other people, the MUA will probably print some escape anyway
(it will not screw up the terminal -- it usually knows what it can safely
pass to it).

I can't talk about "most" here, only local conditions, i.e. northern Europe 
where both the legacy ISO encodings are very common with a steady increase in 
UTF-8 usage, in the Linux community. People using OSS in windows almost 
exclusively get the windows-1252 (for most practical purposes the same as 
ISO-8859-1).

Even a *very* small set of random people you will wind up with people having 
different locales.

A small set of *random* people will likely have different locales. But
a project that would use non-ascii filenames would probably use some
particular language and thus be run by people that all speak that language --
which means they are not random at all and probably will use the same locale.

-- 
						 Jan 'Bulb' Hudec [off-list ref]

söndag 24 juni 2007 skrev Jan Hudec:

On Sun, Jun 24, 2007 at 13:14:45 +0200, Robin Rosenberg wrote:

quoted

I can't talk about "most" here, only local conditions, i.e. northern 

Europe 

quoted

where both the legacy ISO encodings are very common with a steady increase 

in 

quoted

UTF-8 usage, in the Linux community. People using OSS in windows almost 
exclusively get the windows-1252 (for most practical purposes the same as 
ISO-8859-1).

Even a *very* small set of random people you will wind up with people 

having 

quoted

different locales.

A small set of *random* people will likely have different locales. But
a project that would use non-ascii filenames would probably use some
particular language and thus be run by people that all speak that 

language --

which means they are not random at all and probably will use the same 

locale.

I was still in referernce to those "local conditions" at that point. It was 
not meant as a universal statement. Substitutute that for "A small bunch of 
swedish speaking people from Stockholm".

-- robin