Re: git on MacOSX and files with decomposed utf-8 file names

On Mon, 21 Jan 2008, Kevin Ballard wrote:

quoted

It's what shows up when you sha1sum, but it's also as simple as what shows
up when you do an "ls -l" and look at a file size.

It does? Why on earth should it do that? Filename doesn't contribute to the
listed filesize on OS X.

Umm. What's this inability to see that data is data is data?

Why do you think Unicode has anything in particular to do with filenames?

Those same unicode strings are often part of the file data itself, and 
then that encoding damn well is visible in "ls -l".

Doing

	echo ä > file
	ls -l file

sure shows that "underlying octet" thing that you wanted to avoid so much. 
My point was that those underlying octets are always there, and they do 
matter. The fact that the differences may not be visible when you compare 
the normalized forms doesn't make it any less true.

You can choose to put blinders on and try to claim that normalization is 
invisible, but it's only invisible TO THOSE THINGS THAT DON'T WANT TO SEE 
IT.

But that doesn't change the fact that a lot of things *do* see it. There 
are very few things that are "Unicode specific", and a *lot* of tools that 
are just "general data tools".

And git tries to be a general data tool, not a Unicode-specific one.

I'm not sure what you mean. The byte sequence is different from Latin1 to
UTF-8 even if you use NFC, so I don't think, in this case, it makes any
difference whether you use NFC or NFD.

Yes, the codepoints are the same in Latin1 and UTF-8 if you use NFC, but 
that's hardly relevant. Please correct me if I'm wrong, but I believe 
Latin1->UTF-8->Latin1 conversion will always produce the same Latin1 
text whether you use NFC or NFD.

The problem is that the UTF-8 form is different, so if you save things in 
UTF-8 (which we hopefully agree is a sane thing to do), then you should 
try to use a representation that people agree on.

And NFC is the more common normalization form by far, so by normalizing to 
something else, you actually de-normalize as far as those other people are 
concerned.

So if you have to normalize, at least use the normal form!

The only reason it's particularly inconvenient is because it's different from
what most other systems picked. And if you want to blame someone for that,
blame Unicode for having so many different normalization forms.

I blame them for encouraging normalization at all.

It's stupid.

You don't need it.

The people who care about "are these strings equivalent" shouldn't do a 
"memcmp()" on them in the first place. And if you don't do a memcmp() on 
things, then you don't need to normalize. 

So you have two cases:
 (a) the cases that care about *identity*. They don't want normalization
 (b) the cases that care about *equivalence*. And they shouldn't do 
      octet-by-octet comparison.

See? Either you want to see equivalence, or you don't. And in neither case 
is normalization the right thing to do (except as *possibly* an internal 
part of the comparison, but there are actually better ways to check for 
equivalence than the brute-force "normalize both and compare results 
bitwise").

			Linus