Re: git on MacOSX and files with decomposed utf-8 file names

On Sat, Jan 19, 2008 at 09:55:45AM -0500, Kevin Ballard wrote:

quoted

[And please stop calling by normalization what is not. Mac does NOT
normalize Unicode strings, it uses some sub-standard conversion,
which neither produce a normalized string nor is guaranteed to be
stable across versions of Unicode.]

From what the HFS+ technote says, it produces a variant of Normal  
Form D. 

There is no such thing in the standard as a variant of NFD. Moreover,
even if this conversion were described in the standard, it would never
called as normalization, because normalization means conversion that
makes all equivalent strings having identical binary representations.
HFS+ conversion does not met this criterion, so it not normalization.

This variant, while not guaranteed to be stable across  
versions of HFS+, but in practice it is stable.

What would you prefer I call it?

Apple calls it as decomposition, which is correct even if it is not full
decomposition as stated in the technote.

quoted

quoted

This doesn't mean treating filenames as unicode strings is wrong, it
just means that the world would be much better if every filesystem  
had
the same behaviour here. It's kinda like the endian issue, except
there's no simple solution here.

Actually, there is, if you care to do something. You can write a  
wrapper
around readdir(3) that will recodes filenames in Unicode Normal  
Forms C.
This does not require much knowledge of Git -- what it requires the
desire to do something to solve the problem. Of course, this step  
alone
is not a complete solution (it does not solve case-insensitive issue),
but the first step in the right direction...

I'm not sure how that would solve anything. Sure, it would provide a  
stable, known encoding for git to compare filenames against, but that  
would only work if the filename is known to be Unicode, and as it has  
been pointed out on other filesystems the filename can be whatever  
encoding the user chooses (which, IMHO, is a flaw).

I believe that Git internally should use only UTF-8 for encoding file
names, commit messages, etc. The problem with some other filesystems
should be addressed separately (by those who work on those systems or
at least have access to them). Regardless interoperability with other
systems, this change alone should solve the issue that was described
in the first message of this thread.

Dmitry