Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.

Hi!

Makes sense too. I think the whole API would have to be changed to use 
TCHAR*.

I'd rather just say wchar_t explicitely. I'm not particularly fond of macros 
that change under your feet just because you fail to define a symbol 
somewhere...

Then you need to do the right conversion at the right places, this will be 
quite tricky, painful work, but there is probably no way around that.

In the other project I worked on we ended up wrapping all file-related calls 
in our own porting interface, and then let each platform we compiled for 
implement their own methods for handling Unicode paths. For Windows it's 
trivial since all APIs are Unicode. For Unix-like OSes it's tricky as you 
have to take the locale settings into account, but fortunately the world is 
slowly moving towards UTF-8 locales, which eases the pain a bit.

Note that not only conversions will be needed but you'll also need to 
adjust all routines handling filenames to use the proper Unicode version. 
(strchr -> _tstrchr, open -> _topen, strcpy -> _tstrcpy, strlen -> 
_tcslen, ...).

Not necessarily. If the code can be set up to use UTF-8 char* internally, 
not everything needs to be rewritten (I've done that too, only took a 
couple of years to move the codebase over to all-Unicode).

-- 
\\// Peter - http://www.softwolves.pp.se/

Peter Krefting wrote:

In the other project I worked on we ended up wrapping all file-related calls 
in our own porting interface, and then let each platform we compiled for 
implement their own methods for handling Unicode paths. For Windows it's 
trivial since all APIs are Unicode. For Unix-like OSes it's tricky as you 
have to take the locale settings into account, but fortunately the world is 
slowly moving towards UTF-8 locales, which eases the pain a bit.

Have you thought about all the consequences this would have for the
*nix people here? [*]

Even if you pretend that Git did always enforce UTF-8 paths in its
trees, so that there's no backward compatibility to be cared for,
you're still in a world of hurt when trying to check out such paths
under a locale (or whatever setting might control this new encoding
logic) that does not support the whole range of UTF-8.

Like, say, the C locale.

Next you get to see to it that the users can spell all filenames even
if their locale doesn't let them, since they'll want to do things like
'git show $rev:$file' with them.

With backwards compatibility it's even worse as you're suddenly
imposing extra restrictions on what a valid filename in the repository
must look like.


[*] I'm _extremely_ tempted to write "people using non-broken OSes",
but let's pretend to be neutral for a second.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Thomas Rast:

Have you thought about all the consequences this would have for the *nix 
people here? [*]

Yeah. It will fix problems trying to check out a Git repository created by 
me in a iso8859-1 locale on a machine using a utf-8 locale, where both ends 
would like to have a file named "Ü".

Or, hopefully, a careful adoption of this on Windows won't affect Unixes and 
other systems with pre-Unicode APIs at all, since the Windows code would be 
in the "compat" directory.

you're still in a world of hurt when trying to check out such paths under 
a locale (or whatever setting might control this new encoding logic) that 
does not support the whole range of UTF-8.

Yeah. That would be a case similar to the casing problem on Windows.

With backwards compatibility it's even worse as you're suddenly imposing 
extra restrictions on what a valid filename in the repository must look 
like.

Indeed. It is unfortunate that this wasn't properly specified to start with. 
It's mostly a minor issue since *most* people will not use non-ASCII file 
names. At least for most of the kind of projects that Git have attracted so 
far, so the problem is not that big. The problem is if Git is to attract 
"the masses". Especially on Windows, where file names using non-ASCII are 
common, this needs to be addressed eventually.

[*] I'm _extremely_ tempted to write "people using non-broken OSes", but 
let's pretend to be neutral for a second.

In most cases, I would most definitely agree with you on calling it that, 
but when it comes to Unicode support, Windows is one of the least broken 
OSes (with Symbian being my favourite).

-- 
\\// Peter - http://www.softwolves.pp.se/

* Peter Krefting [off-list ref] [09-03-02 21:41]:

Indeed. It is unfortunate that this wasn't properly specified to start with. 
It's mostly a minor issue since *most* people will not use non-ASCII file 
names. At least for most of the kind of projects that Git have attracted so 
far, so the problem is not that big. The problem is if Git is to attract "the 
masses". Especially on Windows, where file names using non-ASCII are common, 
this needs to be addressed eventually.

Using no encoding for filenames was the obvious (and I would argue)
correct choice. Unix filenames are specified to be a sequence of bytes,
excluding '/' and '\0'. A lot of these sequences are not valid UTF-8.
Further, the encoding needed for filenames depends on the encoding used
in the source code for referencing these files. Again, for the unix file
handling functions, this means no encoding.

Changing the filename (on checkout), so that the user sees an Ü
regardless of his or her locale (instead of an \0xDC, which only
resolves to an Ü on latin-1) would be an absolutely broken concept here.

quoted

[*] I'm _extremely_ tempted to write "people using non-broken OSes", but let's 
pretend to be neutral for a second.

In most cases, I would most definitely agree with you on calling it that, but 
when it comes to Unicode support, Windows is one of the least broken OSes (with 
Symbian being my favourite).

IMHO having encoding specific open functions is begging for problems.

 - Lars.

On Mon, Mar 02, 2009 at 09:41:57PM +0100, Peter Krefting wrote:

In most cases, I would most definitely agree with you on calling it that,_
but when it comes to Unicode support, Windows is one of the least broken__
OSes (with Symbian being my favourite).

The C Standard requires that the type wchar_t is capable of representing
any character in the current locale. If Windows uses UTF-16 as internal
encoding (so, it can work with symbols outside of the BMP), it means you
cannot have 16-bit wchar_t and be compliant with the C standard...

Dmitry

Dmitry Potapov:

The C Standard requires that the type wchar_t is capable of representing 
any character in the current locale. If Windows uses UTF-16 as internal 
encoding (so, it can work with symbols outside of the BMP), it means you 
cannot have 16-bit wchar_t and be compliant with the C standard...

No, that's not quite correct. wchar_t is defined to be "an integer type whose 
range of values can represent distinct codes for all members of 
the largest extended character set specified among the supported locales". 
Since Windows defines all local character sets as Unicode-based, having 
wchar_t defined as Unicode means that it can represent everything.

-- 
\\// Peter - http://www.softwolves.pp.se/

Lars Noschinski:

Using no encoding for filenames was the obvious (and I would argue) 
correct choice. Unix filenames are specified to be a sequence of bytes, 
excluding '/' and '\0'.

I know the Unix way of thinking lends itself to such a design. This is one 
of the few cases where I personally think Unix has got it wrong, and Windows 
(NT) has got it right. But then again, Unix' design pre-dates the locale 
issue by quite some time, so it is not difficult to see where it comes from.

Changing the filename (on checkout), so that the user sees an Ü regardless 
of his or her locale (instead of an \0xDC, which only resolves to an Ü on 
latin-1) would be an absolutely broken concept here.

Why would it? It is my view as a user on my files that define how file names 
are looked upon. If I have three machines, one Linux box using a iso8859-1 
locale, an OS X box (where, I would believe, file APIs use UTF-8, someone 
please correct me if I'm wrong), and a Windows box (which uses UTF-16 on the 
file system layer, but does provide compatibility functions that use char 
pointers), and create a file on each of these called "Ü.txt" (which would be 
the sequence "DC 2E 74 78 74" on the Linux box, "C3 9C 2E 74 78 74" (or 
probably something else since I believe OS X decomposes the string) on the 
OS X box and "00DC 002E 0074 0078 0074" on the Windows box, I see these 
three file names as equal.

If I would create a Git repo on each of the three machines and put the file 
name in it, and then clone that on one of the other machines. *I* would 
assume that the file names were converted to fit the host operating system.

IMHO having encoding specific open functions is begging for problems.

Indeed. That's why I like Windows' wchar_t APIs, and dislike Unix' and 
Linux' char APIs that, in some ways, depend on the user locale.

-- 
\\// Peter - http://www.softwolves.pp.se/

* Peter Krefting [off-list ref] [09-03-03 12:54]:

Lars Noschinski:

quoted

Changing the filename (on checkout), so that the user sees an Ü regardless of 
his or her locale (instead of an \0xDC, which only resolves to an Ü on 
latin-1) would be an absolutely broken concept here.

Why would it? It is my view as a user on my files that define how file names 
are looked upon. If I have three machines, one Linux box using a iso8859-1 
locale, an OS X box (where, I would believe, file APIs use UTF-8, someone 
please correct me if I'm wrong), and a Windows box (which uses UTF-16 on the 
file system layer, but does provide compatibility functions that use char 
pointers), and create a file on each of these called "Ü.txt" (which would be 
the sequence "DC 2E 74 78 74" on the Linux box, "C3 9C 2E 74 78 74" (or 
probably something else since I believe OS X decomposes the string) on the OS X 
box and "00DC 002E 0074 0078 0074" on the Windows box, I see these three file 
names as equal.

Because a function in the source code refers to (e.g.) "DC 2E 74 78 74",
not "C3 9C 2E 74 78 74" nor "00DC 0024 0074 0078 0074". And it does so
regardless of the locale.

The file name may look funny depending on your locale, but if you rename
the file to fit your local enconding, it would not work.

On Tue, Mar 3, 2009 at 2:48 PM, Peter Krefting [off-list ref] wrote:

Dmitry Potapov:

quoted

The C Standard requires that the type wchar_t is capable of representing
any character in the current locale. If Windows uses UTF-16 as internal
encoding (so, it can work with symbols outside of the BMP), it means you
cannot have 16-bit wchar_t and be compliant with the C standard...

No, that's not quite correct. wchar_t is defined to be "an integer type
whose range of values can represent distinct codes for all members of the
largest extended character set specified among the supported locales". Since
Windows defines all local character sets as Unicode-based, having wchar_t
defined as Unicode means that it can represent everything.

No, it does not, if you have wchar_t that is only 16-bit wide, because
characters
outside of the BMP have integer values in Unicode greater than 65535...

Dmitry

Lars Noschinski [off-list ref] writes:

* Peter Krefting [off-list ref] [09-03-03 12:54]:

quoted

Lars Noschinski:

quoted

Changing the filename (on checkout), so that the user sees an Ü regardless of 
his or her locale (instead of an \0xDC, which only resolves to an Ü on 
latin-1) would be an absolutely broken concept here.

Why would it? It is my view as a user on my files that define how file names 
are looked upon. If I have three machines, one Linux box using a iso8859-1 
locale, an OS X box (where, I would believe, file APIs use UTF-8, someone 
please correct me if I'm wrong), and a Windows box (which uses UTF-16 on the 
file system layer, but does provide compatibility functions that use char 
pointers), and create a file on each of these called "Ü.txt" (which would be 
the sequence "DC 2E 74 78 74" on the Linux box, "C3 9C 2E 74 78 74" (or 
probably something else since I believe OS X decomposes the string) on the OS X 
box and "00DC 002E 0074 0078 0074" on the Windows box, I see these three file 
names as equal.

Because a function in the source code refers to (e.g.) "DC 2E 74 78 74",
not "C3 9C 2E 74 78 74" nor "00DC 0024 0074 0078 0074". And it does so
regardless of the locale.

The only actual language I know where I've seen people use non-ascii names for
referenced files, i.e. classes, is Java and there you specify the encoding to
the compiler. Class names are not byte sequences there. XML files are another
case where references files are defined in unicode. I assume this applies to
C# and other modern languages too.

The file name may look funny depending on your locale, but if you rename
the file to fit your local enconding, it would not work.

In the Java case, you /have/ to "rename" or the build will break. Build systems like Ant
or Maven require you to "rename" too regardless of what you build. A C Git clone
will produce unbuildable code, but JGit will produce a working one for unicode
aware systems and documentation, the case where unicode filenames are more common
than in source, will look good.

-- robin

PS. I readded the people you forgot to Cc

Dmitry Potapov:

No, it does not, if you have wchar_t that is only 16-bit wide, because 
characters outside of the BMP have integer values in Unicode greater than 
65535...

UTF-16 allows you to reference all of Unicode (i.e up to U+10FFFF) using 
surrogate pairs. That means that not all characters can be represented as a 
single wchar_t, that is true. The problem with changing wchar_t is that it 
was defined to use 16-bit values at a time where Unicode was defined to use 
16-bit code points (but they soon figured out that was not enough).

Anyway, this is getting off-topic. Please feel free reply in private.

-- 
\\// Peter - http://www.softwolves.pp.se/

On Wed, Mar 04, 2009 at 11:51:15AM +0100, Peter Krefting wrote:

The problem with changing wchar_t is that_
it was defined to use 16-bit values at a time where Unicode was defined_
to use 16-bit code points (but they soon figured out that was not_
enough).

I do realize that is a problem, and unfortunately there is no easy and
quick fix to it. But you brought Windows as an example of good Unicode
support... Well, to my mind, it is not, at least, not for C programs.
You have two serious problems here:
1. wchar_t is too small to hold all Unicode characters as it is required
   by C standard.
2. UTF-8 support is broken in C runtime library.

In fact, if UTF-8 were supported by C runtime, we would not have this thread
in the first place... Now, it is possible to wrap all C functions used by Git to
make them work with UTF-8, but it is a lot of work...

Dmitry