[RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.

When opening a file through open() or fopen(), the path passed is
UTF-8 encoded. To handle this on Windows, we need to convert the
path string to UTF-16 and use the Unicode-based interface.
---
Windows does support file names using arbitrary Unicode characters, you just 
need to use its wchar_t interfaces instead of the char ones (the char ones 
just gets converted into wchar_t on the API level anyway, for the same 
reasons). This is the beginnings of support for UTF-8 file names on Git on 
Windows.

Since there is no real file system abstraction beyond using stdio (AFAIK), I 
need to hack it by replacing fopen (and open). Probably opendir/readdir as 
well (might be trickier), and possibly even hack around main() to parse the 
wchar_t command-line instead of the char copy.

This will lose all chances of Windows 9x compatibility, but I don't know if 
there are any attempts of supporting it anyway?

Please note that MultiByteToWideChar() will reject any invalid UTF-8 
strings, perhaps it should just fall back to a regular open()/fopen() in 
that case?

No Signed-Off line since this is unfinished, just presenting rough sketches 
of an idea.

  compat/mingw.c |   60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
  compat/mingw.h |    3 ++
  2 files changed, 62 insertions(+), 1 deletions(-)

diff --git a/compat/mingw.c b/compat/mingw.c
index e25cb4f..8b19b80 100644
--- a/compat/mingw.c
+++ b/compat/mingw.c

@@ -9,13 +9,30 @@ int mingw_open (const char *filename, int oflags, ...)
  {
  	va_list args;
  	unsigned mode;
+	wchar_t *unicode_filename;
+	int unicode_filename_len;
  	va_start(args, oflags);
  	mode = va_arg(args, int);
  	va_end(args);

  	if (!strcmp(filename, "/dev/null"))
  		filename = "nul";
-	int fd = open(filename, oflags, mode);
+
+	unicode_filename_len = MultiByteToWideChar(CP_UTF8, 0, filename, -1, NULL, 0);
+	if (0 == unicode_filename_len) {
+		errno = EINVAL;
+		return -1;
+	};
+
+	unicode_filename = xmalloc(unicode_filename_len * sizeof (wchar_t));
+	if (NULL == unicode_filename) {
+		errno = ENOMEM;
+		return -1;
+	}
+	MultiByteToWideChar(CP_UTF8, 0, filename, -1, unicode_filename, unicode_filename_len);
+	int fd = _wopen(unicode_filename, oflags, mode);
+	free(unicode_filename);
+
  	if (fd < 0 && (oflags & O_CREAT) && errno == EACCES) {
  		DWORD attrs = GetFileAttributes(filename);
  		if (attrs != INVALID_FILE_ATTRIBUTES && (attrs & FILE_ATTRIBUTE_DIRECTORY))

@@ -24,6 +41,47 @@ int mingw_open (const char *filename, int oflags, ...)
  	return fd;
  }

+FILE *mingw_fopen (const char *filename, const char *mode)
+{
+	wchar_t *unicode_filename, *unicode_mode;
+	int unicode_filename_len, unicode_mode_len;
+	FILE *fh;
+
+	unicode_filename_len = MultiByteToWideChar(CP_UTF8, 0, filename, -1, NULL, 0);
+	if (0 == unicode_filename_len) {
+		errno = EINVAL;
+		return NULL;
+	};
+
+	unicode_filename = xmalloc(unicode_filename_len * sizeof (wchar_t));
+	if (NULL == unicode_filename) {
+		errno = ENOMEM;
+		return NULL;
+	}
+	MultiByteToWideChar(CP_UTF8, 0, filename, -1, unicode_filename, unicode_filename_len);
+
+	unicode_mode_len = MultiByteToWideChar(CP_UTF8, 0, mode, -1, NULL, 0);
+	if (0 == unicode_mode_len) {
+		free(unicode_filename);
+		errno = EINVAL;
+		return NULL;
+	};
+
+	unicode_mode = xmalloc(unicode_mode_len * sizeof (wchar_t));
+	if (NULL == unicode_mode) {
+		free(unicode_mode);
+		errno = ENOMEM;
+		return NULL;
+	}
+	MultiByteToWideChar(CP_UTF8, 0, mode, -1, unicode_mode, unicode_mode_len);
+
+	fh = _wfopen(unicode_filename, unicode_mode);
+	free(unicode_filename);
+	free(unicode_mode);
+
+	return fh;
+}
+
  static inline time_t filetime_to_time_t(const FILETIME *ft)
  {
  	long long winTime = ((long long)ft->dwHighDateTime << 32) + ft->dwLowDateTime;

diff --git a/compat/mingw.h b/compat/mingw.h
index 4f275cb..235df0a 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h

@@ -142,6 +142,9 @@ int sigaction(int sig, struct sigaction *in, struct sigaction *out);
  int mingw_open (const char *filename, int oflags, ...);
  #define open mingw_open

+FILE *mingw_fopen (const char *filename, const char *mode);
+#define fopen mingw_fopen
+
  char *mingw_getcwd(char *pointer, int len);
  #define getcwd mingw_getcwd

-- 
1.6.0.2.1172.ga5ed0

Peter Krefting schrieb:

When opening a file through open() or fopen(), the path passed is
UTF-8 encoded.

I don't think that this assumption is valid. Whenever the Windows API has
to convert between Unicode strings and char* strings, it uses the current
"ANSI code page". As far as I know, the UTF-8 codepage (65001) cannot be
used as the "current ANSI code page". Users will always have some code
page set that is not UTF-8.

For example, if the user specifies a file name on the command line, than
it will not enter git in UTF-8, but in the current "ANSI" or "OEM code
page" encoding. If git prints a file name under the assumption that it is
UTF-8 encoded, then it will be displayed incorrectly because the system
uses a different encoding.

Since there is no real file system abstraction beyond using stdio
(AFAIK), I need to hack it by replacing fopen (and open). Probably
opendir/readdir as well (might be trickier), and possibly even hack
around main() to parse the wchar_t command-line instead of the char copy.

I think you are grossly underestimating the venture that you want to
undertake here.

Please come up with a plan how you are going to deal with the various
issues. File names enter and leave the system through different channels:

- the command line and terminal window
- object database (tree objects)
- opendir/readdir; opening files or directories for reading or writing

And there is probably some more... How do you treat encodings in these
channels? What if the file names are not valid UTF-8? Etc.

The biggest obstacle will be that git does not have a notion of "file name
encoding" - it simply treats a file name as a stream of bytes. There is no
place to write an encoding. If the byte streams are regarded as having an
encoding, then you can have ambiguities, mixed encodings, or invalid
characters. You would have to deal with this in some way.

This will lose all chances of Windows 9x compatibility, but I don't know
if there are any attempts of supporting it anyway?

Windows 9x is already out of the loop. We use GetFileInformationByHandle()
that is only available since Windows 2000.

-- Hannes

Johannes Sixt:

I don't think that this assumption is valid.

Depends on where you are coming from. For the files stored in the Git 
repositories, I believe all file names are supposed to be UTF-8 encoded 
(just like commit messages and user names are). That's the assumption I 
started working from.

Users will always have some code page set that is not UTF-8.

Indeed. And as long as the char-pointer interfaces in stdio and elsewhere 
work on that assumption, we have a problem.

For example, if the user specifies a file name on the command line, than
it will not enter git in UTF-8, but in the current "ANSI" or "OEM code
page" encoding.

That problem is already solved as we do have a wchar_t command line 
available. If you pass a file name that is not representable in the current 
"ANSI" codepage on the command line, it will come out as garbage in the 
char* version, but will be correct in the wchar_t* version. Thus we need to 
convert that to utf-8 and use that instead.

If git prints a file name under the assumption that it is UTF-8 encoded, 
then it will be displayed incorrectly because the system uses a different 
encoding.

Here setting the local codepage to UTF-8 *might* work, although I haven't 
tested that. Or always use the wchar_t versions of printf and friends.

I think you are grossly underestimating the venture that you want to 
undertake here.

I've done this before with other software, so, yes, I know it is quite a big 
undertaking. That is also why I started out with a minimal RFC patch to see 
if there was any interest in working with this.

Please come up with a plan how you are going to deal with the various
issues. File names enter and leave the system through different channels:

- the command line and terminal window

GetCommandLineW() as decribed above.

- object database (tree objects)

Those file names are supposedly always UTF-8.

- opendir/readdir; opening files or directories for reading or writing

Wrap file open and directory read to use the wchar_t versions, converting 
that to UTF-8 strings at the API level.

And there is probably some more... How do you treat encodings in these 
channels? What if the file names are not valid UTF-8? Etc.

Ill-formed UTF-8 should just be rejected. Invalid UTF-8 is worse. I'm not 
sure what the Linux version does, when running in a UTF-8 locale. Does it 
allow ill-formed or illegal UTF-8 sequences?

NTFS allows almost any sequence of wchar_t's, it doesn't even have to be 
valid UTF-16.

The biggest obstacle will be that git does not have a notion of "file name 
encoding" - it simply treats a file name as a stream of bytes.

Yeah, that is one of the major bugs in its design, IMHO. But almost everyone 
seems to assume that file names are UTF-8 strings anyway, so in the absence 
of any other information, it's a good assumption as any to make.

If the byte streams are regarded as having an encoding, then you can have 
ambiguities, mixed encodings, or invalid characters. You would have to 
deal with this in some way.

Considering we already see problems with file names that cannot properly be 
represented on some file systems (case-only differences in the Linux kernel 
when checked out on Windows; Mac OS' built-in Unicode normalization of file 
names, etc.)

Windows 9x is already out of the loop.

Good.

-- 
\\// Peter - http://www.softwolves.pp.se/

Hi,

On Mon, 2 Mar 2009, Peter Krefting wrote:

Johannes Sixt:

quoted

I don't think that this assumption is valid.

Depends on where you are coming from. For the files stored in the Git 
repositories, I believe all file names are supposed to be UTF-8 encoded 
(just like commit messages and user names are). That's the assumption I 
started working from.

No.  As far as Git is concerned, the file names are just as much blobs as 
the file contents.

The fact that Windows messes with this notion just as it messes with the 
file contents (think the endless story whose name is CR/LF) shows only how 
"well" designed the concepts in Windows are.

And as it stands, we have at least two issues on the msysGit issue tracker 
that complain that Git does not work with localized file names properly.

So no, file names are not UTF-8 at all, especially not on Windows.

Do not get me wrong, I really welcome you taking care of the issue, but I 
do not think that forcing UTF-8 is a solution.

Thanks & sorry,
Dscho

Johannes Schindelin:

No.  As far as Git is concerned, the file names are just as much blobs as 
the file contents.

I've struggled with the same problems on Linux before, since its file 
systems doesn't have the concept of characters, either. I guess it's just 
design principles, but as far as I am concerned, having file names be 
constructed from characters makes a lot more sense than having them 
constructed from bytes.

Git does the right thing in assuming commit messages and user names be UTF-8 
characters, though, it would have been nice to have file names covered by 
the same constraints.

The fact that Windows messes with this notion just as it messes with the 
file contents (think the endless story whose name is CR/LF) shows only how 
"well" designed the concepts in Windows are.

In this case, yes, Windows' way of doing does make more sense, at least to 
me. And as far as text files are concerned, treating text as sequences of 
bytes are in most cases not a very smart thing to do, either, but it's hard 
not to given how most computers are constructed.

And as it stands, we have at least two issues on the msysGit issue tracker 
that complain that Git does not work with localized file names properly.

So no, file names are not UTF-8 at all, especially not on Windows.

I am not trying to make file names *on Windows* to be UTF-8. I am trying to 
make file names on Windows be Windows file names, i.e UTF-16 Unicode. It's 
just that since Git internally uses the char* APIs, and from what I have 
seen in most other cases assume that char* text is UTF-8, I am trying to 
convert from Windows' view of path names to Git's (UTF-16 to UTF-8) and back.

The other way would be to keep the char* APIs but convert to the Windows 
locale encoding ("ANSI codepage"), but that will break horribly as not all 
file names that can be used on a file system can be represented as such. 
Plus, all calls to a Windows API using a char* path name *is* converted into 
UTF-16 anyway, since that is what is used internally in the Windows NT 
subsystems.

Do not get me wrong, I really welcome you taking care of the issue, but I a
do not think that forcing UTF-8 is a solution.

Some kind of handling of Git repositories where file names are not UTF-8 
would probably need to be added, yes.

-- 
\\// Peter - http://www.softwolves.pp.se/

Peter Krefting schrieb:

Johannes Sixt:

quoted

If git prints a file name under the assumption that it is UTF-8
encoded, then it will be displayed incorrectly because the system uses
a different encoding.

Here setting the local codepage to UTF-8 *might* work, although I
haven't tested that. Or always use the wchar_t versions of printf and
friends.

You cannot expect users to switch the locale. For example, I have to test
our software with Japanese settings: I *cannot* switch to UTF-8 just
because of git.

Can you set the local codepage per program? (I don't know.) It might help
here, but it doesn't help in all cases, particularly in certain pipelines:

  git ls-files -o
  git ls-files -o | git update-index --add --stdin
  find . -name \*.jpg | git update-index --add --stdin

- What encoding should 'ls-files' use for its output? Certainly not always
UTF-8: stdout should use the local code page so that the file names are
interpreted correctly by the terminal window (it expects the local code page).

- What encoding should 'update-index' expect from its input? Can you be
sure that other programs generate UTF-8 output?

How do you solve that?

-- Hannes

Johannes Sixt:

Can you set the local codepage per program? (I don't know.)

The locale is set per thread, and gets reset when the program exits. So 
setting the codepage to UTF-8 before outputting should work. That should 
also work for displaying the log to the terminal if you have UTF-8 log 
messages.

Converting it to wchar_t and using wprintf and similar should be safer, 
though (and I have no idea what happens if you try to pipe the output to 
something else).

- What encoding should 'ls-files' use for its output? Certainly not always 
UTF-8: stdout should use the local code page so that the file names are 
interpreted correctly by the terminal window (it expects the local code 
page).

That is exactly why trying to mix "protocol" data ("plumbing" in Git's case) 
and user output will always come back and bite you, one way or another. I 
haven't really the faintest how pipes work with Unicode on Windows. 
Somewhere along the line there will probably be some conversions, which 
would cause interesting issues.

Better not use pipes, then. Heh. I sense that there is a slight problem with 
the architecture of Git and trying to get it to behave on Windows... :-)

- What encoding should 'update-index' expect from its input? Can you be 
sure that other programs generate UTF-8 output?

Theoretically, if all the internal stuff is hacked around to output Unicode, 
and the thread codepage is set up to use UTF-8, it should "just work". And 
if run directly from the shell, it should still be converted to whatever the 
system is set up to emit. That would mean, however, that a Git program that 
internally runs

   git-foo | git-bar | git-gazonk

might behave differently compared to if a user would enter it on the 
command-line.

-- 
\\// Peter - http://www.softwolves.pp.se/

Johannes Sixt:

quoted

Can you set the local codepage per program? (I don't know.)

The locale is set per thread, and gets reset when the program exits. So 
setting the codepage to UTF-8 before outputting should work. That should 
also work for displaying the log to the terminal if you have UTF-8 log 
messages.

Messing with locale is probably going to break subtly. An explicit approach
is better, respecting the user's locale when necessary.

Converting it to wchar_t and using wprintf and similar should be safer, 
though (and I have no idea what happens if you try to pipe the output to 
something else).

quoted

- What encoding should 'ls-files' use for its output? Certainly not always 
UTF-8: stdout should use the local code page so that the file names are 
interpreted correctly by the terminal window (it expects the local code 
page).

That is exactly why trying to mix "protocol" data ("plumbing" in Git's case) 
and user output will always come back and bite you, one way or another. I 
haven't really the faintest how pipes work with Unicode on Windows. 
Somewhere along the line there will probably be some conversions, which 
would cause interesting issues.

Pipes are just bytes so you have to know what you're piping by convention
or protocol. You can ask for the console output page, which may be set to
a multibyte locale or unicode and maybe trust that.... (just guessing, really).

Better not use pipes, then. Heh. I sense that there is a slight problem with 
the architecture of Git and trying to get it to behave on Windows... :-)

architecture? Like the "architecture" of species? No, it's evolution.
If that applies to the linux kernel, it's not so strange it applies to git too.

quoted

- What encoding should 'update-index' expect from its input? Can you be 
sure that other programs generate UTF-8 output?

Theoretically, if all the internal stuff is hacked around to output Unicode, 
and the thread codepage is set up to use UTF-8, it should "just work". And 

msys doesn't seem to understand UTF-8 at all, so depending on that to work
seems futile. Simply bypassing the locale for any internal work is probably the 
most sane thing. That also won't depend of the quality of the locale support in 
the runtime. Start by making the git commands working without msys bash,
and figure a way to fix msys later, unless someone has a very good idea on
how to fix msys.

if run directly from the shell, it should still be converted to whatever the 
system is set up to emit. That would mean, however, that a Git program that 
internally runs

   git-foo | git-bar | git-gazonk

might behave differently compared to if a user would enter it on the 
command-line.

You might also want to check out my work in the area. See 

http://www.jgit.org/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=shortlog;h=i18n

The goal is locale neutrality yielding the "expected", in the users eyes, result regardless
of locale as much as possible. Junio didn't want to have it for five years, so I
guess there's still three and half to go. Hopefully he can change his mind. That branch
is heavily outdated by now, as some of functionality have been introduced by other
means like logoutputencoding and other parts of git have been rewritten.

Related to this, JGit assumes UTF-8 on reading. If it's not valid UTF-8 we try the user's 
locale (rougly) and on writing object meta data, including any sort of identifier, 
we always write UTF-8 when have to be explicit. We let the runtime decide on how
to encode file names in the file system using the user's locale.

I'd be almost happy with a solution that works when people are interacting using
the subset that is convertible between the character sets in use.

-- robin

Robin Rosenberg:

Pipes are just bytes so you have to know what you're piping by convention 
or protocol. You can ask for the console output page, which may be set to 
a multibyte locale or unicode and maybe trust that.... (just guessing, 
really).

You can get cmd.exe to write data to pipes and redirections as UTF-16 
Unicode (cmd.exe /u), perhaps there is a way to capitalise on that? 
"Unfortunately", the Git stuff is mostly called from a bash shell inside 
msys, so it requires a "bit" more work...

architecture? Like the "architecture" of species? No, it's evolution.

There's still an architecture there, somewhere. Perhaps not intended or 
specified, but there definitely is one :-)

http://www.jgit.org/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=shortlog;h=i18n

The goal is locale neutrality yielding the "expected", in the users eyes, 
result regardless of locale as much as possible.

Ah, yes, that looks like an interesting starting point. I already assumed 
that Git on Linux would use UTF-8 for everything already, since it already 
does that for the commit messages despite me using an iso8859-1 locale. 
Apparently I haven't done my homework.

We let the runtime decide on how to encode file names in the file system 
using the user's locale.

That's good. That's what I'm trying to achieve. Or, rather, avoid the user 
locale altogether (which is easy on Windows since the file names are always 
stored in Unicode, and the user locale can be bypassed).

I'd be almost happy with a solution that works when people are interacting 
using the subset that is convertible between the character sets in use.

You mean like the "invariant" character set? :-) Using Unicode internally 
(in whatever encoding) is nice, the problem is when you have to interact 
with the world around you.

-- 
\\// Peter - http://www.softwolves.pp.se/

måndag 02 mars 2009 21:52:41 skrev Peter Krefting [off-list ref]:

Robin Rosenberg:

quoted

I'd be almost happy with a solution that works when people are interacting 
using the subset that is convertible between the character sets in use.

You mean like the "invariant" character set? :-) Using Unicode internally 
(in whatever encoding) is nice, the problem is when you have to interact 
with the world around you.

Not sure what that is. I mean that in a local nordic, setting people can use iso-8859-1|15/windows-1252/UTF-8 for their needs be means of converting the characters as-needed without loss, with very few practial restrictions. 

For a larger setting that won't do, but then the need is typically less since people tend to use ASCII only, or you jump to all unicode.

Just because I use UTF-8 doesn't mean I use start using more characters in practice.

-- robin

Robin Rosenberg:

Not sure what that is.

"Invariant" is defined in an old RFC as the common subset of several 
ASCII-like and ASCII-based encodings. This was back before the MIME days, 
IIANM.

I mean that in a local nordic, setting people can use 
iso-8859-1|15/windows-1252/UTF-8 for their needs be means of converting 
the characters as-needed without loss, with very few practial 
restrictions.

Indeed. The trick is to have the storage (in this case, Git and it's tree 
objects) storing the file name data in a commonly agreed-upon way. Then it 
is simple to convert at the end-points.

Just because I use UTF-8 doesn't mean I use start using more characters in 
practice.

Most people do not, no. But using a Unicode encoding means that they at 
least have the option. Sometimes, having to mangle stuff down to ASCII is a 
pain.

-- 
\\// Peter - http://www.softwolves.pp.se/

On Mon, Mar 02, 2009 at 09:47:22AM +0100, Peter Krefting wrote:

When opening a file through open() or fopen(), the path passed is
UTF-8 encoded. To handle this on Windows, we need to convert the
path string to UTF-16 and use the Unicode-based interface.

IMHO, you grossly underestimate what is needed to enable UTF-8 encoding
in Windows. AFAIK, Microsoft C runtime library does not support UTF-8,
so you have to wrap all C functions taking 'char*' as an input parameter.
For example, think about what is going to happen if Git tries to print
a simple error message:
  fprintf (stderr, "unable to open %s", path);

Since there is no real file system abstraction beyond using stdio_
(AFAIK), I need to hack it by replacing fopen (and open). Probably_
opendir/readdir as well (might be trickier), and possibly even hack_
around main() to parse the wchar_t command-line instead of the char copy.

And the command-line is not the only source of file names. Some Git
commands read list of files from stdin usually though the pipe. In
what encoding are they going to be?

Dmitry

Dmitry Potapov:

IMHO, you grossly underestimate what is needed to enable UTF-8 encoding in 
Windows. AFAIK, Microsoft C runtime library does not support UTF-8, so you 
have to wrap all C functions taking 'char*' as an input parameter.

I have to wrap all file-related functions, at least.

For example, think about what is going to happen if Git tries to print a 
simple error message: fprintf (stderr, "unable to open %s", path);

Yeah. That's a problem. That might be solvable by setting the thread locale 
to something UTF-8 based and have the console window convert to the output 
codepage (that is what it does when you use wprintf and friends).

And the command-line is not the only source of file names. Some Git 
commands read list of files from stdin usually though the pipe. In what 
encoding are they going to be?

Indeed. Pipes are a problem.

-- 
\\// Peter - http://www.softwolves.pp.se/

Slightly related; A new cygwin (not msysgit-related) version with UTF-8 support was announced. Most notably:

- New setlocale implementation allows to specify POSIX locale strings.
  You can now use, for instance in bash, `export LC_ALL=en_US.UTF-8'.
  The language and territory will be ignored for now, the charset
  will be used by multibyte-releated functions.

- UTF-8 filenames are supported now. 

- Support UTF-8 in console window.

This certainly makes it more feasable to interoperate with *nix repos that has non-ascii metadata and file names.

-- robin