Git, Mac OS X and German special characters

Hi,

I have been using git (version 1.7.1) for quite some time to
backup/synchronize folders between my two linux workstations at home and
at work and my Apple laptop. Synchronization works well except for some
strange behavior under Mac OS X (10.6).

I have commited some files with German special characters, say,
"Übersicht.xls" under linux and pulled them on the Macbook. However, on
the laptop git status says:

# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       "U\314\210bersicht.xls"
nothing added to commit but untracked files present (use "git add" to track)

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?

Thanks,
Matthias

On Thu, May 20, 2010 at 07:26, Matthias Moeller
[off-list ref] wrote:

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?

#       "U\314\210bersicht.xls"

You probably have to configure your shell on OSX to render UTF-8
correctly. It's just showing the raw escaped byte sequence instead of
a character there.

There isn't anything wrong with OSX in this case, filename encoding on
any POSIX system is only done by convention. You'll find that you have
similar problems on Linux if you encode filename in Big5 or
UTF-32.

Linux will happily accept it, but your shell / other applications will
render it as unknown goo because they expect UTF-8.

Ævar Arnfjörð Bjarmason venit, vidit, dixit 20.05.2010 10:34:

On Thu, May 20, 2010 at 07:26, Matthias Moeller
[off-list ref] wrote:

quoted

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?

quoted

#       "U\314\210bersicht.xls"

You probably have to configure your shell on OSX to render UTF-8
correctly. It's just showing the raw escaped byte sequence instead of
a character there.

There isn't anything wrong with OSX in this case, filename encoding on
any POSIX system is only done by convention. You'll find that you have
similar problems on Linux if you encode filename in Big5 or
UTF-32.

Linux will happily accept it, but your shell / other applications will
render it as unknown goo because they expect UTF-8.

No, the problem with git status is not the display. Matthias' problem is
that git status reports a tracked file as untracked. The reason is that
on HFS+, you create a file with name A and get a file with name B, where
A and B are different representations of the same name. There seems to
be no way to reliably detect which one HFS+ uses.

Michael

On 20 May 2010 10:34, Ævar Arnfjörð Bjarmason [off-list ref] wrote:

On Thu, May 20, 2010 at 07:26, Matthias Moeller
[off-list ref] wrote:

quoted

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?

quoted

#       "U\314\210bersicht.xls"

You probably have to configure your shell on OSX to render UTF-8
correctly. It's just showing the raw escaped byte sequence instead of
a character there.

There isn't anything wrong with OSX in this case, filename encoding on
any POSIX system is only done by convention. You'll find that you have
similar problems on Linux if you encode filename in Big5 or
UTF-32.

Linux will happily accept it, but your shell / other applications will
render it as unknown goo because they expect UTF-8.

Except that isnt a normalized utf8 representation of capital U umlaut,
code point U+00DC, (utf8 c3,9c), instead presumably it has been
decomposed into a captial U followed by a combining character to add
in the umlaut, which IMO is pretty weird.

As far as i can tell the filename:

"Übersicht.xls"

Should be stored in utf8 as:

"\303\234bersicht.xls"

Also minor nit. UTF-32, as it contains nuls for latin-1 chars would be
much much worse than utf8 :-)

cheers,
Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

On 20 May 2010 10:50, Michael J Gruber [off-list ref] wrote:

Ævar Arnfjörð Bjarmason venit, vidit, dixit 20.05.2010 10:34:

quoted

On Thu, May 20, 2010 at 07:26, Matthias Moeller
[off-list ref] wrote:

quoted

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?

quoted

#       "U\314\210bersicht.xls"

You probably have to configure your shell on OSX to render UTF-8
correctly. It's just showing the raw escaped byte sequence instead of
a character there.

There isn't anything wrong with OSX in this case, filename encoding on
any POSIX system is only done by convention. You'll find that you have
similar problems on Linux if you encode filename in Big5 or
UTF-32.

Linux will happily accept it, but your shell / other applications will
render it as unknown goo because they expect UTF-8.

No, the problem with git status is not the display. Matthias' problem is
that git status reports a tracked file as untracked. The reason is that
on HFS+, you create a file with name A and get a file with name B, where
A and B are different representations of the same name. There seems to
be no way to reliably detect which one HFS+ uses.

Judging by the example given the problem is that HFS+ decomposes
Unicode file names into latin1+combining characters instead of using
normalized utf8.

This implies that if the utf8 is normalized first using canonical
unicode normalization rules (to eliminate the combining character)
that it can then be compared.

Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Hej,
I have the same problem here.
Below there is a patch, which may solve the problem.
(Yes, whitespaces are broken. I'm still fighting with
git format-patch -s --cover-letter -M --stdout origin/master | git 
imap-send)
But this patch may be a start point for improvements.
Comments welcome
BR
/Torsten



Improved interwork between Mac OS X and linux when umlauts are used
When a git repository containing utf-8 coded umlaut characters
is cloned onto an Mac OS X machine, the Mac OS system will convert
all filenames returned by readdir() into denormalized utf-8.
As a result of this conversion, git will not find them on disk.
This helps by treating the NFD and NFD version of filenames as
identical on Mac OS.






Signed-off-by: Torsten Bögershausen <redacted>
---
name-hash.c |   40 ++++++++++++++++++++++++++++++++++++++++
utf8.c      |   55 ++++++++++++++++++++++++++++++++++++++++++++++++-------
utf8.h      |   11 +++++++++++
3 files changed, 99 insertions(+), 7 deletions(-)

diff --git a/name-hash.c b/name-hash.c
index 0031d78..e6494e8 100644
--- a/name-hash.c
+++ b/name-hash.c

@@ -7,6 +7,7 @@
  */

#define NO_THE_INDEX_COMPATIBILITY_MACROS
#include "cache.h"
+#include "utf8.h"

/*
  * This removes bit 5 if bit 6 is set.

@@ -100,6 +101,25 @@ static int same_name(const struct cache_entry *ce, 

const char *name, int namelen
     return icase && slow_same_name(name, namelen, ce->name, len);
}

+#ifdef __APPLE__
+struct cache_entry *index_name_exists2(struct index_state *istate, 
const char *name, int icase)
+{
+    int namelen = (int)strlen(name);
+    unsigned int hash = hash_name(name, namelen);
+    struct cache_entry *ce;
+
+    ce = lookup_hash(hash, &istate->name_hash);
+    while (ce) {
+        if (!(ce->ce_flags & CE_UNHASHED)) {
+            if (same_name(ce, name, namelen, icase))
+                return ce;
+        }
+        ce = ce->next;
+    }
+    return NULL;
+}
+#endif
+
struct cache_entry *index_name_exists(struct index_state *istate, const 
char *name, int namelen, int icase)
{
     unsigned int hash = hash_name(name, namelen);

@@ -115,5 +135,25 @@ struct cache_entry *index_name_exists(struct 

index_state *istate, const char *na
         }
         ce = ce->next;
     }
+#ifdef __APPLE__
+    {
+        char *name_nfc_nfd;
+        name_nfc_nfd = str_nfc2nfd(name);
+        if (name_nfc_nfd) {
+            ce = index_name_exists2(istate, name_nfc_nfd, icase);
+            free(name_nfc_nfd);
+            if (ce)
+                return ce;
+        }
+        name_nfc_nfd = str_nfd2nfc(name);
+        if (name_nfc_nfd) {
+            ce = index_name_exists2(istate, name_nfc_nfd, icase);
+            free(name_nfc_nfd);
+            if (ce)
+                return ce;
+        }
+    }
+#endif
+
     return NULL;
}

diff --git a/utf8.c b/utf8.c
index 84cfc72..8e794dc 100644
--- a/utf8.c
+++ b/utf8.c

@@ -2,6 +2,11 @@

#include "strbuf.h"
#include "utf8.h"

+#ifdef __APPLE__
+static iconv_t my_iconv_nfd2nfc = (iconv_t) -1;
+static iconv_t my_iconv_nfc2nfd = (iconv_t) -1;
+#endif
+
/* This code is originally from http://www.cl.cam.ac.uk/~mgk25/ucs/ */

struct interval {

@@ -424,18 +429,13 @@ int is_encoding_utf8(const char *name)

#else
     typedef char * iconv_ibp;
#endif
-char *reencode_string(const char *in, const char *out_encoding, const 
char *in_encoding)
+
+char *reencode_string_iconv(const char *in, iconv_t conv)
{
-    iconv_t conv;
     size_t insz, outsz, outalloc;
     char *out, *outpos;
     iconv_ibp cp;

-    if (!in_encoding)
-        return NULL;
-    conv = iconv_open(out_encoding, in_encoding);
-    if (conv == (iconv_t) -1)
-        return NULL;
     insz = strlen(in);
     outsz = insz;
     outalloc = outsz + 1; /* for terminating NUL */

@@ -469,7 +469,48 @@ char *reencode_string(const char *in, const char 

*out_encoding, const char *in_e
             break;
         }
     }
+    return out;
+}
+
+char *reencode_string(const char *in, const char *out_encoding, const 
char *in_encoding)
+{
+    iconv_t conv;
+    char *out;
+
+    if (!in_encoding)
+        return NULL;
+    conv = iconv_open(out_encoding, in_encoding);
+    if (conv == (iconv_t) -1)
+        return NULL;
+    out = reencode_string_iconv(in, conv);
     iconv_close(conv);
     return out;
}
+
+#ifdef __APPLE__
+char*
+str_nfc2nfd(const char *in)
+{
+    if (my_iconv_nfc2nfd == (iconv_t) -1) {
+        my_iconv_nfc2nfd = iconv_open("utf-8-mac", "utf-8");
+        if (my_iconv_nfc2nfd == (iconv_t) -1) {
+            return NULL;
+        }
+    }
+    return reencode_string_iconv(in, my_iconv_nfc2nfd);
+}
+
+char*
+str_nfd2nfc(const char *in)
+{
+    if (my_iconv_nfd2nfc == (iconv_t) -1){
+        my_iconv_nfd2nfc = iconv_open("utf-8", "utf-8-mac");
+        if (my_iconv_nfd2nfc == (iconv_t) -1) {
+            return NULL;
+        }
+    }
+    return reencode_string_iconv(in, my_iconv_nfd2nfc);
+}
+#endif /* APPLE */
+
#endif

diff --git a/utf8.h b/utf8.h
index ebc4d2f..db29c8a 100644
--- a/utf8.h
+++ b/utf8.h

@@ -13,8 +13,19 @@ int strbuf_add_wrapped_text(struct strbuf *buf,

#ifndef NO_ICONV
char *reencode_string(const char *in, const char *out_encoding, const 
char *in_encoding);
+char *reencode_string_iconv(const char *in, iconv_t conv);
+#ifdef __APPLE__
+char *str_nfc2nfd(const char *in);
+char *str_nfd2nfc(const char *in);
+#else
+#define str_nfc2nfd(in) (NULL)
+#define str_nfd2nfc(in) (NULL)
+#endif
#else
#define reencode_string(a,b,c) NULL
+#define reencode_string2(a,b) NULL
+#define str_nfc2nfd(in) (NULL)
+#define str_nfd2nfc(in) (NULL)
#endif

#endif
-- 
1.7.1.dirty










On 20.05.10 10:50, Michael J Gruber wrote:

Ævar Arnfjörð Bjarmason venit, vidit, dixit 20.05.2010 10:34:
   

quoted

On Thu, May 20, 2010 at 07:26, Matthias Moeller
[off-list ref]  wrote:
     

quoted

I have been searching the web for help and found lengthy discussions
which state that this is a common problem of the HFS+ filesystem.
What I did not find was a solution to this problem. Is there a solution
to this problem?
       

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?

     

quoted

#       "U\314\210bersicht.xls"
       

You probably have to configure your shell on OSX to render UTF-8
correctly. It's just showing the raw escaped byte sequence instead of
a character there.

There isn't anything wrong with OSX in this case, filename encoding on
any POSIX system is only done by convention. You'll find that you have
similar problems on Linux if you encode filename in Big5 or
UTF-32.

Linux will happily accept it, but your shell / other applications will
render it as unknown goo because they expect UTF-8.
     

No, the problem with git status is not the display. Matthias' problem is
that git status reports a tracked file as untracked. The reason is that
on HFS+, you create a file with name A and get a file with name B, where
A and B are different representations of the same name. There seems to
be no way to reliably detect which one HFS+ uses.

Michael
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
   

Torsten Bögershausen venit, vidit, dixit 20.05.2010 11:02:

Hej,
I have the same problem here.
Below there is a patch, which may solve the problem.
(Yes, whitespaces are broken. I'm still fighting with
git format-patch -s --cover-letter -M --stdout origin/master | git 
imap-send)
But this patch may be a start point for improvements.
Comments welcome
BR
/Torsten



Improved interwork between Mac OS X and linux when umlauts are used
When a git repository containing utf-8 coded umlaut characters
is cloned onto an Mac OS X machine, the Mac OS system will convert
all filenames returned by readdir() into denormalized utf-8.
As a result of this conversion, git will not find them on disk.
This helps by treating the NFD and NFD version of filenames as
identical on Mac OS.






Signed-off-by: Torsten Bögershausen <redacted>

You signed off, but is Markus Kuhn's code from UCS GPL2-licensed?
Also, a few tests would be nice.

I remember we had threads on this issue in the past. I haven't checked
yet (Thunderbird pruned my nntp history), but it is worth checking that
you addressed any issues mentioned there.

I have no Mac so I can't test, sorry. Would be happy to run Mac OS in a
vm, but you know...

Thanks for looking into this!

Michael

On 05/20/2010 10:50 AM, Michael J Gruber wrote:

quoted

Is this problem particular to Git, or do you also get it if you
e.g. rsync from the Linux box to the Mac OS X box?
    

No, the problem with git status is not the display. Matthias' problem is
that git status reports a tracked file as untracked. The reason is that
on HFS+, you create a file with name A and get a file with name B, where
A and B are different representations of the same name. There seems to
be no way to reliably detect which one HFS+ uses.
  

Yes, the problem is not the display but the filesystem. I had similar
problems with unison some time ago.
But there was a special fix for utf-8 and Mac OS X in one of the newer
unison versions.

Matthias

On 20.05.2010 10:50, Michael J Gruber wrote:

There seems to be no way to reliably detect which one HFS+ uses.

IIRC, HFS+ always stores and reports the file names with decomposed UTF-8,
even if the file is created using composed UTF-8. IMHO, Git should
standardize on the file and text encoding (e.g. commit messages) used in the
repository, so such problems can't occur. SVN has standardized on "UTF-8" in
the repository, but had/s similar problems on OS X with the decomposed
characters:

 http://subversion.tigris.org/issues/show_bug.cgi?id=2464

Tom

2010/5/20 Michael J Gruber [off-list ref]:

I remember we had threads on this issue in the past. I haven't checked
yet (Thunderbird pruned my nntp history), but it is worth checking that
you addressed any issues mentioned there.

This was the monster thread on it:

  http://thread.gmane.org/gmane.comp.version-control.git/70688

Linus added support for the case-insensitivity aliasing issue:

  1102952 (Make git-add behave more sensibly in a case-insensitive environment,
           2008-03-22)
  6835550 (When adding files to the index, add support for
case-independent matches,
           2008-03-22)

But he couldn't care less about HFS+ brain-damage, so he left that for
others. And hey, it's only taken 2 years for someone to step up to the
plate. :-)

j.

2010/5/20 Torsten Bögershausen [off-list ref]:

Improved interwork between Mac OS X and linux when umlauts are used
When a git repository containing utf-8 coded umlaut characters
is cloned onto an Mac OS X machine, the Mac OS system will convert
all filenames returned by readdir() into denormalized utf-8.
As a result of this conversion, git will not find them on disk.
This helps by treating the NFD and NFD version of filenames as
identical on Mac OS.

So this is an edge case, but what happens if a repo has both the NFC
and NFD representation of a given name? It should be handled the same
way as if a repo has both "File" and "file" and you try to check-out
onto a case-insensitive filesystem.

Additionally, note this paragraph from 1102952 (Make git-add behave
more sensibly in a case-insensitive environment, 2008-03-22):

    However, if we actually have *both* a file called "File" and one called
    "file", and they don't have the same lstat() information (ie we're on a
    case-sensitive filesystem but have the "core.ignorecase" flag set), we
    will error out if we try to add them both.

To be consistent, shouldn't we have a core.HFSPlusCompat that can be
set on non-braindamaged filesystems to prevent filenames which would
alias on HFS+ from entering the repo?

   http://developer.apple.com/mac/library/technotes/tn/tn1150.html#UnicodeSubtleties

j.

On Thu, May 20, 2010 at 11:50 AM, Jay Soffian [off-list ref] wrote:

To be consistent, shouldn't we have a core.HFSPlusCompat that can be
set on non-braindamaged filesystems to prevent filenames which would
alias on HFS+ from entering the repo?

  http://developer.apple.com/mac/library/technotes/tn/tn1150.html#UnicodeSubtleties

And here's an implementation (BSD-style license):

  http://src.chromium.org/svn/trunk/src/base/file_path.cc

See GetHFSDecomposedForm and HFSFastUnicodeCompare.

C++ so it'll require adapting.

j.