From: Mark Junker <hidden> Date: 2016-06-15 22:44:05
Hi,
I have some files like "Lüftung.txt" in my repository. The strange thing
is that I can pull / add / commit / push those files without problem but
git-status always complains that thoes files are untraced (but not
missing). My assumption is that it's a problem with the way MacOSX
stores the file names (decomposed UTF-8). So something like
"Lüftung.txt" becomes "Lüftung.txt".
It seems that git-status does two things:
1. Find files under version control (i.e. search for missing files)
2. Find files not under version control (i.e. search for untracked files)
I guess that the first look-up succeeds because MacOS X converts
composed UTF-8 to decomposed UTF-8 when searching for a file. But it
seems that the second look-up takes the file names as-is (decomposed)
without converting them to composed UTF-8.
Is there an easy way to fix this behaviour? It's really annoying to see
all those "untracked" files that are already under version control when
executing a git-status.
Regards,
Mark
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:05
Hi,
On Wed, 16 Jan 2008, Mark Junker wrote:
I have some files like "Lüftung.txt" in my repository. The strange thing is
that I can pull / add / commit / push those files without problem but
git-status always complains that thoes files are untraced (but not missing).
This is a known problem. Unfortunately, noone has implemented a fix,
although if you're serious about it, I can point you to threads where it
has been hinted how to solve the issue.
FWIW the issue is that Mac OS X decides that it knows better how to encode
your filename than you could yourself.
Ciao,
Dscho
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:05
On Jan 16, 2008, at 10:34 AM, Johannes Schindelin wrote:
On Wed, 16 Jan 2008, Mark Junker wrote:
quoted
I have some files like "Lüftung.txt" in my repository. The strange
thing is
that I can pull / add / commit / push those files without problem but
git-status always complains that thoes files are untraced (but not
missing).
This is a known problem. Unfortunately, noone has implemented a fix,
although if you're serious about it, I can point you to threads
where it
has been hinted how to solve the issue.
FWIW the issue is that Mac OS X decides that it knows better how to
encode
your filename than you could yourself.
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard of
that has a problem with OS X using Unicode.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:05
Hi,
On Wed, 16 Jan 2008, Kevin Ballard wrote:
On Jan 16, 2008, at 10:34 AM, Johannes Schindelin wrote:
quoted
On Wed, 16 Jan 2008, Mark Junker wrote:
quoted
I have some files like "Lüftung.txt" in my repository. The strange
thing is that I can pull / add / commit / push those files without
problem but git-status always complains that thoes files are
untraced (but not missing).
This is a known problem. Unfortunately, noone has implemented a fix,
although if you're serious about it, I can point you to threads where
it has been hinted how to solve the issue.
FWIW the issue is that Mac OS X decides that it knows better how to
encode your filename than you could yourself.
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard of that
has a problem with OS X using Unicode.
No. That's not at all the problem. Mac OS X insists on storing _another_
encoding of your filename. Both are UTF-8. Both encode the _same_
string. Yet they are different, bytewise. For no good reason.
Stop spreading FUD. Git can handle Unicode just fine. In fact, Git does
not _care_ how the filename is encoded, it _respects_ the user's choice,
not only of the encoding _type_, but the _encoding_, too.
Okay?
Hth,
Dscho
From: Jakub Narebski <hidden> Date: 2016-06-15 22:44:05
Johannes Schindelin [off-list ref] writes:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
On Jan 16, 2008, at 10:34 AM, Johannes Schindelin wrote:
quoted
On Wed, 16 Jan 2008, Mark Junker wrote:
quoted
I have some files like "Lüftung.txt" in my repository. The strange
thing is that I can pull / add / commit / push those files without
problem but git-status always complains that thoes files are
untraced (but not missing).
This is a known problem. Unfortunately, noone has implemented a fix,
although if you're serious about it, I can point you to threads where
it has been hinted how to solve the issue.
FWIW the issue is that Mac OS X decides that it knows better how to
encode your filename than you could yourself.
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard of that
has a problem with OS X using Unicode.
No. That's not at all the problem. Mac OS X insists on storing _another_
encoding of your filename. Both are UTF-8. Both encode the _same_
string. Yet they are different, bytewise. For no good reason.
To be more exact encoding used to _create_ file differs from encoding
returned when _reading directory_...
Stop spreading FUD. Git can handle Unicode just fine. In fact, Git does
not _care_ how the filename is encoded, it _respects_ the user's choice,
not only of the encoding _type_, but the _encoding_, too.
...which means that sequence of bytes differ. And Git by design is
(both for filenames and for blob contents) encoding agnostic.
HFS+ is just _stupid_. And unfortunately Git doesn't support stupid
filesystems (e.g. case insensitive filesystems) well.
--
Jakub Narebski
Poland
ShadeHawk on #git
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:05
On Jan 16, 2008, at 11:46 AM, Jakub Narebski wrote:
quoted
quoted
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard
of that
has a problem with OS X using Unicode.
No. That's not at all the problem. Mac OS X insists on storing
_another_
encoding of your filename. Both are UTF-8. Both encode the _same_
string. Yet they are different, bytewise. For no good reason.
To be more exact encoding used to _create_ file differs from encoding
returned when _reading directory_...
quoted
Stop spreading FUD. Git can handle Unicode just fine. In fact,
Git does
not _care_ how the filename is encoded, it _respects_ the user's
choice,
not only of the encoding _type_, but the _encoding_, too.
...which means that sequence of bytes differ. And Git by design is
(both for filenames and for blob contents) encoding agnostic.
HFS+ is just _stupid_. And unfortunately Git doesn't support stupid
filesystems (e.g. case insensitive filesystems) well.
There's two different ways to do filesystem encodings. One is to have
the fs simply not care about encoding, which is what the linux world
seems to prefer. Sure, this is great in that what you create the file
with is what you get back, but on the other hand, given an arbitrary
non-ASCII file on disk, you have absolutely no idea what the encoding
should be and you can't display it without making assumptions (yes you
can use heuristics, but you're still making assumptions). Filesystems
like HFS+ that standardize the encoding, on the other hand, make it
such that you always know what the encoding of a file should be, so
you can always display and use the filename intelligently. It also
means it plays much nicer in a non-ASCII world, since you don't have
to worry about different normalizations of a given string referring to
different files (it's one thing to be case-sensitive, but claiming
that "föo" and "föo" are different files just because one uses a
composed character and the other doesn't is extremely user-
unfriendly). On the other hand, what you create the file with may not
be what you read back later, since the name has been standardized.
It's hard to say one is better than the other, they're just different
ways of doing it. However, I have noticed that everybody who's voiced
an opinion on this list in favor of the encoding-agnostic approach
seem to be unwilling to accept that any other approach might have
validity, to the extent of calling an OS/filesystem that does things
different stupid or insane. This strikes me as extremely elitist and
risks alienating what I expect to be a fast-growing group of users
(i.e. OS X users).
I'm willing to give Linus a free pass on calling other OS's stupid and
insane, as I don't think Linux would exist as it does today without
his strong opinions, but I don't think this should give carte blanche
to the rest of the community for this inflammatory behavior.
I should note that I'm only taking the time to discuss this because,
despite the fact that I'm new to git, I really like it and I want it
to work better. And one area that it has a problem with is the de-
facto filesystem on my OS of choice. However, attempts to discuss the
problem invariable end up with multiple people calling my OS stupid
and insane simply because it differs in a particular design decision.
This is not a good way to build a community or to build a better
product, and I hope it can be improved.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Jakub Narebski <hidden> Date: 2016-06-15 22:44:05
On Wed, 16 Jan 2008, Kevin Ballard wrote:
On Jan 16, 2008, at 11:46 AM, Jakub Narebski wrote:
quoted
quoted
quoted
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard
which has a problem with OS X using Unicode.
No. That's not at all the problem. Mac OS X insists on storing
_another_ encoding of your filename. Both are UTF-8. Both encode
the _same_ string. Yet they are different, bytewise. For no good
reason.
To be more exact encoding used to _create_ file differs from encoding
returned when _reading directory_...
quoted
Stop spreading FUD. Git can handle Unicode just fine. In fact,
Git does not _care_ how the filename is encoded, it _respects_ the
user's choice, not only of the encoding _type_, but the _encoding_,
too.
...which means that sequence of bytes differ. And Git by design is
(both for filenames and for blob contents) encoding agnostic.
HFS+ is just _stupid_. And unfortunately Git doesn't support stupid
filesystems (e.g. case insensitive filesystems) well.
By the way, calling HFS+ stupid, or rather calling at least two
different normalizations of UTF-8 (two different encodings) used for
writing and reading filenames stupid is wrong _for me_. I have quoted
Linus here, when I think I should use other description.
There's two different ways to do filesystem encodings. One is to have
the fs simply not care about encoding, which is what the linux world
seems to prefer. Sure, this is great in that what you create the file
with is what you get back, but on the other hand, given an arbitrary
non-ASCII file on disk, you have absolutely no idea what the encoding
should be and you can't display it without making assumptions (yes you
can use heuristics, but you're still making assumptions). Filesystems
like HFS+ that standardize the encoding, on the other hand, make it
such that you always know what the encoding of a file should be, so
you can always display and use the filename intelligently. It also
means it plays much nicer in a non-ASCII world, since you don't have
to worry about different normalizations of a given string referring to
different files (it's one thing to be case-sensitive, but claiming
that "föo" and "föo" are different files just because one uses a
composed character and the other doesn't is extremely user-
unfriendly).
For me it looks like a layering violation... but my knowledge about
filesystem is cluse to nil. IMHO it is VFS and libc which should do the
translating.
On the other hand, what you create the file with may not
be what you read back later, since the name has been standardized.
It's hard to say one is better than the other, they're just different
ways of doing it.
But using one encoding to create file, and another when reding filenames
is strange. It is IMHO better to simply refuse creating filenames which
are outside chosen encoding / normalization. But having different
encodings used for reading and writing on the level of filesystem
access (not on level of UI) is strange.
However, I have noticed that everybody who's voiced
an opinion on this list in favor of the encoding-agnostic approach
seem to be unwilling to accept that any other approach might have
validity, to the extent of calling an OS/filesystem that does things
different stupid or insane. This strikes me as extremely elitist and
risks alienating what I expect to be a fast-growing group of users
(i.e. OS X users).
First, it is Git philosophy and very core of design to be encoding
agnostic (to be "content tracker"). Second, using the same sequence of
bytes on filesystem, in the index, and in 'tree' objects ensures good
performance... this is something to think about if you want to add
patches which would deal with HFS+ API/UI quirks.
[cut]
--
Jakub Narebski
Poland
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:05
On Jan 16, 2008, at 4:51 PM, Jakub Narebski wrote:
quoted
On the other hand, what you create the file with may not
be what you read back later, since the name has been standardized.
It's hard to say one is better than the other, they're just different
ways of doing it.
But using one encoding to create file, and another when reding
filenames
is strange. It is IMHO better to simply refuse creating filenames
which
are outside chosen encoding / normalization. But having different
encodings used for reading and writing on the level of filesystem
access (not on level of UI) is strange.
It's not using different encodings, it's all Unicode. However, it
accepts different normalization variants of Unicode, since it can read
them all and it would be folly to require everybody to conform to its
own special internal variant. But it does have to normalize them,
otherwise how would it detect the same filename using different
normalizations? Also, it may seem strange to have different names
between reading and writing, but that's only if you think of the name
as a sequence of bytes - when treated as a sequence of characters, you
get the same result. In other words, you're used to filenames as
bytes, HFS+ treats filenames as strings.
quoted
However, I have noticed that everybody who's voiced
an opinion on this list in favor of the encoding-agnostic approach
seem to be unwilling to accept that any other approach might have
validity, to the extent of calling an OS/filesystem that does things
different stupid or insane. This strikes me as extremely elitist and
risks alienating what I expect to be a fast-growing group of users
(i.e. OS X users).
First, it is Git philosophy and very core of design to be encoding
agnostic (to be "content tracker"). Second, using the same sequence of
bytes on filesystem, in the index, and in 'tree' objects ensures good
performance... this is something to think about if you want to add
patches which would deal with HFS+ API/UI quirks.
Sure, it makes sense from a performance perspective, but it causes
problems with HFS+ and any other filesystem that behaves the same way.
In the previous discussion about case-sensitivity, somebody suggested
using a lookup table to map between git's internal representation and
the name the filesystem returns, which seems like a decent idea and
one that could be enabled with a config parameter to avoid penalizing
repos on other filesystems. But I don't know enough about the
internals of git to even think of trying to implement it myself.
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:05
Hi,
On Wed, 16 Jan 2008, Kevin Ballard wrote:
It's not using different encodings, it's all Unicode.
But that's the _point_! It _is_ Unicode, yet it uses _different_
encodings of the _same_ string.
Now, this discussion gets really annoying. The real question is: will you
do something about it, or reply with another 500-line email?
Ciao,
Dscho
It's not using different encodings, it's all Unicode. However, it accepts
different normalization variants of Unicode, since it can read them all and it
would be folly to require everybody to conform to its own special internal
variant. But it does have to normalize them, otherwise how would it detect the
same filename using different normalizations?
That's a singularly *stupid* argument.
Here, let me rephrase that same idiotic argument:
"But it does have to uppercase them, otherwise how would it detect the
same filename using different cases?"
..and if you don't see how that's *exactly* the same argument, you really
are stupid.
The fact is, normalization is wrong.
It's wrong when you normalize upper/lower case (no, the word "Polish" is
not the same as "polish"), and it's equally wrong when you normalize for
"looks similar".
In other words, you're used to filenames as bytes, HFS+ treats filenames
as strings.
No. HFS+ treats users as idiots and thinks that it should "fix" the
filename for them. And it causes problems.
It causes problems for exactly the same reasons case-independence causes
problems, because it's EXACTLY THE SAME ISSUE. People may think that "but
they are the same", but they aren't. Case matters. And so does "single
character" vs "two character overlay".
Does it always matter? Hell no. But the problem with a filesystem that
thinks it knows better is that when it *sometimes* matters, the filesystem
simply DOES THE WRONG THING.
Can't you understand that?
Linus
Does it always matter? Hell no. But the problem with a filesystem that
thinks it knows better is that when it *sometimes* matters, the filesystem
simply DOES THE WRONG THING.
Can't you understand that?
Side note: there are ways to do it right.
You can:
- not do conversion at all (which is always right). Not corrupting the
user data means that the user never gets something back that he didn't
put in
(And, btw, the "security" argument is total BS. The fact that two
characters look the same does not mean that they should act the same,
and it is *not* a security feature. Quite the reverse. Having programs
that get different results back from what they actually wrote, *that*
tends to be a security issue, because now you have a confused program,
and I guarantee that there are more bugs in unexpected cases than in
the expected ones)
- Not accept data in formats that you don't like. This is also always
right, but can be rather impolite.
- Not accept data in formats that you don't like, and give people
explicit conversion and comparison routines so that they can then make
their own decisions and they are *aware* of the conversion (so that
they don't come back to the problem of being confused)
So there are certainly many ways to handle things like this.
The one thing you shouldn't do is to silently convert data behind the
programs back, without even giving any way to disable it (and that disable
has to be on a use-by-use casis, not some "disable/enable for all users of
this filesystem", because you can - and do - have different programs that
have different expectations).
And finally: all of the above is true at *all* levels. It doesn't matter
one whit whether the automatic conversion conversion is in the kernel or
in a library. Doing it on a library level has advantages (namely the whole
"disable/enable" thing tends to get *much* easier to do, and applications
can decide to link against a particular version to get the behaviour
*they* want, for example).
So doing it inside the kernel is just about the worst possible case,
exactly because it makes it really hard to do a "on a case-by-case" basis.
Yes, Linux does it too, but it does it only for filesystems that are
*defined* to be insane. OS X really should have known better. Especially
since they already fixed the applications (ie they do allow for
case-sensitive filesystems).
I can understand normalization when it's about case-insensitivity (there
are lots of _technical_ reasons to do it there), but once you let the
case-insensitivity go, there just isn't any excuse any more.
Linus
El 16/1/2008, a las 16:43, Kevin Ballard escribió:
On Jan 16, 2008, at 10:34 AM, Johannes Schindelin wrote:
quoted
On Wed, 16 Jan 2008, Mark Junker wrote:
quoted
I have some files like "Lüftung.txt" in my repository. The strange
thing is
that I can pull / add / commit / push those files without problem
but
git-status always complains that thoes files are untraced (but not
missing).
This is a known problem. Unfortunately, noone has implemented a fix,
although if you're serious about it, I can point you to threads
where it
has been hinted how to solve the issue.
FWIW the issue is that Mac OS X decides that it knows better how to
encode
your filename than you could yourself.
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard of
that has a problem with OS X using Unicode.
As far as I know, Subversion has basically exactly the same problem,
and any time you consume/produce files on Mac OS X that are be
consumed/produced on other platforms you will run into this kind of
issue, with any software.
Tell Mac OS X to write a file with "ó" in the file name ("\xc3\xb3" in
UTF-8), and it will "normalize" it prior to writing by converting it
into a decomposed form (that is, ASCII "o" followed by "\xcc\x81", or
"combining acute accent"). So they're both valid Unicode, both valid
UTF-8, and they encode exactly the same characters but the byte stream
is different.
If you only work on Mac OS X then this will never be a problem because
all the files you create and therefore all the files you add to your
Git repository will have their names in decomposed UTF-8. But when you
start cloning repositories containing files added on other systems,
systems which might use precomposed rather than decomposed UTF-8 then
you'll run into exactly this kind of problem. The git.git repo has one
such file itself (gitweb/test/Märchen, if I remember correctly, which
Git reports as untracked).
Now, Mac OS X's behaviour is not entirely "insane" as some would
claim; there is indeed a rationale behind it even if you don't agree
with it, but it *does* produce some unfortunate teething problems for
people wanting to use Mac OS X in a cross-platform environment.
Here are some Apple docs on the subject:
http://developer.apple.com/qa/qa2001/qa1173.htmlhttp://developer.apple.com/qa/qa2001/qa1235.html
I personally wish that UTF-8 didn't allow different normalization
forms; then this kind of problem wouldn't arise. But it has arisen and
we have to live with it. Some workarounds have been proposed for Git,
but I haven't seen any convincing proposals yet.
Cheers,
Wincent
On 16. jan.. 2008, at 17.32, Johannes Schindelin wrote:
quoted
quoted
FWIW the issue is that Mac OS X decides that it knows better how to
encode your filename than you could yourself.
More like, Mac OS X has standardized on Unicode and the rest of the
world hasn't caught up yet. Git is the only tool I've ever heard of
that
has a problem with OS X using Unicode.
No. That's not at all the problem. Mac OS X insists on storing
_another_
encoding of your filename. Both are UTF-8. Both encode the _same_
string. Yet they are different, bytewise. For no good reason.
Stop spreading FUD. Git can handle Unicode just fine. In fact, Git
does
not _care_ how the filename is encoded, it _respects_ the user's
choice,
not only of the encoding _type_, but the _encoding_, too.
"FUD" is a bit strong, don't you think? HFS+ is the way it is and it
would be nice if Git could deal with it.
The problem is that HFS+ normalizes filenames to avoid multiple files
that appear to have the same name (eg "M<A WITH UMLAUT>rchen" vs
"Ma<UMLAUT MODIFIER>rchen", in gitweb/test). This is sort of like
case sensitivity, but filenames are normalized when a file is
_created_. Git, not unreasonably, expects a file to keep the name it
was created with.
As far as I can tell, as long as you add all your internationally
becharactered files to git from an HFS+ file system using a gui or
command-line completion, you'll be okay; trouble starts when you check
in a file with the composed form of a character, by typing the name on
the command line (I'm not sure about this one) or committing on
another OS. Git will store the filename in composed form, but the
Mac's filesystem will decompose the filename when you check the file
out.
The result looks like this:
vredefort:[git]% git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to
track)
(this is directly after checking out git.git @ v1.5.4-rc3)
There are two things to note here. One is that Git thinks that there
is a new file called "gitweb/test/Märchen" (decomposed) when it's
"really" just the same "gitweb/test/Märchen" (precomposed) that's in
the repository. The other is that git _thinks_ that the "gitweb/test/
Märchen" (precomposed) it's expecting is still there, because the
filesystem, when asked for "gitweb/test/Märchen" in any form will
return the file "gitweb/test/Märchen" (decomposed).
Trying to check out the "next" branch at this point is a pain since
next's "Märchen" would overwrite the untracked "Märchen".
I can't provide links to any previous discussions about this, but
here's Apple's Technical Q&A on the subject:
http://developer.apple.com/qa/qa2001/qa1235.html
Finding a sane way of allowing git to handle this behaviour is left as
an exercise for the reader.
Eyvind Bernhardsen
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:05
On Jan 16, 2008, at 5:32 PM, Linus Torvalds wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
It's not using different encodings, it's all Unicode. However, it
accepts
different normalization variants of Unicode, since it can read them
all and it
would be folly to require everybody to conform to its own special
internal
variant. But it does have to normalize them, otherwise how would it
detect the
same filename using different normalizations?
That's a singularly *stupid* argument.
Here, let me rephrase that same idiotic argument:
"But it does have to uppercase them, otherwise how would it detect
the
same filename using different cases?"
..and if you don't see how that's *exactly* the same argument, you
really
are stupid.
You're right, it doesn't actually have to store the normalized form.
And yes, it's possible to compare without normalizing them.
Admittedly, I don't know much about the implementation details of
unicode, but I would assume that the easiest way to compare two
strings is to normalize them first. But in the case of the filesystem,
normalization actually is important if you're thinking about filenames
in terms of characters rather than bytes. When I feed the filesystem a
given unicode string, it has to find the file I'm talking about -
should it do a relatively expensive unicode-sensitive comparison of
all the filenames with the one I gave it, or should it just normalize
all names and do the much cheaper lookup that way? I don't know about
you, but I'd prefer to let my filesystem normalize the name and run
faster.
The fact is, normalization is wrong.
It's wrong when you normalize upper/lower case (no, the word
"Polish" is
not the same as "polish"), and it's equally wrong when you normalize
for
"looks similar".
There's a difference between "looks similar" as in "Polish" vs
"polish", and actually is the same string as in "Ma<UMLAUT
MODIFIER>rchen" vs "M<A WITH UMLAUT>rchen". Capitalization has a valid
semantic meaning, normalization doesn't. The only way to argue that
normalization is wrong is by providing a good reason to preserve the
exact byte sequence, and so far the only reason I've seen is to help
git. Applications in general don't care one whit about the byte
sequence of the filename, they care about the underlying file the name
represents. Additionally, it would be a terrible experience for a user
to enter "Märchen" and have the application say "sorry, I can't find
this file" simply because the application used decomposed characters
and the filename used composed characters. Unless the user is
knowledgeable about the OS, filesystems, and unicode, they wouldn't
have a hope of figuring out what the problem was.
quoted
In other words, you're used to filenames as bytes, HFS+ treats
filenames
as strings.
No. HFS+ treats users as idiots and thinks that it should "fix" the
filename for them. And it causes problems.
How do you figure? When I type "Märchen", I'm typing a string, not a
byte sequence. I have no control over the normalization of the
characters. Therefore, depending on what program I'm typing the name
in, I might use the same normalization as the filename, or I might
miss. It's completely out of my control. This is why the filesystem
has to step in and say "You composed that character differently, but I
know you were trying to specify this file".
It causes problems for exactly the same reasons case-independence
causes
problems, because it's EXACTLY THE SAME ISSUE. People may think that
"but
they are the same", but they aren't. Case matters. And so does "single
character" vs "two character overlay".
There are valid reasons for case to matter, but what reason is there
for "single character" vs" two character overlay" to matter in
filenames? They're different representations of the exact same string,
and that's what a filename is - a string.
It seems like your arguments stem from the assumption that the user
cares about the byte sequence that represents the filename, which is
wrong. The user has no idea what the byte sequence is - the user cares
about the string. Normalization is meant to help computers, not users,
and claiming that different normalizations of the same string produces
different meaningful strings is complete bunk.
If you were to have two different files on your system, both of them
called "Märchen", but one precomposed and one decomposed, how would
you specify which one you wanted? Unless Linux has a special text
input system which gives the user control over the normalization of
their typed characters, you'd have to write out the UTF-8 bytes
manually.
I just don't understand this insistence on treating the specific byte
sequence that makes up the filename as significant.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:05
On Jan 16, 2008, at 5:23 PM, Johannes Schindelin wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
It's not using different encodings, it's all Unicode.
But that's the _point_! It _is_ Unicode, yet it uses _different_
encodings of the _same_ string.
Now, this discussion gets really annoying. The real question is:
will you
do something about it, or reply with another 500-line email?
I wish I could do something about it. But right now I'm a full-time
student trying to do contracting jobs on the side, and I don't believe
I have the time to learn enough about the guts of git to try and make
any changes to something as core as index filename handling. I just
want people here to recognize that this is a valid problem instead of
simply dismissing it as "HFS+ is insane, lets just ignore this issue".
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
There's a difference between "looks similar" as in "Polish" vs "polish", and
actually is the same string as in "Ma<UMLAUT MODIFIER>rchen" vs "M<A WITH
UMLAUT>rchen". Capitalization has a valid semantic meaning, normalization
doesn't.
That simply isn't true.
Normalization actually has real semantic meaning. If it didn't, there
would never ever be a reason why you'd use the non-normalized form in the
first place.
Others have argued the exact same thing for capitalization. "A" is the
same letter as "a". Except there is a distinction.
The same is true of "a<UMLAUT MODIFIER>" and "<a WITH UMLAUT>". Yes, it's
the same "chacter" in either case. Except when there is a distinction.
And there *are* cases where there are distinctions. Especially inside
computers. For one thing, you may not be talking about "characters on
screen", but you may be talking about "key sequences". And suddenly
"a<UMLAUT MODIFIER>" is a two-key sequence, and "<a WITH UMLAUT>" is a
single-key sequence, and THEY ARE DIFFERENT.
See?
"a" and "A" are the same letter. But sometimes case matters.
Multi-character UTF-8 sequences may be the same character. But sometimes
the sequence matters.
Same exact thing.
The only way to argue that normalization is wrong is by providing a
good reason to preserve the exact byte sequence, and so far the only reason
I've seen is to help git.
Git doesn't care. Just use the *same* sequence everywhere. Make sure
something doesn't change it. Because if something changes it, git will
track it.
How do you figure? When I type "Märchen", I'm typing a string, not a byte
sequence. I have no control over the normalization of the characters.
Therefore, depending on what program I'm typing the name in, I might use the
same normalization as the filename, or I might miss. It's completely out of my
control. This is why the filesystem has to step in and say "You composed that
character differently, but I know you were trying to specify this file".
Pure and utter garbage.
What you are describing is an *input method* issue, not a filesystem
issue.
The fact that you think this has anything what-so-ever to do with
filesystems, I cannot understand.
Here's an example: I can type Märchen two different ways on my keyboard: I
can press the 'ä' key (yes, I have one, I have a Swedish keyboard), or I
could press the '¨' key and the 'a' key.
See: I get 'ä' and 'ä' respectively.
And as I send this email off, those characters never *ever* got written as
filenames to any filesystem. But they *did* get written as part of
text-files to the disk using "write()", yes.
And according to your *insane* logic, that write() call should have
converted them to the same representation, no?
Hell no! That conversion has absolutely nothing to do with the filesystem.
It's done at a totally different layer that actually knows what it is
doing, and turned them both into \xc3\xa4 (and then, the email client
probably will turn this into Latin1, and send it out as a single-byte
'\xe4' character).
See? Putting the conversion in the filesystem IS INSANE. You wouldn't make
the filesystem convert the characters in the data stream (because it would
cause strange data conversion issues) AND FOR EXACTLY THE SAME REASON it
shouldn't do it for filenames either!
And your claim that "you have no control over the normalization of
characters" is simply insane. Of course you have. It's just not supposed
to be at the filesystem level - whether it's a write() call or a creat()
call!
Linus
On Wed, Jan 16, 2008 at 03:39:36PM -0500, Kevin Ballard wrote:
On Jan 16, 2008, at 11:46 AM, Jakub Narebski wrote:
quoted
HFS+ is just _stupid_. And unfortunately Git doesn't support stupid
filesystems (e.g. case insensitive filesystems) well.
There's two different ways to do filesystem encodings. One is to have
the fs simply not care about encoding, which is what the linux world
seems to prefer.
There is no technical reason for *kernel* to care about file name
encoding. It is something that can be and should be dealt with in
the user space (except some special cases like smbfs).
Sure, this is great in that what you create the file
with is what you get back,
And also because a user space program can deal with it much more
gracefully...
but on the other hand, given an arbitrary
non-ASCII file on disk, you have absolutely no idea what the encoding
should be and you can't display it without making assumptions (yes you
can use heuristics, but you're still making assumptions).
Wrong. If you have a policy that all file names are stored in UTF-8
encoding then there is no problem here. It should not be a kernel
problem to care about encoding, besides you cannot fully solve it
in the kernel space anyway...
Filesystems
like HFS+ that standardize the encoding,
Yeah, right... Like Microsoft likes to "standardize" everything, which
in practice means forcing on others something fundamentally broken and
that does not follow any existing standard precisely:
===
IMPORTANT:
The terms used in this Q&A, decomposed and precomposed, roughly
correspond to Unicode Normal Forms D and C, respectively. However, most
volume formats do not follow the exact specification for these normal
forms.
===
http://developer.apple.com/qa/qa2001/qa1173.html
Not to mention that the use of decomposed Unicode as the standard is
outright silly -- no sane person writes in "decomposed" Unicode...
on the other hand, make it
such that you always know what the encoding of a file should be, so
you can always display and use the filename intelligently.
Somehow I have no problem with displaying non-ASCII names on Linux.
I can see both Unicode Normal Forms C and D encoded symbols without
any problem, though the kernel is completely unaware about them.
It also
means it plays much nicer in a non-ASCII world, since you don't have
to worry about different normalizations of a given string referring to
different files (it's one thing to be case-sensitive, but claiming
that "föo" and "föo" are different files
As you typed them, they both are exactly the same, and both of them are
in the Normal Forms C (which Mac calls as precomposed). So why do you
use one encoding in your writings and the other in your file names?
just because one uses a
composed character and the other doesn't is extremely user-
unfriendly). On the other hand, what you create the file with may not
be what you read back later, since the name has been standardized.
It's hard to say one is better than the other, they're just different
ways of doing it. However, I have noticed that everybody who's voiced
an opinion on this list in favor of the encoding-agnostic approach
seem to be unwilling to accept that any other approach might have
validity, to the extent of calling an OS/filesystem that does things
different stupid or insane. This strikes me as extremely elitist and
risks alienating what I expect to be a fast-growing group of users
(i.e. OS X users).
I am sure everyone here is scared to death... I mean we have used to
hear such threats from some MS salespeople, but from a Mac guy? It is
really scare....
Wake up, and stop shooting this nonsense at us. If you have technical
reasons why your solution is better, let us know. So far, you do not
sound very convincing here. Why do think that the issue of encoding can
not be dealt with in the user space? Why does Mac OS X uses so-called
decomposed Unicode, which even does not follow any standard precisely?
Why does Mac OS X chose to decompose characters while it does not
solve any real issue?
And one area that it has a problem with is the de-
facto filesystem on my OS of choice.
I suppose it would be much better a subject for discussion...
At least, it would be more likely to result in that Git working
better on your OS.
However, attempts to discuss the
problem invariable end up with multiple people calling my OS stupid
and insane simply because it differs in a particular design decision.
First, no one called Mac OS X insane, but case insensitive filesystems,
and there are good reasons to think so, because no one has demonstrated
so far any advantage of that approach, but disadvantages are quite
obvious to anyone -- comparison of a stored file list with readdir()
is much more problematic, and you cannot say that you have solved the
problem with encoding if you force other people to *duplicate* some
logic that Mac OS X does in its kernel just to get things working...
So, no one thinks it is insane because it is different, but because it
requires much more efforts to do the same thing -- compare two file
lists, and this operation is important for Git to work properly...
Dmitry
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 11:38 PM, Linus Torvalds wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
The only way to argue that normalization is wrong is by providing a
good reason to preserve the exact byte sequence, and so far the
only reason
I've seen is to help git.
Git doesn't care. Just use the *same* sequence everywhere. Make sure
something doesn't change it. Because if something changes it, git will
track it.
The problem is that you don't control the sequence that everybody uses.
See this example:
melo@speed(~)$ uname -a
Linux speed.simplicidade.org 2.6.9-55.ELsmp #1 SMP Wed May 2 14:28:44
EDT 2007 i686 i686 i386 GNU/Linux
melo@speed(~)$ set | grep LANG
LANG=en_US.UTF-8
melo@speed(~)$ mkdir t
melo@speed(~)$ cd t
melo@speed(~/t)$ git init
Initialized empty Git repository in .git/
melo@speed(~/t)$ touch á
melo@speed(~/t)$ git-add á
melo@speed(~/t)$ git-commit -m "added a in utf8"
Created initial commit 7a473a2: added a in utf8
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 "\303\241"
melo@speed(~/t)$ export LANG=en_US
melo@speed(~/t)$ touch á
melo@speed(~/t)$ ls -la
total 12
drwxrwxr-x 3 melo melo 4096 Jan 16 23:44 .
drwx--x--x 31 melo melo 4096 Jan 16 23:43 ..
-rw-rw-r-- 1 melo melo 0 Jan 16 23:44 á
-rw-rw-r-- 1 melo melo 0 Jan 16 23:43 á
drwxrwxr-x 8 melo melo 4096 Jan 16 23:43 .git
melo@speed(~/t)$ git-add á
melo@speed(~/t)$ git-commit -m "added a in iso-latin-1"
Created commit 4282fca: Oláx!
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 "\341"
So two (simulated in this test) users who use different LANG settings
will be in trouble in no time.
What I take from this conversation is that I have to specify, for
each project I work on, which encoding we should use, across all
users, before they start using git with files with accented chars.
The difference I see between us is that if I tell my filesystem that
I want to name my file with a particular string encoded in X, users
using encoding Y will be able to read it correctly. I like my
filesystem to make that work for me.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 6:38 PM, Linus Torvalds wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
There's a difference between "looks similar" as in "Polish" vs
"polish", and
actually is the same string as in "Ma<UMLAUT MODIFIER>rchen" vs
"M<A WITH
UMLAUT>rchen". Capitalization has a valid semantic meaning,
normalization
doesn't.
That simply isn't true.
Normalization actually has real semantic meaning. If it didn't, there
would never ever be a reason why you'd use the non-normalized form
in the
first place.
My understanding is that normalization is there to help the computer.
That doesn't give it any semantic meaning, because all normal forms of
a given string still represent the exact same string to the user.
Others have argued the exact same thing for capitalization. "A" is the
same letter as "a". Except there is a distinction.
The argument for case insensitivity is different than the argument for
normalization. I certainly hope you understand why they are different
arguments, or there's really no point in going further.
The same is true of "a<UMLAUT MODIFIER>" and "<a WITH UMLAUT>". Yes,
it's
the same "chacter" in either case. Except when there is a distinction.
And there *are* cases where there are distinctions. Especially inside
computers. For one thing, you may not be talking about "characters on
screen", but you may be talking about "key sequences". And suddenly
"a<UMLAUT MODIFIER>" is a two-key sequence, and "<a WITH UMLAUT>" is a
single-key sequence, and THEY ARE DIFFERENT.
See?
"a" and "A" are the same letter. But sometimes case matters.
Multi-character UTF-8 sequences may be the same character. But
sometimes
the sequence matters.
Same exact thing.
You're right, sometimes the sequence matters. As in key sequences. But
we're not talking about key sequences, we're talking about strings.
Just because it matters sometimes doesn't mean it matters all the time.
quoted
The only way to argue that normalization is wrong is by providing a
good reason to preserve the exact byte sequence, and so far the
only reason
I've seen is to help git.
Git doesn't care. Just use the *same* sequence everywhere. Make sure
something doesn't change it. Because if something changes it, git will
track it.
And how am I supposed to use the same sequence everywhere? When I type
"Märchen", I don't know which form I'm typing, nor should I. It's not
something that I, as a user, should have to know. Especially if I pass
this name through various other utilities before using it - I have no
idea if another utility is going to end up normalizing the name, and
it shouldn't matter, as they are equivalent strings.
quoted
How do you figure? When I type "Märchen", I'm typing a string, not
a byte
sequence. I have no control over the normalization of the characters.
Therefore, depending on what program I'm typing the name in, I
might use the
same normalization as the filename, or I might miss. It's
completely out of my
control. This is why the filesystem has to step in and say "You
composed that
character differently, but I know you were trying to specify this
file".
Pure and utter garbage.
What you are describing is an *input method* issue, not a filesystem
issue.
The fact that you think this has anything what-so-ever to do with
filesystems, I cannot understand.
Here's an example: I can type Märchen two different ways on my
keyboard: I
can press the 'ä' key (yes, I have one, I have a Swedish keyboard),
or I
could press the '¨' key and the 'a' key.
See: I get 'ä' and 'ä' respectively.
On a US keyboard I only have one way of typing ä, and I have no idea
whether it ends up precomposed or decomposed in the resulting byte
stream. And I don't care. Because I'm typing characters, not bytes. I
could be typing in a file in ISO-Latin-1 and I still wouldn't care,
because it looks the same to me. If my filesystem did make a
distinction between the normal forms, and I see that I have a file
named "Märchen", how am I supposed to type that at my keyboard? I
don't know which normal form it's using.
The fact that you think the normalization of the string matters, I
don't understand.
And as I send this email off, those characters never *ever* got
written as
filenames to any filesystem. But they *did* get written as part of
text-files to the disk using "write()", yes.
And according to your *insane* logic, that write() call should have
converted them to the same representation, no?
Hell no! That conversion has absolutely nothing to do with the
filesystem.
It's done at a totally different layer that actually knows what it is
doing, and turned them both into \xc3\xa4 (and then, the email client
probably will turn this into Latin1, and send it out as a single-byte
'\xe4' character).
See? Putting the conversion in the filesystem IS INSANE. You
wouldn't make
the filesystem convert the characters in the data stream (because it
would
cause strange data conversion issues) AND FOR EXACTLY THE SAME
REASON it
shouldn't do it for filenames either!
What a fabulous straw man argument you just put together. I hope you
don't need me to point out why this argument is fundamentally flawed.
And your claim that "you have no control over the normalization of
characters" is simply insane. Of course you have. It's just not
supposed
to be at the filesystem level - whether it's a write() call or a
creat()
call!
I'm speaking as a user, and as such, I shouldn't even have to know
that it's possible to write the same character in multiple different
ways. As a user, HFS+ behaves exactly the way I want it to. You were
talking earlier about not messing with the "user data", but what is
the "user data"? It's the string, not the byte sequence. That's all I
care about - the string. That's all the OS cares about, that's all any
application I use cares about, and that's all git should care about.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
The difference I see between us is that if I tell my filesystem that I want to
name my file with a particular string encoded in X, users using encoding Y
will be able to read it correctly. I like my filesystem to make that work for
me.
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*, you don't seem to get it.
An OS that silently changes the contents of your files is *crap*.
Get it?
An OS that silently changes the contents of your directories is *crap*.
Get it now?
Linus
My understanding is that normalization is there to help the computer. That
doesn't give it any semantic meaning, because all normal forms of a given
string still represent the exact same string to the user.
THAT IS NOT TRUE!
How the hell does the computer know what the string means?
Hint: it does not.
The fact is, the user may use a non-normalized string on purpose. It's not
your place to say that the user is wrong. Your "undestanding" is simply
wrong. Two strings are *different* if they are [un]normalized differently.
Really.
The exact same way the word Polish and polish are different, just because
they are capitalized differently.
The argument for case insensitivity is different than the argument for
normalization. I certainly hope you understand why they are different
arguments, or there's really no point in going further.
You do not understand.
In *order* to do case-insensitivity, you generally need to normalize (and
do other things too - normalization is just *one* of the things you need
to do).
So if you are a case-insensitive filesystem, then normalization is sane.
But if you aren't, then there is no reason to normalize.
You're right, sometimes the sequence matters. As in key sequences. But we're
not talking about key sequences, we're talking about strings.
You define "string" to be something totally made-up.
In your world "string" means "normalized". BUT IT'S NOT TRUE!
You define normalization to be a property of strings, without any actual
backing for why that would be.
The fact is, *looks the same* is very very different from *is the same*.
But you seem to be too stupid to undestand the differce.
Linus
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 12:16 AM, Linus Torvalds wrote:
On Wed, 16 Jan 2008, Pedro Melo wrote:
quoted
The difference I see between us is that if I tell my filesystem
that I want to
name my file with a particular string encoded in X, users using
encoding Y
will be able to read it correctly. I like my filesystem to make
that work for
me.
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*, you don't seem to
get it.
I get that you think its the same thing.
What I don't get is why a user should be forced to know what type of
encoding he and the other users are using on all the layers going
down to the filesystem. If two users on different systems or in
different configurations, choose the same unicode string as the name,
why do we need to make it harder for things to just work out?
The content of the file is sacred, we both agree on that. We disagree
on the filename, because for me it's more important that equal
strings, even if encoded to different byte sequences, should be
treated as the same file.
An OS that silently changes the contents of your files is *crap*.
Get it?
I was not talking about content of files, those are sacred. I was
talking about filenames. Those *for me* are not, but are for you. No
problem, we just have different values: I want my computer to work
for me, not me working for the computer. I'm willing to accept a file
system or other layer that normalizes encoding of filenames if that
makes the end-user life easier, specially in a tool distributed by
nature.
An OS that silently changes the contents of your directories is
*crap*.
Get it now?
As I said before, we disagree on file meta-data, not on file
contents. For you, byte in must be the same byte out. For me string
in must be the same string out.
And as I said in the previous email, what I learned today is that in
a distributed project using git, and if you need to use accented
characters, I need to tell all the users to use the same LANG settings.
It's important information, at least for me.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Wed, 16 Jan 2008, Linus Torvalds wrote:
So if you are a case-insensitive filesystem, then normalization is sane.
Actually, no. Even an case-challenged filesystem should keep the
_original_ name around, if only for the exact same argument you used
earlier: if the user chooses to capitalise some letters, but not others,
it is not the filesystem's place to "correct" that.
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Thu, 17 Jan 2008, Pedro Melo wrote:
The content of the file is sacred, we both agree on that. We disagree on
the filename, because for me it's more important that equal strings,
even if encoded to different byte sequences, should be treated as the
same file.
Why should the filename be _stored_ normalised? I agree on the lookup,
yes, but not the storage.
Hth,
Dscho
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 12:33 AM, Johannes Schindelin wrote:
On Wed, 16 Jan 2008, Linus Torvalds wrote:
quoted
So if you are a case-insensitive filesystem, then normalization is
sane.
Actually, no. Even an case-challenged filesystem should keep the
_original_ name around, if only for the exact same argument you used
earlier: if the user chooses to capitalise some letters, but not
others,
it is not the filesystem's place to "correct" that.
For the record, HFS+ is case-insensitive but case-preserving so I
believe they keep the original filename around. I don't have the spec
in front of me, but from memory I believe that this is what they do.
But I think that focusing on HFS+ is loosing sight of the real
problem. It's not about encoding at the filesystem, but encoding
inside the git structures.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008, at 12:35 AM, Johannes Schindelin wrote:
On Thu, 17 Jan 2008, Pedro Melo wrote:
quoted
The content of the file is sacred, we both agree on that. We
disagree on
the filename, because for me it's more important that equal strings,
even if encoded to different byte sequences, should be treated as the
same file.
Why should the filename be _stored_ normalised? I agree on the
lookup,
yes, but not the storage.
Personally I don't care how you store it. It's an implementation
detail, and you should choose the best one for your use cases. If
that means that you store the original version and a normalized
version just for lookups, fine.
What I think its important is that if two users use different
encodings for the same string in a filename, git should treat that as
the same file.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Thu, 17 Jan 2008, Pedro Melo wrote:
On Jan 17, 2008, at 12:33 AM, Johannes Schindelin wrote:
quoted
On Wed, 16 Jan 2008, Linus Torvalds wrote:
quoted
So if you are a case-insensitive filesystem, then normalization is
sane.
Actually, no. Even an case-challenged filesystem should keep the
_original_ name around, if only for the exact same argument you used
earlier: if the user chooses to capitalise some letters, but not
others, it is not the filesystem's place to "correct" that.
For the record, HFS+ is case-insensitive but case-preserving so I
believe they keep the original filename around.
For the record, that's only the default setting. AFAIK you can configure
it to care about case, too.
Also for the record, the whole thread was about HFS+ _not_ keeping the
original filename around, but _only_ a normalised version of it.
But I think that focusing on HFS+ is loosing sight of the real problem.
It's not about encoding at the filesystem, but encoding inside the git
structures.
So far I have not seen anyone talking _seriously_ about this issue. Only
a few shouts "you should support", and a few shouts back "I don't care
about insane filesystems".
Therefore, I fully agree with you that we're losing sight of the real
problem.
Ciao,
Dscho
I'm speaking as a user, and as such, I shouldn't even have to know that it's
possible to write the same character in multiple different ways.
The thing is, you seem to argue that what OS X does helps you as the user.
But you are arguing based on incorrect assumptions.
First off, we've had years and years and years of usage of non-corrupting
filesystems (pretty much every UNIX OS around since day 1, and many other
OS's too), and it's simply not true that it's a problem. You see the
filename in the file dialog, and you open it, and you're done. OS X isn't
any "easier" in this regard.
In fact, this whole thread comes from the fact that the OS X choice that
you *think* is easier, is in fact not easier at all. It's not easier for
the user, it's not easier for the application programmer, and the really
sad part is that it's very much *not* easier for OS X itself either (ie
they had to literally write extra code with nasty tables to do it, and it
really does hurt them in performance and complexity).
And _that_ is why the OS X situation is so sad. Apple literally added
extra code to make things slower and more complex *and* harder to use
reliably.
Does it show up in normal behaviour? Of course not. You'd probably never
see it in real life outside of test-suites. People simply don't even tend
to use filenames outside of US-ASCII, and when they do use them, input
methods really *do* tend to do the normalization for you.
But when it comes to automation (which is what computers are all about),
the OS X choice is literally the wrong one. And there's no _upside_. It's
all downside. Which is why it's so stupid.
I bet it only exists because OS X engineers didn't really even think about
it, and they just assumed that "normalization is helpful". They took your
stance - thinking it was worth it, without ever really thinking it
through.
Linus
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 8:16 PM, Linus Torvalds <torvalds@linux-foundation.org
> wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
I'm speaking as a user, and as such, I shouldn't even have to know
that it's
possible to write the same character in multiple different ways.
The thing is, you seem to argue that what OS X does helps you as the
user.
But you are arguing based on incorrect assumptions.
First off, we've had years and years and years of usage of non-
corrupting
filesystems (pretty much every UNIX OS around since day 1, and many
other
OS's too), and it's simply not true that it's a problem. You see the
filename in the file dialog, and you open it, and you're done. OS X
isn't
any "easier" in this regard.
In fact, this whole thread comes from the fact that the OS X choice
that
you *think* is easier, is in fact not easier at all. It's not easier
for
the user, it's not easier for the application programmer, and the
really
sad part is that it's very much *not* easier for OS X itself either
(ie
they had to literally write extra code with nasty tables to do it,
and it
really does hurt them in performance and complexity).
And _that_ is why the OS X situation is so sad. Apple literally added
extra code to make things slower and more complex *and* harder to use
reliably.
Does it show up in normal behaviour? Of course not. You'd probably
never
see it in real life outside of test-suites. People simply don't even
tend
to use filenames outside of US-ASCII, and when they do use them, input
methods really *do* tend to do the normalization for you.
But when it comes to automation (which is what computers are all
about),
the OS X choice is literally the wrong one. And there's no _upside_.
It's
all downside. Which is why it's so stupid.
I bet it only exists because OS X engineers didn't really even think
about
it, and they just assumed that "normalization is helpful". They took
your
stance - thinking it was worth it, without ever really thinking it
through.
Linus
I believe it exists because HFS+ was created at a time when the Mac
was moving from a multi-encoding world (which was a nightmare) to a
Unicode world and they wanted to remove ambiguity in filenames. But I
wasn't around when they made this decision so this is just a guess.
-Kevin Ballard
I believe it exists because HFS+ was created at a time when the Mac was moving
from a multi-encoding world (which was a nightmare) to a Unicode world and
they wanted to remove ambiguity in filenames. But I wasn't around when they
made this decision so this is just a guess.
I do agree. And I think starting out case-insensitive (something they must
really hate by now) also made it less of an issue. When you're
case-insensitive, the issues with any UTF-8 normalization are simply
swamped by all the issues of case, so you probably don't even think about
it very much.
The big problem with any name rewriting is that I can open file 'xyz', and
I literally have a very hard time knowing whether that file I know I
opened and created has anything to do with the file 'Xyz' that I see when
I do a readdir().
Are they the same? Maybe. But it's literally hard to tell on OS X. I can
do an fstat() on my file descriptor and on the directory entry, and if
they get the same d_ino they *probably are the same entry, but even then
it actually could have been a hardlink (and my 'xyz' is really *another*
name for it entirely, and the filesystem is actually case-sensitive and
'Xyz' was a *different* name that somebody else did!).
See? If you're creating a content tracker, these kinds of issues are not
"idle chatter". It's really *really* important. Was that file the one I
was told to track? Or was it a temporary file that was just hardlinked?
This is why case-insensitivity is so hard: you have a very real "aliasing"
on the filesystem level, where all those really *different* pathnames end
up being the same thing.
And all the same issues show up with utf-8 rewriting, so if you normalize
utf-8 names, you actually end up having almost all the same problems that
a case-insensitive filesystem has. They're just much rarer in practice, so
you just won't hit them as often - but when you do, they are equally
painful!
(In fact, they can be a whole lot *more* painful, because now they are
really rare, and really confusing when they happen!)
But if you come from a case-insensitive background, all the UTF-8
rewriting really looks like such a small problem compared to all the
horrid problems that you had with different locales and cases, so I
suspect they didn't even realize what a big mistake they did!
Linus
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 11:08 PM, Linus Torvalds wrote:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
I believe it exists because HFS+ was created at a time when the Mac
was moving
from a multi-encoding world (which was a nightmare) to a Unicode
world and
they wanted to remove ambiguity in filenames. But I wasn't around
when they
made this decision so this is just a guess.
I do agree. And I think starting out case-insensitive (something
they must
really hate by now) also made it less of an issue. When you're
case-insensitive, the issues with any UTF-8 normalization are simply
swamped by all the issues of case, so you probably don't even think
about
it very much.
Those of us who grew up on a case-insensitive filesystem don't find
there to be any problem with it. I can count on one hand the number of
times I've run into a problem caused by a case-insensitive filesystem.
That number is 1. And that 1 time is when git screwed up trying to
track CS4536 and cs4536 in the same directory (see earlier thread).
The big problem with any name rewriting is that I can open file
'xyz', and
I literally have a very hard time knowing whether that file I know I
opened and created has anything to do with the file 'Xyz' that I see
when
I do a readdir().
That's only true if you don't know what type of filesystem you're on.
And, in the vast majority of cases (in fact, a content tracker is the
only exception I can think of), it doesn't matter. If the user said
'xyz' and you can stat() it, great, that's what the user wanted! Just
because it's really called 'Xyz' on the filesystem doesn't make any
difference.
Are they the same? Maybe. But it's literally hard to tell on OS X. I
can
do an fstat() on my file descriptor and on the directory entry, and if
they get the same d_ino they *probably are the same entry, but even
then
it actually could have been a hardlink (and my 'xyz' is really
*another*
name for it entirely, and the filesystem is actually case-sensitive
and
'Xyz' was a *different* name that somebody else did!).
See? If you're creating a content tracker, these kinds of issues are
not
"idle chatter". It's really *really* important. Was that file the
one I
was told to track? Or was it a temporary file that was just
hardlinked?
But git is a content tracker, so even if it's really a different
hardlink that shouldn't matter, it's still referencing the same
content. Go ahead and track whatever name the user specified
originally, as long as it maps to a file on disk with the expected
content you're set. If the file is really called 'foo' and I told git
to track 'Foo', I'm perfectly happy with it continuing to think 'foo'
is 'Foo' until I use 'git mv Foo foo'.
This is why case-insensitivity is so hard: you have a very real
"aliasing"
on the filesystem level, where all those really *different*
pathnames end
up being the same thing.
I don't see that as being a problem. Think of it, if you will, as if
every single file simply had an implicit hardlink for every possible
case or normalization variant. The whole point of the filename is that
it is meta-information, used as an identifier and not as actual
content, and thus it is perfectly fine for it to be a real string,
subject to interpretation, rather than treated as a sacred binary blob
like content is. The whole purpose of the name is to identify the
inode in question, and case and normalization aren't particularly
relevant here. As long as we can identify the file, we're happy.
And all the same issues show up with utf-8 rewriting, so if you
normalize
utf-8 names, you actually end up having almost all the same problems
that
a case-insensitive filesystem has. They're just much rarer in
practice, so
you just won't hit them as often - but when you do, they are equally
painful!
(In fact, they can be a whole lot *more* painful, because now they are
really rare, and really confusing when they happen!)
But if you come from a case-insensitive background, all the UTF-8
rewriting really looks like such a small problem compared to all the
horrid problems that you had with different locales and cases, so I
suspect they didn't even realize what a big mistake they did!
Again, as someone who grew up in a case-insensitive world, there's no
problems here. I wish I could tell you that it causes problems, I wish
I could agree with you, but I can't.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Martin Langhoff <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008 5:30 PM, Kevin Ballard [off-list ref] wrote:
Those of us who grew up on a case-insensitive filesystem don't find
there to be any problem with it. I can count on one hand the number of
I guess you haven't used unix tools much. The ever-popular HEAD perl
utility (which does an HTTP HEAD against a URL), when installed,
silently overwrites the head shell utility, which is used for all
sorts of things, some even in startup scripts. Ooops! I've been hit by
this more than once - and if you google for it, it hurt a lot of
people.
That's only true if you don't know what type of filesystem you're on.
And, in the vast majority of cases (in fact, a content tracker is the
only exception I can think of), it doesn't matter. If the user said
Hmmm. Many important tools - that I wouldn't want to ever fail! - have
similar needs to git. Backup/restore and file replication tools for
example.
quoted
This is why case-insensitivity is so hard: you have a very real
"aliasing"
on the filesystem level, where all those really *different*
pathnames end
up being the same thing.
I don't see that as being a problem. Think of it, if you will, as if
every single file simply had an implicit hardlink for every possible
case or normalization variant. The whole point of the filename is that
Ok - but how do you track the directory then (in git's terms, the
tree). There's no way to tell what the user wants. Does the user want
a copy of the file with different capitalization, or is the OS playing
games?
it is meta-information, used as an identifier and not as actual
content, and thus it is perfectly fine for it to be a real string,
subject to interpretation,
I don't think you *actually* want it subject to interpretation.
Again, as someone who grew up in a case-insensitive world, there's no
problems here. I wish I could tell you that it causes problems, I wish
I could agree with you, but I can't.
Probably because you have been surrounded by tools that have a lot of
extra code to cope with the case insensitive way of life, and learned
to not do things that are completely valid, just to avoid trouble.
Which is ok, but I don't think it makes the OS design decision
defensible.
cheers,
m
From: Jay Soffian <hidden> Date: 2016-06-15 22:44:06
So here's what I can see as being useful additions to git:
* Allowing a repo to be *optionally* configured to disallow two files
in a directory that can cause aliasing problems, with options for
unicode normalization aliasing and/or case-insensitivity aliasing. Can
this already be done via hooks and someone just needs to write the
appropriate hooks?
* Having git warn during checkout if there are files which alias in
the working copy filesystem. I guess it might be interesting if there
were a mechanism in this situation for telling git which of the
aliases you want checked out, though that doesn't seem like a very
good feature.
Thoughts (besides "patches welcomed")?
j.
FWIW, here's Sun's take on the issue of filesystems and i18n:
Pretty sane, from a quick read-through, although most of it seems to not
be about general issue, as about "let's emulate others correctly on their
filesystems" (ie the rules are different for NTFS and HFS+, little enough
discussion about "native" preferred logic).
However, while they don't consider normalization on file creates to be the
"preferred solution", they *do* consider filename comparison with
canonical equivalence to be that. Which means that you can get the same
odd problems:
fd = open(filename, O_CREAT);
+
readdir()
can actually return a *different* filename than the one we just created,
if it already existed in the directory under the different normalization.
So it's basically "normalization-preserving, but normalization-ignoring"
(the same way many filesystems are case-preserving, but case-ignoring). I
don't much like it either, but as with case, the "preserving" behaviour is
probably the nicer one.
I'd guess the problems are harder to trigger in practice, but you can
still get some pretty hairy cases. It's just painful when readdir() and
your own file creation doesn't have any obvious 1:1 relationship.
Linus
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 11:51 PM, Martin Langhoff wrote:
On Jan 17, 2008 5:30 PM, Kevin Ballard [off-list ref] wrote:
quoted
Those of us who grew up on a case-insensitive filesystem don't find
there to be any problem with it. I can count on one hand the number
of
I guess you haven't used unix tools much. The ever-popular HEAD perl
utility (which does an HTTP HEAD against a URL), when installed,
silently overwrites the head shell utility, which is used for all
sorts of things, some even in startup scripts. Ooops! I've been hit by
this more than once - and if you google for it, it hurt a lot of
people.
I can imagine. However, I've never been hit by such a situation. This
doesn't mean a case-insensitive filesystem is a problem per se, it
means interactions between a case-insensitive and a case-sensitive
filesystem can be a problem. That doesn't mean either way is "correct"
it just means both don't work well together.
I like ice cream, and I like steak, but I sure don't think a mixture
of steak and ice cream would go well together. Do you?
quoted
That's only true if you don't know what type of filesystem you're on.
And, in the vast majority of cases (in fact, a content tracker is the
only exception I can think of), it doesn't matter. If the user said
Hmmm. Many important tools - that I wouldn't want to ever fail! - have
similar needs to git. Backup/restore and file replication tools for
example.
Both of which would be replicating the directory contents, not a
listing of files specified by the user. If, as a user, I were to say
"please replicate file FOO" and the file was really called "foo", I
wouldn't be in the least surprised to see the tool take me at my word
and produce a file called "FOO" with the contents of "foo". But in
general, things like this operate on the filesystem, not on the user
args.
quoted
quoted
This is why case-insensitivity is so hard: you have a very real
"aliasing"
on the filesystem level, where all those really *different*
pathnames end
up being the same thing.
I don't see that as being a problem. Think of it, if you will, as if
every single file simply had an implicit hardlink for every possible
case or normalization variant. The whole point of the filename is
that
Ok - but how do you track the directory then (in git's terms, the
tree). There's no way to tell what the user wants. Does the user want
a copy of the file with different capitalization, or is the OS playing
games?
If I say "track FOO", I probably mean it. So go ahead and track "FOO",
even if you end up tracking the contents of file "foo". I certainly
won't blame the tool for doing what I told it.
quoted
it is meta-information, used as an identifier and not as actual
content, and thus it is perfectly fine for it to be a real string,
subject to interpretation,
I don't think you *actually* want it subject to interpretation.
Sure I do. I find it very convenient, for example, to say "cd
documents/school" when I really want to go to "Documents/School".
Similarly, if I'm trying to reference gitweb/tests/Märchen, I'm quite
happy to not have to figure out what normalization the filename is
using and attempt to replicate that (especially as I have no idea
which normalization my input mechanism uses - unlike Linus, I don't
have a key dedicated to ä, and even if I did I wouldn't necessarily
expect it to use precomposed vs decomposed). I can't think of a single
reason why I'd want to be able to have 2 different files named
"Märchen" on my disk. On the other hand, treating unicode
normalization as significant can pose security risks - how am I to
know that the file that is named "foo.txt" is really the same file
"foo.txt" that I last saw? Someone I know on IRC sent me this
image[1], which shows 6 files all apparently named "foo.txt" on a disk
image. This is possible because on a case-sensitive HFS+ volume, the
file system doesn't ignore ignorables when comparing filenames (it
does on a case-insensitive HFS+ system), and so all of those filenames
look identical up until you actually pipe their names through xxd and
look at the byte sequence. When this sort of tomfoolery is possible, I
simply cannot trust the names of any of my files anymore.
[1]: http://sailor月.com/imgs/ignorable.png
quoted
Again, as someone who grew up in a case-insensitive world, there's no
problems here. I wish I could tell you that it causes problems, I
wish
I could agree with you, but I can't.
Probably because you have been surrounded by tools that have a lot of
extra code to cope with the case insensitive way of life, and learned
to not do things that are completely valid, just to avoid trouble.
Which is ok, but I don't think it makes the OS design decision
Extra code? I don't think so. The only reason I'd need extra code is
if I were attempting to explicitly detect the "real" filename for a
user-supplied argument, by scanning the directory contents until I
found a file that was equivalent to the given argument. But there's no
reason to do that. None of the code I've ever written, or any of the
code I've ever seen, has had to do any extra work because it was on a
case-insensitive filesystem. I contribute to a packaging system for
the Mac called MacPorts, and I've never seen any patches on any of the
4000+ ports to handle case insensitivity (granted, I haven't looked at
every port, but I've looked at a significant fraction). It's a
complete non-issue.
The content of files is sacred. The filename is only there to provide
a handle to locate the contents. I don't see any problem with
expanding the equivalency scope of the filename to accept multiple
encodings and cases. The only arguments I can see that have any
validity at all are the ones that sound like "we use case-sensitive
filesystems, and your case-insensitivity and normalization are causing
problems with our tools! Conform to our world!". As I said above, this
isn't a problem of case-insensitivity or normalization, it's a problem
of interaction between two incompatible viewpoints. All I want to do
is make git play nicer in an HFS+ world, and this would be far easier
if you guys were willing to admit this is a problem that should be
solved in the tool rather than a problem with the system.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
For those on Mac OS X: it is possible to create a case-sensitive HFS+
partition and
use it with git. You even can just create a disk image and mount it.
However,
I wouldn't quite try to use it as startup filesystem...
-Geert
PS. I'm working on a proposal/patch for addressing the UFS/case
sensitivity issues.
Will try to mail something later this week.
I was going to post this earlier, but wanted to search the archives
first. Here are the commands assuming you don't want to or can't
partition a drive and format as ufs (I don't care for HFS+ much). I
can't believe I didn't find the command in the git list archives, so
voilà:
$ hdiutil create -size 300m -fs UFS foo.dmg
...............................................................................
created: /Users/mitch/foo.dmg
$ hdiutil attach foo.dmg
/dev/disk2 GUID_partition_scheme
/dev/disk2s1 Apple_UFS /Volumes/untitled
$ cd /Volumes/untitled && git clone git://git.kernel.org/pub/scm/git/
git.git
... snipped ...
$ cd git && git status
# On branch master
nothing to commit (working directory clean)
After git clone in HFS+ land...
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# gitweb/test/MaÌrchen
nothing added to commit but untracked files present (use "git add" to
track)
Should I just add this to the wiki? Then we can all go back to
ignoring the insane filesystems.
Mitch
On Jan 17, 2008, at 12:13 AM, Geert Bosch wrote:
For those on Mac OS X: it is possible to create a case-sensitive HFS
+ partition and
use it with git. You even can just create a disk image and mount it.
However,
I wouldn't quite try to use it as startup filesystem...
-Geert
PS. I'm working on a proposal/patch for addressing the UFS/case
sensitivity issues.
Will try to mail something later this week.
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
El 17/1/2008, a las 5:08, Linus Torvalds escribió:
On Wed, 16 Jan 2008, Kevin Ballard wrote:
quoted
I believe it exists because HFS+ was created at a time when the Mac
was moving
from a multi-encoding world (which was a nightmare) to a Unicode
world and
they wanted to remove ambiguity in filenames. But I wasn't around
when they
made this decision so this is just a guess.
I do agree. And I think starting out case-insensitive (something
they must
really hate by now) also made it less of an issue.
I hope you're right (about them hating it), but we'll see. They've
just opened the source for the ZFS port they're working on. By the
time it goes final and becomes the default FS, replacing HFS+,
probably within a couple of years, we'll see if they make the same two
design decisions which cause the kinds of problems being discussed
here (case-insensitivity, and ubiquitous FS-level UTF-8 normalization).
I've done a dumb search in the ZFS source code for "CASE" and see that
it can in theory support case-insensitivity as an optional feature.
The potential is there for Apple to use this. I personally hope that
they don't, because as has already been pointed out, these little
tricks tend to make life more difficult for users rather than helping
them (the day I have two files in the same directory called "Märchen"
and want to specify one of them on the command line I'll worry about
that when I come to it).
http://fuzzy.wordpress.com/2007/06/09/zfsandfilesystemoptions/
Cheers,
Wincent
El 17/1/2008, a las 8:11, Mitch Tishmack escribió:
I was going to post this earlier, but wanted to search the archives
first. Here are the commands assuming you don't want to or can't
partition a drive and format as ufs (I don't care for HFS+ much). I
can't believe I didn't find the command in the git list archives, so
voilà:
$ hdiutil create -size 300m -fs UFS foo.dmg
...............................................................................
created: /Users/mitch/foo.dmg
$ hdiutil attach foo.dmg
/dev/disk2 GUID_partition_scheme
/dev/disk2s1 Apple_UFS /Volumes/untitled
$ cd /Volumes/untitled && git clone git://git.kernel.org/pub/scm/git/
git.git
... snipped ...
$ cd git && git status
# On branch master
nothing to commit (working directory clean)
After git clone in HFS+ land...
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add"
to track)
Should I just add this to the wiki?
Definitely.
Then we can all go back to ignoring the insane filesystems.
While it's a nice workaround, it really is just that (a workaround)
because performance will be suboptimal in a repository running on a
disk image (and many of switched to Git because of its speed).
Cheers,
Wincent
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008, at 5:22 AM, Wincent Colaiuta wrote:
El 17/1/2008, a las 8:11, Mitch Tishmack escribió:
quoted
I was going to post this earlier, but wanted to search the archives
first. Here are the commands assuming you don't want to or can't
partition a drive and format as ufs (I don't care for HFS+ much). I
can't believe I didn't find the command in the git list archives,
so voilà:
$ hdiutil create -size 300m -fs UFS foo.dmg
...............................................................................
created: /Users/mitch/foo.dmg
$ hdiutil attach foo.dmg
/dev/disk2 GUID_partition_scheme
/dev/disk2s1 Apple_UFS /Volumes/
untitled
$ cd /Volumes/untitled && git clone git://git.kernel.org/pub/scm/
git/git.git
... snipped ...
$ cd git && git status
# On branch master
nothing to commit (working directory clean)
After git clone in HFS+ land...
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add"
to track)
Should I just add this to the wiki?
Definitely.
quoted
Then we can all go back to ignoring the insane filesystems.
While it's a nice workaround, it really is just that (a workaround)
because performance will be suboptimal in a repository running on a
disk image (and many of switched to Git because of its speed).
Not only is it suboptimal, it's also not acceptable, plain and simple.
If an individual wants to do that, sure, but it's simply not an
appropriate solution in general for this problem. I certainly don't
want to have to attach a disk image every time I want access to
anything I keep in a git repo, nor do I want to be restricted to
keeping everything within a certain filesystem on disk. Additionally,
while I'm not certain it's impossible, it's certainly very difficult
to attach a disk image without anybody logged into the system at the
GUI, as diskarbitrationd won't be running.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008, at 9:02 AM, Andrew Heybey wrote:
Geert Bosch [off-list ref] writes:
quoted
For those on Mac OS X: it is possible to create a case-sensitive HFS+
partition and
use it with git. You even can just create a disk image and mount it.
However,
I wouldn't quite try to use it as startup filesystem...
This is starting to stray far afield, but the first thing I did when I
got a Macbook was to reinstall it with case-sensitive HFS as the boot
file system. Works fine, including with git. The only problem I have
had is that FileVault does not work. There are rumored to be some
third-part apps that do not work but I do not use that many of those
anyway.
andrew
The main problem with this approach is you know for certain that using
HFSX as the boot partition is barely tested by Apple, and certainly
untested by third-party apps. This means the potential for breakage is
extremely high.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Thu, 17 Jan 2008, Kevin Ballard wrote:
On Jan 17, 2008, at 5:22 AM, Wincent Colaiuta wrote:
quoted
While it's a nice workaround, it really is just that (a workaround)
because performance will be suboptimal in a repository running on a
disk image (and many of switched to Git because of its speed).
Not only is it suboptimal, it's also not acceptable, plain and simple.
If it's not acceptable, do something about it (and I don't mean writing 50
emails). If you don't want to do something about it, I have to assume that
you accept it as-is.
Ciao,
Dscho
(the day I have two files in the same directory called "Märchen" and
want to specify one of them on the command line I'll worry about that
when I come to it).
Side note: the thing is, the reason people shouldn't worry about it is
that this is a *trivial* thing to handle. You really don't even need to
know what you're doing. And you can test it today, easily.
Having two (differently encoded) files like that is really no different
from the traditional UNIX FAQ of "how do I remove a file starting with
'-'" or even more closely "how do I remove a file that has a character in
it that I cannot get at the keyboard".
In other words, on a bog-standard UNIX (and yes, in this case, I bet OS X
works fine too for this test), just try this
filename1=$(echo -e "hello\002there")
filename2=$(echo -e "hello\003there")
echo Odd file > "$filename1"
echo Another odd file > "$filename2"
and now you have a filename that is actually rather hard to type on the
command line. In fact, for me they even *look* the same:
[torvalds@woody ~]$ ll hello*
-rw-rw-r-- 1 torvalds torvalds 9 2008-01-17 08:23 hello?there
-rw-rw-r-- 1 torvalds torvalds 17 2008-01-17 08:23 hello?there
See?
Even in my graphical browser, those two filenames look 100% *identical*. I
could give you a screen-shot, but I'm lazy. Just take my word for it, or
just fire up konqueror on Linux (but it may well depend on the particular
font you're using).
[ And yes, for other browsers, you might have something that shows them as
different characters - depending on the font, it might show up as a
small box with [00 02] vs [00 03] in it, for example. But that's also
actually 100% true of the two different encodings of 'ä' - you could
easily have a file broswer that shows the multi-character as a
multi-character, exactly to distinguish them and show that one of them
isn't "normalized"!
The point is, once the filesystem doesn't corrupt the data, it's always
easy to get at, and there is never any ambiguity. ]
How is this different from "Märchen" spelled with two different encodings
for that "ä"?
I'll tell you: it's not at all different. It's 100% the exact same issue.
And does that make you perhaps go "Hunh? How do I remove it, or open it?"
And the fact is, those "idential looking" filenames (and thus they must be
the same, and something should have normalized them to the same thing,
no?) are obviously two different files, and they are *really*easy* to edit
and look at.
Fire up that graphical browser again, and it doesn't even matter whether
the filename looks identical or not, it shows up as two different files,
and you can drag them around independently, rename them there, and at
least my file browser shows clearly which is which, because I get a small
icon with a preview in it, so I directly see which one is the "Odd file"
and which one is the "Another odd file".
So the whole "but they _look_ the same" argument is just total BS. In just
about all character encodings there has always been unique and different
"characters" that _look_ the same on screen, and it has never really made
them actually *be* the same, and it has never been a valid argument for
them being considered the same.
Because even when they *look* the same, that file browser that didn't show
the difference in names visually, still showed them correctly as two
separate files, and I could still just rename them by hand by
right-clicking on them and picking "rename".
So "look the same" is really not a new thing, nor is it even a really hard
thing. Yes, people can get confused by it, but hey, people can get
confused by *anything*. People get confused by filenames starting with a
"-", yet nobody sane really says that filenames cannot start with a dash.
Linus
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008, at 10:57 AM, Johannes Schindelin wrote:
On Thu, 17 Jan 2008, Kevin Ballard wrote:
quoted
On Jan 17, 2008, at 5:22 AM, Wincent Colaiuta wrote:
quoted
While it's a nice workaround, it really is just that (a workaround)
because performance will be suboptimal in a repository running on a
disk image (and many of switched to Git because of its speed).
Not only is it suboptimal, it's also not acceptable, plain and
simple.
If it's not acceptable, do something about it (and I don't mean
writing 50
emails). If you don't want to do something about it, I have to
assume that
you accept it as-is.
I never said I don't want to do anything about it. However, I do
believe that it will take a significant investment of time and energy
to learn all the gooey details of how git handles filenames and how
the index works and all that jazz, which is knowledge that other
people already have. I believe that, for me to solve this problem
independently, it may require so much time that it never gets done
(after all, I am fairly busy). However, if other people who already
have this knowledge are willing to help, that would make this task far
easier, especially given that if nobody else even acknowledges that
this is a problem I don't have much hope of getting a patch accepted.
So again, I'm certainly going to try, but working by myself it simply
may never get done.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
From: Mark Junker <hidden> Date: 2016-06-15 22:44:06
Linus Torvalds schrieb:
In other words, on a bog-standard UNIX (and yes, in this case, I bet OS X
works fine too for this test), just try this
filename1=$(echo -e "hello\002there")
filename2=$(echo -e "hello\003there")
echo Odd file > "$filename1"
echo Another odd file > "$filename2"
and now you have a filename that is actually rather hard to type on the
command line. In fact, for me they even *look* the same:
[torvalds@woody ~]$ ll hello*
-rw-rw-r-- 1 torvalds torvalds 9 2008-01-17 08:23 hello?there
-rw-rw-r-- 1 torvalds torvalds 17 2008-01-17 08:23 hello?there
See?
Sorry, but you're using different characters that look the same. But
Kevins point was that it's a different thing if you use two characters
that look the same or the same character with different encodings. This
makes this HFS-specific problem different from the "look the same"- or
the "case-insensitivity"-issues.
BTW: I also read about your argument that you wouldn't convert file data
to normalized UTF-8 (I agree with you that this would be nonsense) and
therefore filenames shouldn't be converted too. This is something where
I have to disagree because a filename (like ctime, mtime, atime, ...)
are meta data (while file contents isn't) and - until now - I would've
guessed that you agree on this point because git doesn't care about
filenames but contents.
IMHO it would be the best solution when git stores all string meta data
in UTF-8 and converts it to the target systems file system encoding.
That would fix all those problems with different locales and file system
encodings ...
However, I have to agree that the enforced character set conversion
causes more problems than it solves.
Regards,
Mark
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 6:09 PM, Mark Junker wrote:
Linus Torvalds schrieb:
IMHO it would be the best solution when git stores all string meta
data in UTF-8 and converts it to the target systems file system
encoding. That would fix all those problems with different locales
and file system encodings ...
+1.
And I would suggest the use of RFC 3454 as the guidelines for UTF-8
normalization.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Thu, 17 Jan 2008, Pedro Melo wrote:
On Jan 17, 2008, at 6:09 PM, Mark Junker wrote:
quoted
IMHO it would be the best solution when git stores all string meta
data in UTF-8 and converts it to the target systems file system
encoding. That would fix all those problems with different locales and
file system encodings ...
+1.
-1.
It's just too arrogant to force your particular preferences down the
throat of every git user.
Ciao,
Dscho
From: Mark Junker <hidden> Date: 2016-06-15 22:44:06
Johannes Schindelin schrieb:
It's just too arrogant to force your particular preferences down the
throat of every git user.
It's not arrogant to make a suggestion. Where is your alternative solution?
However, what about storing an additional information like the file
system encoding (for every file)? This would result in the same
behaviour (and speed) as today as long as the file system encoding is
the same. Conversion will only be done when the targets file system
encoding is different.
BTW: This reminds me of the code page switching stuff back in the times
of MS-DOS 4/5. This really wasn't funny.
Regards,
Mark
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 6:18 PM, Johannes Schindelin wrote:
On Thu, 17 Jan 2008, Pedro Melo wrote:
quoted
On Jan 17, 2008, at 6:09 PM, Mark Junker wrote:
quoted
IMHO it would be the best solution when git stores all string meta
data in UTF-8 and converts it to the target systems file system
encoding. That would fix all those problems with different
locales and
file system encodings ...
+1.
-1.
It's just too arrogant to force your particular preferences down the
throat of every git user.
Do you agree that you need to store or at least calculate a
normalized version of each filename to see if you are already
tracking the file, to take in account all the the filesystems out
there who are not case-preserving, case-sensitive?
If so, do you think those rules should be an option? Or a preference?
Should I specify in my config file that I want my filenames to be
normalized?
Ignoring encoding, and case-sensitive issues in the git index creates
problems for those people who want/need to use non-ascii chars in
their filenames, and have some change of being able to collaborate
with other users on different operating systems.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
Sorry, but you're using different characters that look the same. But Kevins
point was that it's a different thing if you use two characters that look the
same or the same character with different encodings.
But that's exactly the case he gave - 'ä' vs 'a¨' are exactly that:
different strings (not even characters: the second is actually a
multi-character) that just look the same.
You try to twist the argument by just claiming that they are the same
"character". They aren't, unless you *define* character to be the same as
"glyph". Of course, if you claim that, then you can always support your
argument, but I claim that is a bogus and incorrect axiom to start with!
Too many people confuse "character" and "glyph". They are different.
See, for example
http://en.wikipedia.org/wiki/Unicode
and notice the *many* places where they try to make that distinction
between "character" and "glyph" clear (and also "code values", which are
the actual bytes that encode a character).
See also
http://en.wikipedia.org/wiki/Unicode_normalization
and realize that a Unicode sequence is a sequence of *characters* even if
it is not normalized! Those things are still characters, when they are the
"simpler" non-combined characters.
You are trying to make a totally BOGUS argument, and you base it on the
INCORRECT basis that the TWO characters 'a'+'¨' somehow aren't independent
characters. They *are*. They are *different* characters from 'ä', even
though they may be "Canonically equivalent" as a sequence.
The fact is that "equivalent" does not mean "same". Why cannot people
accept that?
Linus
IMHO it would be the best solution when git stores all string meta data in
UTF-8 and converts it to the target systems file system encoding. That would
fix all those problems with different locales and file system encodings ...
+1.
And I would suggest the use of RFC 3454 as the guidelines for UTF-8
normalization.
The problem is that there is no way to know what the "target system
encoding" is.
And it wouldn't actually solve the bigger problem on OS X anyway: as long
as you are case-insensitive, you'll have all the same problems (ie the
insane OS X filesystem presumably thinks that "MÄRCHEN" and "Märchen" are
also identical, because they are "equivalent" names).
Linus
From: Mark Junker <hidden> Date: 2016-06-15 22:44:06
Linus Torvalds schrieb:
You try to twist the argument by just claiming that they are the same
"character". They aren't, unless you *define* character to be the same as
"glyph". Of course, if you claim that, then you can always support your
argument, but I claim that is a bogus and incorrect axiom to start with!
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 6:42 PM, Linus Torvalds wrote:
Too many people confuse "character" and "glyph". They are different.
This is very true.
The fact is that "equivalent" does not mean "same". Why cannot people
accept that?
I'll shut up now if you can answer me one question, because it
really is a problem for my team.
We have people using windows, people using Macs, and people using
several flavors of Linux desktops. They all have different settings
and if I add a file like áéióú that happens to be UTF-8 encoded, it
will reach a iso-latin-1 user as visual garbage. git will track the
file perfectly, we know that, because the sequence of bytes that my
system used to create the file will be the same on all "sane"
systems, but the file will look "funny" to some users, and we get
complaints for some less enlightened ones.
The answer is that users should not create filenames with non-ascii
characters if they want a consistent experience, right?
This is just so that I can write a best practices document to them...
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
On Thu, Jan 17, 2008 at 06:52:57PM +0000, Pedro Melo wrote:
The answer is that users should not create filenames with non-ascii
characters if they want a consistent experience, right?
This is just so that I can write a best practices document to them...
That's the easist thing to do if you want to assure that things will
mostly work across multiple different OS's, with different levels of
sanity. You might also want to include that it's a bad idea to create
two filenames that are identical on case-insensitive filesystems,
i.e., "makefile" and "Makefile", or "foo.H" and "foo.h" which even
though it works Just Fine on Linux, will likely cause problems on
Windows and MacOS filesystems, and other systems that are insane with
respect to case insensitivity.
- Ted
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hi,
On Jan 17, 2008, at 6:44 PM, Linus Torvalds wrote:
On Thu, 17 Jan 2008, Pedro Melo wrote:
quoted
On Jan 17, 2008, at 6:09 PM, Mark Junker wrote:
quoted
IMHO it would be the best solution when git stores all string
meta data in
UTF-8 and converts it to the target systems file system encoding.
That would
fix all those problems with different locales and file system
encodings ...
+1.
And I would suggest the use of RFC 3454 as the guidelines for UTF-8
normalization.
The problem is that there is no way to know what the "target system
encoding" is.
Correct. Storing or using a normalized version of the filename is
only part of the problem.
The full problem is:
User A <-> filesystem A <-#-> git < ...... > git <-#-> filesystem B <-
> user B.
You have to encode/decode/normalize on all the <-#-> and there is no
magic bullet. Each user would have to tell git "Hey I'm using utf-8"
or "Hey, I'm a masochist using HFS+".
But I think its important for git to store the filenames in something
that at least permits this kind of scenario.
All encoding/decoding/normalization is of course optional, and for
git, it still is a sequence of bytes.
And it wouldn't actually solve the bigger problem on OS X anyway:
as long
as you are case-insensitive, you'll have all the same problems (ie the
insane OS X filesystem presumably thinks that "MÄRCHEN" and
"Märchen" are
also identical, because they are "equivalent" names).
Correct. HFS+ has bigger problems. I'm not sure if this is enough to
solve it.
But it would solve two linux users using different encodings.
And given that the filtering layers are optional, you have to
configure them, it wont bite nobody.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
We have people using windows, people using Macs, and people using several
flavors of Linux desktops. They all have different settings and if I add a
file like áéióú that happens to be UTF-8 encoded, it will reach a iso-latin-1
user as visual garbage.
Yes.
git will track the file perfectly, we know that, because the sequence of
bytes that my system used to create the file will be the same on all
"sane" systems, but the file will look "funny" to some users, and we get
complaints for some less enlightened ones.
I can't really suggest anything else than trying to make everybody use
UTF-8.
[ Not just for filenames, by the way - this is one of the reasons I think
it is so *important* to not corrupt filenames, exactly because this is
in no way filename-specific at all, and filenames are generally "textual
data" exactly the same way a text-file is.
But only totally insane people think that you should force-normalize
text-files, even though all the issues are obviously all the same
regardless of whether it's a filename or a word in textfile. ]
And yes, I also realize that it's not going to be realistic. We're
probably *closer* to that than we used to be, but I don't think you can
even make Windows think FAT is UTF-8.
I don't know how NTFS works (I know it is Unicode-aware, and I think it
encodes filenames in UCS-2 or possibly UTF-16, but there is an obvious 1:1
translation to UTF-8, and since we use C strings, I'd assume/hope Windows
actually uses that unambiguous translation for any filenames).
Under modern Linux and OS X, UTF-8 is basically the only way (older Linux
distros may be set up for Latin1, but at least the newer ones seem to all
default to a UTF-8 locale).
The answer is that users should not create filenames with non-ascii characters
if they want a consistent experience, right?
Oh, absolutely. That takes care of 99.9% of all source projects. Even then
you can have problems with case insensitivity (the Linux kernel sources
are all US-ASCII filenames, for example, but *literally* has many files
that are identical if you ignore case, and that's not unheard of).
So yes, to a first approximation, the answer is to simply avoid using
anything but US-ASCII. It's seldom a big limitation when talking about
filenames.
Linus
On Thu, Jan 17, 2008 at 07:09:43PM +0100, Mark Junker wrote:
Sorry, but you're using different characters that look the same. But
Kevins point was that it's a different thing if you use two characters
that look the same or the same character with different encodings.
No, the encoding was the same -- UTF-8. MacOSX converts one sequence of
Unicode characters to *another* sequence, which are canonical equivalent,
but being canonical equivalent does not mean they are the same characters.
In the same way, as being compatible equivalent does not mean being the
same. As well as, being case-insensitive equivalent does not mean being
the same... Do you remember DOS? It stored all filenames in upper-case,
so they original and stored names are case-insensitive equivalent, but
they are not the same!
Dmitry
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 17, 2008, at 2:11 PM, Linus Torvalds wrote:
[ Not just for filenames, by the way - this is one of the reasons I
think
it is so *important* to not corrupt filenames, exactly because this
is
in no way filename-specific at all, and filenames are generally
"textual
data" exactly the same way a text-file is.
I just don't understand why you insist that the filename is data, when
it is clearly metadata. The filename has two purposes: the identify
the file to the user, and to provide a handle with which to reference
the file contents. The specific byte sequence is in no way sacred.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com
I just don't understand why you insist that the filename is data, when it is
clearly metadata.
Uhh. And exactly how do you know the difference, and why should it matter?
A lot of data is metadata. Look at the git index file. It's *all*
metadata. Does that mean that the OS has the right to corrupt it?
IOW, why do you seem to argue that metadata something you can corrupt, but
not then "regular" data?
Why is it ok to change a filename, when that same filename may *also* be
encoded by the user in a regular data file (think about MD5SUM files, for
example, that include the pathname, but now the pathname is part of the
file data, not on a filesystem).
So filenames are data, they're metadata, they're whatever. None of that
means that it's acceptable to corrupt them, or gives the OS any reason to
say that it "knows better" than the user in how users use them. It's still
the *users* metadata, not the filesystems own metadata!
In many cases, users use filenames *as* data, ie the filename actually has
a meaning in itself, not just as a handle to get the file contents.
If this was truly metadata that isn't visible to the user, and not under
the users control (ie indirect block numbers etc), then you'd have a good
point. At that point, it's obviously entirely up to the filesystem how the
heck it encodes it.
But that's not what filenames are. Filenames are an index specified by the
user, not by the computer.
Linus
From: Robin Rosenberg <hidden> Date: 2016-06-15 22:44:06
torsdagen den 17 januari 2008 skrev Kevin Ballard:
On Jan 17, 2008, at 10:57 AM, Johannes Schindelin wrote:
quoted
On Thu, 17 Jan 2008, Kevin Ballard wrote:
quoted
On Jan 17, 2008, at 5:22 AM, Wincent Colaiuta wrote:
quoted
While it's a nice workaround, it really is just that (a workaround)
because performance will be suboptimal in a repository running on a
disk image (and many of switched to Git because of its speed).
Not only is it suboptimal, it's also not acceptable, plain and
simple.
If it's not acceptable, do something about it (and I don't mean
writing 50
emails). If you don't want to do something about it, I have to
assume that
you accept it as-is.
I never said I don't want to do anything about it. However, I do
believe that it will take a significant investment of time and energy
to learn all the gooey details of how git handles filenames and how
the index works and all that jazz, which is knowledge that other
people already have. I believe that, for me to solve this problem
independently, it may require so much time that it never gets done
(after all, I am fairly busy). However, if other people who already
have this knowledge are willing to help, that would make this task far
easier, especially given that if nobody else even acknowledges that
this is a problem I don't have much hope of getting a patch accepted.
So again, I'm certainly going to try, but working by myself it simply
may never get done.
(This is only for those that think the problem should be solved somehow. The
rest can move on - nothing to see here)
You may look at http://rosenberg.homelinux.net/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=log;h=i18n
for inspiration. It's pretty obsolete by now and only a "proof of concept", i.e.
it can be done, not that it necessarily should be done exactly this way.
Basically it intercepts the user's access to git, i.e. certain commands
and how files are named (since those names represent a user interface). Then
it assumes the internal encoding is UTF-8 (or garbage) converting to and
from the user's local encoding. The heuristics is based on the assumption that
a string (even random onesthat looks like UTF-8, with a very high probablity
actually is UTF-8 encoded.
The test cases might be usable almost as is.
-- robin
From: Robin Rosenberg <hidden> Date: 2016-06-15 22:44:06
torsdagen den 17 januari 2008 skrev Linus Torvalds:
And yes, I also realize that it's not going to be realistic. We're
probably *closer* to that than we used to be, but I don't think you can
even make Windows think FAT is UTF-8.
It's UTF-16 (when needed). I think it's all in the Linux kernel for you
to see.
I don't know how NTFS works (I know it is Unicode-aware, and I think it
encodes filenames in UCS-2 or possibly UTF-16, but there is an obvious 1:1
UTF-16 (was UCS-2 until MS did a s/UCS-2/UTF-16/ on the documentation).
translation to UTF-8, and since we use C strings, I'd assume/hope Windows
actually uses that unambiguous translation for any filenames).
It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired
thingy, but in Asia multi-byte encodings are used. In western Europe it is
Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we
have the cmd prompt which has another encoding in 8-bit mode.
I think there is a cygwin patch that converts to and from UTF-8. An application
can choose to use the "A" or "W" interfaces. The W-API's are the real ones and
the others' are just wrappers that convert to and from UTF-16 before anything
happens (i.e. CreateFileA is slower than CreateFileW and so on).
-- robin
torsdagen den 17 januari 2008 skrev Linus Torvalds:
quoted
And yes, I also realize that it's not going to be realistic. We're
probably *closer* to that than we used to be, but I don't think you can
even make Windows think FAT is UTF-8.
It's UTF-16 (when needed). I think it's all in the Linux kernel for you
to see.
.. well, FAT certainly wasn't. But yes, VFAT probably is. Not that I want
to look at it ;)
quoted
translation to UTF-8, and since we use C strings, I'd assume/hope Windows
actually uses that unambiguous translation for any filenames).
It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired
thingy, but in Asia multi-byte encodings are used. In western Europe it is
Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we
have the cmd prompt which has another encoding in 8-bit mode.
Well, if it uses a 8-bit codepage, then that means that as far as the
POSIX filename interface is concerned, it has nothing what-so-ever to do
with Unicode (ie unicode is just a totally invisible internal encoding
issue, not externally visible).
I assume you have to use some insane Windows-only UCS-2 filename function
to actually see any Unicode behaviour.
Sad. Because there really is no reason to use a local 8-bit codepage when
you could just use UTF-8.
I think there is a cygwin patch that converts to and from UTF-8. An application
can choose to use the "A" or "W" interfaces. The W-API's are the real ones and
the others' are just wrappers that convert to and from UTF-16 before anything
happens (i.e. CreateFileA is slower than CreateFileW and so on).
So the CreateFileW() is the "native UTF-16 interface", and CreateFileA()
is the 8-bit codepage one that has nothing to do with Unicode and is
purely some local thing.
But for a UNIX interface layer, the most logical thing would probably be
to map "open()" and friends not to CreateFileA(), but to
CreateFileW(utf8_to_utf16(filename)).
Once you do that, then it sounds like Windows would basically be Unicode,
and hopefully without any crazy normalization (but presumably all the
crazy case-insensitivity cannot be fixed ;^).
So it probably really only depends on whether you choose to use the insane
8-bit code page translation or whether you just use a sane and trivial
UTF8<->UTF16 conversion.
Anybody know which one cygwin/mingw does?
Linus
From: Brian Dessent <hidden> Date: 2016-06-15 22:44:06
Linus Torvalds wrote:
But for a UNIX interface layer, the most logical thing would probably be
to map "open()" and friends not to CreateFileA(), but to
CreateFileW(utf8_to_utf16(filename)).
Once you do that, then it sounds like Windows would basically be Unicode,
and hopefully without any crazy normalization (but presumably all the
crazy case-insensitivity cannot be fixed ;^).
So it probably really only depends on whether you choose to use the insane
8-bit code page translation or whether you just use a sane and trivial
UTF8<->UTF16 conversion.
Anybody know which one cygwin/mingw does?
Cygwin does not yet support doing the smart thing. At the moment you
can only open() files in the current 8 bit codepage. There is a patch
floating around to allow using UTF-8, but it was rejected for inclusion
because it was considered too hackish. Instead work has been ongoing
for some time to replumb the internal representation of Windows
filenames to use UTF-16 instead of plain chars, so that conversion
overhead can be held at a minimum. In conjuction with dropping Win9x/ME
support this also means the Native APIs like NtCreateFile() can be used
directly, as they are more low level than the Win32 -A and -W functions
and expose more flexibility, such as the ability to implement the
openat() family of functions natively (no pun intended) without
emulation. These two items (unicode and dropping non-NT windows) are
the big features for 1.7.
Of course since a lot of what Cygwin does is translate paths in
sometimes unobvious and complicated ways, there's a lot of path handling
code to adapt, so it's taking a while.
Incidently, the ridiculously short MAX_PATH of 260 on Windows comes from
the Win32 -A version of the functions. The -W API and the Native API
can cope with paths of up to 32k wide chars, so a side benefit of this
should be the ability to finally stop running into length limits. Of
course there's always a catch: when using long filenames with the Win32
-W API or the Native API you can only use absolute paths, so either you
have to live with the 260 limitation for relative paths or you keep
track of the current directory and always do a rel->abs conversion. Or
better, if you stick to the Native API you can do a directory handle
relative openat-type thing which I suppose starts to sound relatively
sane. However, there's another catch here: For some time Cygwin has
maintained a separate and private value of CWD behind Windows' back, and
only synced the two when spawning a non-Cygwin binary. This allows
Windows to happly think the process' CWD is always C:\ or whatever, and
not hold an open handle to the actual CWD. In turn Cygwin uses this to
allow POSIX filesystem behavior of being able to unlink the current dir,
which some programs or build systems assume they can do but is not
possible in straight Win32. This is a roundabout way of saying that
going back to actually having to keep a handle to CWD open again in
order to do relative paths might be complicated.
Brian
From: Andrew Heybey <hidden> Date: 2016-06-15 22:44:06
Geert Bosch [off-list ref] writes:
For those on Mac OS X: it is possible to create a case-sensitive HFS+
partition and
use it with git. You even can just create a disk image and mount it.
However,
I wouldn't quite try to use it as startup filesystem...
This is starting to stray far afield, but the first thing I did when I
got a Macbook was to reinstall it with case-sensitive HFS as the boot
file system. Works fine, including with git. The only problem I have
had is that FileVault does not work. There are rumored to be some
third-part apps that do not work but I do not use that many of those
anyway.
andrew
From: Peter Karlsson <hidden> Date: 2016-06-15 22:44:06
Linus Torvalds:
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*,
This is the same issue as the CRLF issue I posted on earlier, and it
all stems from that git also sees file names as a stream of bytes, not
a string of characters, just as it does text.
An OS that silently changes the contents of your files is *crap*.
Get it?
A program that silently ignores the conventions of the platform it runs
on is *crap*, no matter if the conventions are not the same as for
other platforms.
An OS that silently changes the contents of your directories is *crap*.
Get it now?
A program that silently ignores the conventions of the file system it
tries to store its files on is *crap* :-)
In my perfect world, file names would be stored as a string of characters,
so if I save a file with an å in it, that å would be preserved no
matter if I run Linux on ext2 with my locale is set to latin-1 (which
stores it as byte 0xE5), on Windows with NTFS (which stores it as the
UTF-16 code 0x00E5), on Windows/DOS with FAT (which stores it as the
byte 0x86) or on Mac OS X which stores it as decomposed UTF-8 (whose
byte sequence I don't know at the top of my head). If that was just
stored as U+00E5 in whatever encoding in the filename index, the local
implementation of git can just check it out in the form needed.
--
\\// Peter - http://www.softwolves.pp.se/
On Thu, Jan 17, 2008 at 05:24:01PM -0800, Linus Torvalds wrote:
On Fri, 18 Jan 2008, Robin Rosenberg wrote:
quoted
It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired
thingy, but in Asia multi-byte encodings are used. In western Europe it is
Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we
have the cmd prompt which has another encoding in 8-bit mode.
Yes, the default code page for the command prompt uses so-called OEM
encoding, and GUI programs uses another one, which MS calls as "ANSI"
encoding. However, if you use Cygwin, then you have ANSI encoding in
the command prompt. So, in the same command prompt window, you can have
Cygwin programs using one encoding and other window console programs
using a different encoding.
Well, if it uses a 8-bit codepage, then that means that as far as the
POSIX filename interface is concerned, it has nothing what-so-ever to do
with Unicode (ie unicode is just a totally invisible internal encoding
issue, not externally visible).
Some people tried to set the current code page to 65001, which is
the Microsoft code page for UTF-8. However, it seems that does not
work very well.
http://support.microsoft.com/kb/175392http://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx
It seems to me that Win32 API functions work correctly with
UTF-8 (after all, they are just wrappers over UTF-16 functions),
but Microsoft's C library cannot handle UTF-8 (or any other
encoding that requires more than two bytes per character).
Anybody know which one cygwin/mingw does?
There is a patch for Cygwin that adds UTF-8 support for it, however,
Cygwin maintainers do not like it, so it is not integrated. I think
Cygwin 1.7 will support UTF-8, but I have no idea how soon it will be
released.
I don't know much about mingw, but if I am not mistaken, mingw relies
on Microsoft's C library, so I suppose it uses an "OEM" code page for
console programs by default.
Dmitry
From: Robin Rosenberg <hidden> Date: 2016-06-15 22:44:06
fredagen den 18 januari 2008 skrev Linus Torvalds:
quoted
quoted
translation to UTF-8, and since we use C strings, I'd assume/hope Windows
actually uses that unambiguous translation for any filenames).
It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired
thingy, but in Asia multi-byte encodings are used. In western Europe it is
Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we
have the cmd prompt which has another encoding in 8-bit mode.
Well, if it uses a 8-bit codepage, then that means that as far as the
POSIX filename interface is concerned, it has nothing what-so-ever to do
with Unicode (ie unicode is just a totally invisible internal encoding
issue, not externally visible).
I just had to investigate this a bit, so on a Vista machine I started a cmd
prompt and typed mode con: cp select=65001, selected the lucida font and then
echo å >x.txt and opened it in notepad and it was UTF-8 encoded. So there might
be some hope after all. I don't know how to change the encoding for non-console
apps. I leave that as an excercise for the list.
-- robin
From: Peter Karlsson <hidden> Date: 2016-06-15 22:44:06
Linus Torvalds:
But that's exactly the case he gave - 'ä' vs 'a¨' are exactly that:
different strings (not even characters: the second is actually a
multi-character) that just look the same.
But they are not different strings, they are canonically equivalent as
far as Unicode is concerned. They're even supposed to map to the same
glyph (if the font has an "ä", it should display it in both cases, if
it has an "a" and a combining diaeresis, it should make up one).
You cannot do a binary comparison of text to see if two strings are
equivalent.
You try to twist the argument by just claiming that they are the same
"character". They aren't, unless you *define* character to be the
same as "glyph".
Whereas you are confusing characters and code points.
"ä" and "a¨" use different code points, but they encode the same
character, and from the user's perspective it is the *character* that
is interesting (although he might confuse it with the glyph).
I don't know how NTFS works (I know it is Unicode-aware, and I think
it encodes filenames in UCS-2 or possibly UTF-16,
Actually, NTFS is a bit broken. It sees file names as a string of
16-bit words. It doesn't check that it is valid UTF-16, or even valid
UCS-2, it allows almost anything.
Apple made Mac OS X handle filenames properly, by seeing that file
names are a string of characters, not code points, so they use a
canonical form for all characters (personally, I would have preferred
the pre-composed form, though).
--
\\// Peter - http://www.softwolves.pp.se/
On Fri, Jan 18, 2008 at 10:42:36AM +0100, Robin Rosenberg wrote:
I just had to investigate this a bit, so on a Vista machine I started a cmd
prompt and typed mode con: cp select=65001, selected the lucida font and then
echo å >x.txt and opened it in notepad and it was UTF-8 encoded.
Yes, but have you tried to run any batch file? At least, on WinXP
all batch files silently stopped working after choosing 65001, and
I don't know what else gets broken, because Microsoft C library
does not work with encoding that requires more than two bytes per
character.
So there might
be some hope after all. I don't know how to change the encoding for non-console
apps. I leave that as an excercise for the list.
It is not difficult to change the current encoding in any Windows
application, the real issue is that neither Microsoft C library nor
Cygwin library does not work correctly with UTF-8. There is a patch
for Cygwin though...
Dmitry
On Fri, Jan 18, 2008 at 11:19:21AM +0100, Peter Karlsson wrote:
Linus Torvalds:
quoted
But that's exactly the case he gave - 'ä' vs 'a¨' are exactly that:
different strings (not even characters: the second is actually a
multi-character) that just look the same.
But they are not different strings, they are canonically equivalent as
far as Unicode is concerned.
There are canonically equivalent, but they are different sequences
of characters as Unicode is concerned. In one case, we have one
character in the other case, we have two characters that canonically
equivalent to the first one.
They're even supposed to map to the same
glyph (if the font has an "ä", it should display it in both cases, if
it has an "a" and a combining diaeresis, it should make up one).
By defition, sequences of characters that are canonically equivalent
are both visual and functional equivalent...
You cannot do a binary comparison of text to see if two strings are
equivalent.
Of course, you can't. Who argues otherwise?
quoted
You try to twist the argument by just claiming that they are the same
"character". They aren't, unless you *define* character to be the
same as "glyph".
Whereas you are confusing characters and code points.
I am afraid it is you who confuses "characters" with "abstract
characters", there is no place in the standard saying that
"characters" are "abstract characters" only. On contrary, the
term "characters" is used to refer non abstract characters.
Dmitry
From: Jakub Narebski <hidden> Date: 2016-06-15 22:44:06
Peter Karlsson wrote:
Linus Torvalds wrote:
quoted
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*,
This is the same issue as the CRLF issue I posted on earlier, and it
all stems from that git also sees file names as a stream of bytes, not
a string of characters, just as it does text.
You have to be careful about CRLF conversion, lest you corrupt your
binary files. CRLF conversion is off by default.
quoted
An OS that silently changes the contents of your files is *crap*.
Get it?
A program that silently ignores the conventions of the platform it runs
on is *crap*, no matter if the conventions are not the same as for
other platforms.
quoted
An OS that silently changes the contents of your directories is *crap*.
Get it now?
A program that silently ignores the conventions of the file system it
tries to store its files on is *crap* :-)
Git philosophy to see the contents of files and "contents" of directories
(filenames) as stream of bytes, i.e. to use 'native' encoding works
perfectly well and _fast_ if all developers work in the same environment.
Troubles start if you are working across operating systems, and across
filesystems.
In my perfect world, file names would be stored as a string of characters,
so if I save a file with an å in it, that å would be preserved no
matter if I run Linux on ext2 with my locale is set to latin-1 (which
stores it as byte 0xE5), on Windows with NTFS (which stores it as the
UTF-16 code 0x00E5), on Windows/DOS with FAT (which stores it as the
byte 0x86) or on Mac OS X which stores it as decomposed UTF-8 (whose
byte sequence I don't know at the top of my head). If that was just
stored as U+00E5 in whatever encoding in the filename index, the local
implementation of git can just check it out in the form needed.
Git has for a long time i18n.commitEncoding, and from some time it
saves it in 'encoding' header in commit object (if different from
'uft-8') and has also i18n.logOutputEncoding.
For dealing with different filesystem encodings you would also have
to have both: encoding used in 'tree' objects (by repository) for
filenames saved somewhere in repository, either in tree object (argh!)
or in some kind of .gitconfig file; encoding used by filesystem in
repository config as i18n.filesystemEncoding or something like that.
And think what to put in the on disk index, and in memory index.
NOTE, NOTE, NOTE! If filename is used somewherein the file contents
(manifest-like file, include-like statement), and this filename uses
characters which are differently encoded in different encoding you
are screwed with this fancy system, badly, anyway.
--
Jakub Narebski
Poland
From: Peter Karlsson <hidden> Date: 2016-06-15 22:44:06
Dmitry Potapov:
I am afraid it is you who confuses "characters" with "abstract
characters", there is no place in the standard saying that
"characters" are "abstract characters" only. On contrary, the term
"characters" is used to refer non abstract characters.
Perhaps it's just a case of confusion about naming conventions. I tend
to use "character" as a "grapheme cluster", i.e a "user character" (to
the end user, "ä" and "a"+diaeresis is the same character, no matter if
they would display as different glyphs), whereas some people use
"character" as a "code point", which would be more of a "programmer
character". And then there are some people that still use "character"
interchangibly for "bytes" or "code units" (for UTF-16; a pair of
surrogate code units is still only one "code point").
--
\\// Peter - http://www.softwolves.pp.se/
From: Peter Karlsson <hidden> Date: 2016-06-15 22:44:06
Dmitry Potapov:
because Microsoft C library does not work with encoding that requires
more than two bytes per character.
Indeed. On Windows, you should avoid using UTF-8 and instead use UTF-16
everywhere. That usually works better, and if you run on an NT-based
system it will convert all the data to WinAPI to UTF-16 anyway.
--
\\// Peter - http://www.softwolves.pp.se/