From: David Kastrup <hidden> Date: 2016-06-15 22:44:06
Pedro Melo [off-list ref] writes:
On Jan 17, 2008, at 12:16 AM, Linus Torvalds wrote:
quoted
On Wed, 16 Jan 2008, Pedro Melo wrote:
quoted
The difference I see between us is that if I tell my filesystem that
I want to name my file with a particular string encoded in X, users
using encoding Y will be able to read it correctly. I like my
filesystem to make that work for me.
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*, you don't seem to get
it.
I get that you think its the same thing.
What I don't get is why a user should be forced to know what type of
encoding he and the other users are using on all the layers going down
to the filesystem. If two users on different systems or in different
configurations, choose the same unicode string as the name, why do we
need to make it harder for things to just work out?
If you do the normalization in the right place, things will just work
out. The file system is not the right place.
I'm willing to accept a file system or other layer that normalizes
encoding of filenames if that makes the end-user life easier,
specially in a tool distributed by nature.
Well, as the issue shows it does not make life for the end-user easier.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
From: Pedro Melo <hidden> Date: 2016-06-15 22:44:06
Hello,
On Jan 17, 2008, at 12:32 AM, David Kastrup wrote:
Pedro Melo [off-list ref] writes:
quoted
On Jan 17, 2008, at 12:16 AM, Linus Torvalds wrote:
quoted
On Wed, 16 Jan 2008, Pedro Melo wrote:
quoted
The difference I see between us is that if I tell my filesystem
that
I want to name my file with a particular string encoded in X, users
using encoding Y will be able to read it correctly. I like my
filesystem to make that work for me.
The difference I see between us is that when I tell you that this is
exactly the same thing as your file *contents*, you don't seem to
get
it.
I get that you think its the same thing.
What I don't get is why a user should be forced to know what type of
encoding he and the other users are using on all the layers going
down
to the filesystem. If two users on different systems or in different
configurations, choose the same unicode string as the name, why do we
need to make it harder for things to just work out?
If you do the normalization in the right place, things will just work
out. The file system is not the right place.
No problem, but don't you think that git should to it?
Don't you think its important in a distributed tool that no matter
what system they use, be it linux or solaris, they are able to talk
about a file with non-ascii chars and be the same file to both of them?
That's the point I'm making. The fact that I need to set LANG across
all users of a project is insane...
quoted
I'm willing to accept a file system or other layer that normalizes
encoding of filenames if that makes the end-user life easier,
specially in a tool distributed by nature.
Well, as the issue shows it does not make life for the end-user
easier.
I'm assuming you are talking about HFS+ and the strange normalization
it does.
I'm sorry but that was not the problem I sent. I sent a scenario, in
which two users, using the same linux system but with different LANG
settings cannot use git reliably.
Although this thread started because of HFS+ "choices", the problem
is not really related to HFS+ given that you can have the same issues
even on the same physical <insert flavor here> POSIX system.
Best regards,
--
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo@simplicidade.org
Use XMPP!
That's the point I'm making. The fact that I need to set LANG across
all users of a project is insane...
I don't think I'd call that "insane" (in fact, I think these
discussions would be much less irritating for all involved if we
didn't use that word so often, even when it's not called for). It's
not that different than the whole LF/CRLF line-ending thing.
The real problem is that setting LANG won't help you on Mac OS X; set
LANG to whatever you want and there is *nothing* that you can do to
stop your filenames being normalized into decomposed UTF-8, short of
dropping HFS+. You can use an alternative filesystem, but support for
basically everything except HFS+ is suboptimal in Mac OS X at the
moment.
Cheers,
Wincent
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:06
Hi,
On Thu, 17 Jan 2008, Wincent Colaiuta wrote:
El 17/1/2008, a las 1:40, Pedro Melo escribió:
quoted
That's the point I'm making. The fact that I need to set LANG across
all users of a project is insane...
FWIW if you use another filesystem, such as reiserfs or ext[2-4], the
filenames will be _unaffected_ by your particular setting of LANG. They
will be stored byte-wise exactly like asked for. That's why I call them
"sane".
Hth,
Dscho
That's the point I'm making. The fact that I need to set LANG across
all users of a project is insane...
FWIW if you use another filesystem, such as reiserfs or ext[2-4], the
filenames will be _unaffected_ by your particular setting of LANG. They
will be stored byte-wise exactly like asked for. That's why I call them
"sane".
One of the advantages (the biggest one, in fact, apart from the obvious
US-ASCII down-compatibility and the fact that you can do C-compatible
NUL-terminated strings) of UTF-8 is that it's locale-independent, and
doesn't care about LANG, because it's valid in all languages.
And that's really important. It's important for a very simple reason:
there is almost never such a thing as "a locale" except for US-ASCII. Once
you move away from US-ASCII, it actually tends to be much more common that
you have a *mixture* of locales - often in the same "document" - than to
have one single locale.
It very much happens even in filenames - people "mix" locales in trivial
ways even within a single pathname component (non-US-ASCII filename, but
with a regular file extension), but much more interestingly they do so
within a directory tree (ie you have have translation subdirectories where
the filenames themselves are in another language, and you can have full
pathnames where different components are in different languages, for
example).
And UTF-8 is _wonderful_ for this, because LANG doesn't matter, and
cannot matter, and thus mixing isn't a problem.
Of course, you can screw it up. Locales still can change things like sort
order and capitalization etc, so even if you use UTF-8, you sure can get
into trouble with LANG and thinking that a per-session locale makes sense.
So choosing UTF-8 for the filesystem isn't wrong per se. It's a fine
choice, and has no issues with LANG in itself. Limiting it to strictly
valid UTF-8 encodings is also fine. Limiting it (further) to only
character normalized UTF-8 is also fine.
Most Linux filesystems don't limit it in any way, so you can make
filenames that aren't valid UTF-8 at all, much less normalizing
multi-character sequences.
I personally think that's the best option, but I probably do so mostly
because I know some people still use Latin1 as their only locale (and I
suspect Asia will take decades before it has converted to UTF-8 and will
also have cases where they use other non-UTF locales).
But enforcing clean UTF-8 is not a bad idea per se. Not allowing byte
sequences that aren't a valid UTF-8 encoding (eg \xc0\xc0 is not a valid
UTF-8 character) is fine.
I wouldn't call people crazy for doing that, although it does mean that
you cannot, for example, decide to write a Latin1 filename (which is not
necessarily a *good* idea in this day and age, but I think there's a
difference between "that's not a good idea" and "you cannot do that").
And even limiting the UTF-8 charset further to only the minimal
representation of one particular glyph (ie not allowing multi-character
sequences that can be represented more simply) may be even *more*
big-brother, but would at least not cause the technical aliasing issues. I
personally think that's so controlling as to be stupid (and has no real
advantage), but hey, at least it doesn't *corrupt* anything silently.
So I think that using UTF-8 as a character encoding is a *good* thing to
do, and that automatically means that LANG shouldn't matter for filenames,
but within that choice of UTF-8 there are still mistakes that you can
make. Notably multi-character normalization and case-insensitivity.
Linus
From: Kevin Ballard <hidden> Date: 2016-06-15 22:44:06
On Jan 16, 2008, at 8:41 PM, Linus Torvalds wrote:
On Thu, 17 Jan 2008, Johannes Schindelin wrote:
quoted
On Thu, 17 Jan 2008, Wincent Colaiuta wrote:
quoted
El 17/1/2008, a las 1:40, Pedro Melo escribió:
quoted
That's the point I'm making. The fact that I need to set LANG
across
all users of a project is insane...
FWIW if you use another filesystem, such as reiserfs or ext[2-4], the
filenames will be _unaffected_ by your particular setting of LANG.
They
will be stored byte-wise exactly like asked for. That's why I call
them
"sane".
One of the advantages (the biggest one, in fact, apart from the
obvious
US-ASCII down-compatibility and the fact that you can do C-compatible
NUL-terminated strings) of UTF-8 is that it's locale-independent, and
doesn't care about LANG, because it's valid in all languages.
And that's really important. It's important for a very simple reason:
there is almost never such a thing as "a locale" except for US-
ASCII. Once
you move away from US-ASCII, it actually tends to be much more
common that
you have a *mixture* of locales - often in the same "document" -
than to
have one single locale.
It very much happens even in filenames - people "mix" locales in
trivial
ways even within a single pathname component (non-US-ASCII filename,
but
with a regular file extension), but much more interestingly they do so
within a directory tree (ie you have have translation subdirectories
where
the filenames themselves are in another language, and you can have
full
pathnames where different components are in different languages, for
example).
And UTF-8 is _wonderful_ for this, because LANG doesn't matter, and
cannot matter, and thus mixing isn't a problem.
Of course, you can screw it up. Locales still can change things like
sort
order and capitalization etc, so even if you use UTF-8, you sure can
get
into trouble with LANG and thinking that a per-session locale makes
sense.
So choosing UTF-8 for the filesystem isn't wrong per se. It's a fine
choice, and has no issues with LANG in itself. Limiting it to strictly
valid UTF-8 encodings is also fine. Limiting it (further) to only
character normalized UTF-8 is also fine.
Most Linux filesystems don't limit it in any way, so you can make
filenames that aren't valid UTF-8 at all, much less normalizing
multi-character sequences.
I personally think that's the best option, but I probably do so mostly
because I know some people still use Latin1 as their only locale
(and I
suspect Asia will take decades before it has converted to UTF-8 and
will
also have cases where they use other non-UTF locales).
But enforcing clean UTF-8 is not a bad idea per se. Not allowing byte
sequences that aren't a valid UTF-8 encoding (eg \xc0\xc0 is not a
valid
UTF-8 character) is fine.
I wouldn't call people crazy for doing that, although it does mean
that
you cannot, for example, decide to write a Latin1 filename (which is
not
necessarily a *good* idea in this day and age, but I think there's a
difference between "that's not a good idea" and "you cannot do that").
And even limiting the UTF-8 charset further to only the minimal
representation of one particular glyph (ie not allowing multi-
character
sequences that can be represented more simply) may be even *more*
big-brother, but would at least not cause the technical aliasing
issues. I
personally think that's so controlling as to be stupid (and has no
real
advantage), but hey, at least it doesn't *corrupt* anything silently.
So I think that using UTF-8 as a character encoding is a *good*
thing to
do, and that automatically means that LANG shouldn't matter for
filenames,
but within that choice of UTF-8 there are still mistakes that you can
make. Notably multi-character normalization and case-insensitivity.
Linus
Alright, you've made your point, and I'm willing to concede at least
some of what you've said. So perhaps we can now move onto the more
relevant and practical issue of: HFS+, despite how stupid it may or
may not be, normalizes filenames (and is case-insensitive, which is a
related issue). This causes a problem with git. How can this be solved?
I'm more than willing to do work to solve it, my biggest issue is I
don't believe I actually have the free time to learn the git internals
well enough to actually do proper work on what I would assume is a
fairly performance-critical section of git's code. However, I would be
happy to work with others who are perhaps more knowledgeable in this
area.
-Kevin Ballard
--
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com