Junio C Hamano [off-list ref] writes:
Sam Vilain [off-list ref] writes:
quoted
Some projects may like to enforce a particular encoding is used for
all filenames in the repository. Within the UTF-8 encoding, there are
four normal forms (see http://unicode.org/reports/tr15/), any of which
may be a reasonable repository format choice. Additionally, some
filesystems may have a single encoding that they support when writing
local filenames. To support this, iconv and a normalization library
must have the information they need to perform the correct conversion.
Isn't there a chicken-and-egg problem? The attributes are by
nature per-path, and you need to match the pathname string with
a pattern to decide which attribute definition to apply to a
given path. Before knowing what encoding the pathname you have
just read from readdir(3), how would you match that pathname
with the pattern in the gitattributes file?
I can buy the .git/config (and an in-tree .git-encoding,
perhaps), though.
I admit that Documentação/ja/お読み下さい example was contrived
(the last component is README-in-Japanese), and if anybody still
wanted to have such a tree sanely, the only practical
cross-platform and multi-language way to do so is to have
everything in UTF-8 at the repository level.
In that sense, the project does not need to specify anything,
other than marking that "all of the pathnames in tree objects
are in UTF-8 (we could go stronger, and say which kind of
normalization we want)". As there is no other practical choice
than UTF-8-NFC if you want to be cross-platform, compatible, and
multi-language, the project can just declare that is what it
uses and does not have to mark it any specially.
A particular clone of such a project may want to check
everything out as-is to get an UTF-8 only tree (I'll mention
HFS+ shortly). Another clone may want to get mixed legacy
encodings by running mkdir(utf8_to_latin1("Documentação")) and
creat(utf8_to_eucjp(" お読み下さい")), but that is purely a
local matter and should not be controlled by anything in-tree,
be it .gitattributes or .git-encoding.
On the other hand, it is not so unusual to see a legacy encoding
used in the pathnames, especially if your project does not need
to deal with multi-language issues. In such a repository, I do
not want to enforce that all the paths in tree objects MUST be
UTF-8. If all the project participant agree to work with EUC-JP
pathnames in tree objects, we should not make the users always
go through double conversion going from readdir(3) to index, and
coming from index back to open(2) or creat(2). Again, that is
done by agreement by project participants, so there is nothing
that needs to be specified in-tree.
If the project uses UTF-8-NFC, we would need to adjust check-in
and check-out codepath like Linus's readdir(3) hack suggested,
but that needs to be done only on HFS+. Of course, the project
participants need to be careful not to create files that HFS+
cannot handle (two paths that happen to be equivalent strings
should not be created), but I do not think that is such a big
issue as some people seem to make a big deal out of. If you
want to be interoperable with different filesystems, you should
not create two paths that are different only in case, and if
there are participants who are on such a filesystem, the mistake
is quickly spotted and corrected. It happened in git.git to a
file other than that infamous Märchen. It's exactly the same
issue [*1*].
In short, initially I did not like Linus's readdir(3) hack very
much, but the more I think about it, I like it the better.
We pick a reasonable default (i.e. "no conversion") at the
technical level, and recommend (but do not pay for the overhead
of enforcing) a reasonable normalization as the BCP at the human
level. Only on filesystems that mangle the pathnames, or if you
want legacy encodings on the filesystem, we would need to pay
overhead for conversion and help people with actual code to do
so.
To support the above scenarios, I think each instance of
repository needs to be able to say "this path (specified with a
matching pattern in the filename encoding) should be converted
this way coming in, and that way going out." UTF-8 only project
would have NKC<->NKD on HFS+ partition, and nothing on
everywhere else. EUC-JP project that checks out as-is would
specify nothing either, but people on Shift_JIS platforms would
locally specify that EUC-JP <-> Shift_JIS conversion to be made.
[Footnote]
*1* This is an important point, especially the breakage was
about tests that used files "a" and "A". No pathname
enforcement in git-as-scm would have enforced anything to avoid
the breakage. But there are humans involved in the project and
they are an integral part of ensuring interoperability.
Junio C Hamano schrieb:
To support the above scenarios, I think each instance of
repository needs to be able to say "this path (specified with a
matching pattern in the filename encoding) should be converted
this way coming in, and that way going out." UTF-8 only project
would have NKC<->NKD on HFS+ partition, and nothing on
everywhere else. EUC-JP project that checks out as-is would
specify nothing either, but people on Shift_JIS platforms would
locally specify that EUC-JP <-> Shift_JIS conversion to be made.
Just to sum up what you wrote and to be sure that I understand you
correctly:
Lets have two encodings:
- Encoding for path names stored in the repository
- Encoding for path names from/to file systems
Do conversion only if they are different. Both encodings are configurable.
Regards,
Mark
On 22/01/2008, Junio C Hamano wrote:
If the project uses UTF-8-NFC, we would need to adjust check-in
and check-out codepath like Linus's readdir(3) hack suggested,
but that needs to be done only on HFS+. Of course, the project
participants need to be careful not to create files that HFS+
cannot handle (two paths that happen to be equivalent strings
should not be created), but I do not think that is such a big
issue as some people seem to make a big deal out of. If you
Right, I don't see that as a big issue -- for new files. But we can have
files that were created in the past as non-handleable by HFS+, and later
renamed to something more portable.
More generally, the consensus encoding might change over time. We can
imagine a project which contains, say, a test file which a latin-1 name,
that gets later renamed to a UTF-8 name, (due to a project policy
change), but making necessary to adjust the said test. A checkout of the
earlier version would have that test failing. (But maybe I'm just
handwaving towards a non-existent problem here. I'd consider the issue
as minor anyway.)
want to be interoperable with different filesystems, you should
not create two paths that are different only in case, and if
there are participants who are on such a filesystem, the mistake
is quickly spotted and corrected. It happened in git.git to a
file other than that infamous Märchen. It's exactly the same
issue [*1*].
In short, initially I did not like Linus's readdir(3) hack very
much, but the more I think about it, I like it the better.
We pick a reasonable default (i.e. "no conversion") at the
technical level, and recommend (but do not pay for the overhead
of enforcing) a reasonable normalization as the BCP at the human
level. Only on filesystems that mangle the pathnames, or if you
want legacy encodings on the filesystem, we would need to pay
overhead for conversion and help people with actual code to do
so.
To support the above scenarios, I think each instance of
repository needs to be able to say "this path (specified with a
matching pattern in the filename encoding) should be converted
this way coming in, and that way going out." UTF-8 only project
would have NKC<->NKD on HFS+ partition, and nothing on
everywhere else. EUC-JP project that checks out as-is would
specify nothing either, but people on Shift_JIS platforms would
locally specify that EUC-JP <-> Shift_JIS conversion to be made.
Sounds sane, except maybe the part where you specify paths with a
pattern. Do you really need this layer of complexity? Pattern matching
in different encodings has proven to be troublesome. Usually that's
where UTF-8 normalisation rules and locale-specific behaviours kick in,
esp. when you're starting to use \w or \d characters classes, or case
insensitivity. For example, if you want to do it correctly, "I" will
match /i/ case-insensitively, except in Turkish locales... (Sorry, I'm
just handwaving again here...)
Junio C Hamano wrote:
To support the above scenarios, I think each instance of
repository needs to be able to say "this path (specified with a
matching pattern in the filename encoding) should be converted
this way coming in, and that way going out." UTF-8 only project
would have NKC<->NKD on HFS+ partition, and nothing on
everywhere else.
I think there is another reason to do this - simple sanity. Two people
adding the same filename should not end up with a different tree ID, if
they for whatever reason ended up entering a differing equivalent
variant of the same Unicode NKC form.
But, that rule of sanity breaks the C semantics sanity, so it must be a
per-project setting. Not a necessity, but a good feature I think. It
can be enforced with external scripts/hooks of course.
What happens on the way in and out of the filesystem, I see that as a
side issue. Once you define what the normalized form is for the
project, then the features should just fall into place without messy
heuristics. There is also a correct behaviour when faced with
filesystems that have a different idea about who enforces encoding rules
- so long as you can detect what those ideas are :). It also means that
users can choose to use the same local encoding as their locale, which
might interoperate better with other apps.
The readdir() (case|normalization) tolerance change is good in its own
right, but it's a slightly different scenario, and an independent
question to what is the normalized form. Of course, on case folding,
unicode normalizing filesystems you'd have to have a mixture of these
settings for sane operation.
On the chicken and egg thing, I guess .gitattributes is too late, you're
right - unless you say that at each directory level, the globbing is
always C. But I haven't thought about that very hard. I was just
re-using a mechanism that already exists rather than try to invent
something new. I do agree with Dscho's point that mixing encodings in a
repository is not necessarily a use case worth catering for.
Sam.