Re: utf8 BOM

On 16. mai 2010, at 07.19, Dmitry Potapov wrote:

On Sat, May 15, 2010 at 10:23:52PM +0200, Eyvind Bernhardsen wrote:

quoted

(which hopefully works no matter what your code
page is?  I don't know much about Windows i18n).

Yes, it does. I am not an expert on Windows either, but as far as I
know, BOM are used to mark unicode files, which could be either UTF-8
or UTF-16. BTW, UTF-16 are treated by Git as "binary" now, which may
not always convenient, because impossible to do "merge" or "diff".

Okay, so something that checks text files to see if they're utf16 (maybe just accept anything with a utf16 BOM as utf16?) and converts them to utf8 might be useful on any platform.  Stripping utf8 BOMs and optionally re-adding them on output would be a natural extension.  "core.autoutf", anyone?

quoted

Adding this to convert.c would be more difficult, at least
politically, since I assume it would be Windows-specific code.

I don't think it needs any Windows-specific code. We already have some
functions to convert text from different charsets, which could be used.
But this feature should be developed and tested by people who work on
Windows regularly and need this feature, because there is no substitute
for testing and experience of how well it works in practice. Currently,
I rarely use Windows and can get by clean/smudge filters.

Yeah, the problem is finding someone who needs the feature _and_ is able/willing to implement it.  I try to keep a Unix-like experience on Windows, so I don't usually run into utf8 BOMs.
-- 
Eyvind