Re: mingw, windows, crlf/lf, and git

Johannes Schindelin [off-list ref] writes:

Hi,

On Mon, 12 Feb 2007, Junio C Hamano wrote:

quoted

I agree that we can assume editors can grok files with LF end-of-line 
just fine and we would not need to do the reverse conversion on checkout 
paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely 
on Windows' editors not to fsck up with line endings. The worst case is if 
the file contains _some_ CRLF and _some _LF_. Almost always I had the 
problem that it now converted _all_ LFs to CRLFs. Even those which already 
were converted.

So, if we are to support text mode, it is not one-way. If we do one-way, 
we really do _not_ support text mode, but pre-commit conversion to LF 
style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

 - I do not see how you propose to solve some CRLF and some LF
   case with both-ways conversion.

 - Pre-commit hook would not be sufficient.  In a edit, diff,
   test and then commit cycle, diff and test step needs to look
   at whatever the editor left on the filesystem, so the changes
   to populate-filespec is needed to make diff part work.


   

Hi,

On Mon, 12 Feb 2007, Junio C Hamano wrote:

Johannes Schindelin [off-list ref] writes:

quoted

Hi,

On Mon, 12 Feb 2007, Junio C Hamano wrote:

quoted

I agree that we can assume editors can grok files with LF end-of-line 
just fine and we would not need to do the reverse conversion on checkout 
paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely 
on Windows' editors not to fsck up with line endings. The worst case is if 
the file contains _some_ CRLF and _some _LF_. Almost always I had the 
problem that it now converted _all_ LFs to CRLFs. Even those which already 
were converted.

So, if we are to support text mode, it is not one-way. If we do one-way, 
we really do _not_ support text mode, but pre-commit conversion to LF 
style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

 - I do not see how you propose to solve some CRLF and some LF
   case with both-ways conversion.

Very easy. Forward: s/\r\n/\n/. Backward: s/\(^\|[^\r]\)\n/\r\n/.

 - Pre-commit hook would not be sufficient.  In a edit, diff,
   test and then commit cycle, diff and test step needs to look
   at whatever the editor left on the filesystem, so the changes
   to populate-filespec is needed to make diff part work.

Yes, you are right.

However, since this is all post-1.5.0 (right? Right?) why not go with more 
of Ted's proposal, and make this whole mess also usable for other things 
than just crlf issues?

And I _really_ think that you do not help Windows people by doing this 
one-way thing.

Ciao,
Dscho

On Mon, 12 Feb 2007, Junio C Hamano wrote:

quoted

Hi,

On Mon, 12 Feb 2007, Junio C Hamano wrote:

quoted

I agree that we can assume editors can grok files with LF end-of-line
just fine and we would not need to do the reverse conversion on checkout
paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely
on Windows' editors not to fsck up with line endings. The worst case is if
the file contains _some_ CRLF and _some _LF_. Almost always I had the
problem that it now converted _all_ LFs to CRLFs. Even those which already
were converted.

So, if we are to support text mode, it is not one-way. If we do one-way,
we really do _not_ support text mode, but pre-commit conversion to LF
style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

- I do not see how you propose to solve some CRLF and some LF
  case with both-ways conversion.

the expectation is that the some-of-each situation is unlikly to happen if you 
convert all the time.

and if you do end up with a mixed ending file, the next time you check it in 
from a windows box it should clean it up.

David Lang

Johannes Schindelin wrote:

However, since this is all post-1.5.0 (right? Right?) why not go with more 
of Ted's proposal, and make this whole mess also usable for other things 
than just crlf issues

Whatever is done, it needs to be robust to the notion that people will 
fail to set the correct file type somewhere. Current cvsnt is fairly 
good at autodetecting and setting text vs binary file type, and enforces 
this across all platforms, so things don't go awry too often. It is in 
my experience more reliable than subversion, which basically relies upon 
file extensions mapping to mime types to identify content. All of which 
is a very much too low standard of accuracy for a version control 
system: I lost many files per year due to the above nonsense, so I worry 
about trying to create a very general transform solution and not making 
it really, really failsafe. Having projects define individual globbing 
patterns is good, double checking the content for sanity is an absolute 
must, but I don't think that is enough. I suspect the solution should 
include round-trip conversion when creating blobs to assure that the 
input can be exactly reconstructed by the inverse transformation (and 
therefore possibly rejecting input with mixed line endings). A similar 
check could be applied on checkout.

Perhaps I'm too paranoid, but I've been burnt way too many times by 
text/binary mode stuff to let this part be trivialized. Maybe it only 
gets enabled by core.ImReallyParanoid, but I want that option.

Mark

Hi,

On Mon, 12 Feb 2007, Mark Levedahl wrote:

Perhaps I'm too paranoid, but I've been burnt way too many times by 
text/binary mode stuff to let this part be trivialized. Maybe it only 
gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally 
opposed to enabling that option by default on all platforms. I am okay 
with .gitattributes (but I would call it .gitfiletypes), but I am _not_ 
okay with git being _too much_ fscked up by Windows. Microsoft has done 
enough harm already.

Ciao,
Dscho

Johannes Schindelin [off-list ref] wrote:

On Mon, 12 Feb 2007, Mark Levedahl wrote:

quoted

Perhaps I'm too paranoid, but I've been burnt way too many times by 
text/binary mode stuff to let this part be trivialized. Maybe it only 
gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally 
opposed to enabling that option by default on all platforms. I am okay 
with .gitattributes (but I would call it .gitfiletypes), but I am _not_ 
okay with git being _too much_ fscked up by Windows. Microsoft has done 
enough harm already.

Indeed; this type of checking should only occur if there is a filter
applied to a file.  Most files in most projects would hopefully
just be considered to be byte streams to Git, like they are today,
and thus not incur any additional overhead, beyond matching their
type to determine they are in fact just a byte stream.

The type could be cached in the index; or at least a single bit
which says "I'm just a byte stream, thanks" so that the matching
only needs to occur during an initial read-tree.

-- 
Shawn.

On Mon, 12 Feb 2007, Shawn O. Pearce wrote:

Johannes Schindelin [off-list ref] wrote:

quoted

On Mon, 12 Feb 2007, Mark Levedahl wrote:

quoted

Perhaps I'm too paranoid, but I've been burnt way too many times by
text/binary mode stuff to let this part be trivialized. Maybe it only
gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally
opposed to enabling that option by default on all platforms. I am okay
with .gitattributes (but I would call it .gitfiletypes), but I am _not_
okay with git being _too much_ fscked up by Windows. Microsoft has done
enough harm already.

Indeed; this type of checking should only occur if there is a filter
applied to a file.  Most files in most projects would hopefully
just be considered to be byte streams to Git, like they are today,
and thus not incur any additional overhead, beyond matching their
type to determine they are in fact just a byte stream.

The type could be cached in the index; or at least a single bit
which says "I'm just a byte stream, thanks" so that the matching
only needs to occur during an initial read-tree.

for the limited case of line endings it may be reasonable to define the internal 
git format to be lf, and if you are running on a platform that uses this nativly 
no transition is needed

one possible way to make this be a general feture is to have the helper script 
have a --needed flag that tells git if it would do anything on the current 
platform or not. this way you don't need to run it (and sanity check it) if it's 
not needed.

David Lang

Johannes Schindelin wrote:

Hi,

On Mon, 12 Feb 2007, Mark Levedahl wrote:

quoted

Perhaps I'm too paranoid, but I've been burnt way too many times by 
text/binary mode stuff to let this part be trivialized. Maybe it only 
gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally 
opposed to enabling that option by default on all platforms. I am okay 
with .gitattributes (but I would call it .gitfiletypes), but I am _not_ 
okay with git being _too much_ fscked up by Windows. Microsoft has done 
enough harm already.

I would assume that none of this crlf stuff exists at all on Linux / 
Unix / Posix, so if done right has zero impact outside of the Windows 
nuthouse. Inside that, folks are already so used to incredible slowness 
in file I/O that I'm not sure the round tripping I suggest as a check 
would be very noticeable, but in any case I fully agree it should be 
optional even there. However, if git could support something that never 
screws up, absolutely guaranteeing data integrity in the presence of 
these transforms, that would be a first in this arena and I believe a 
significant selling point.

Mark

On Mon, Feb 12, 2007 at 07:59:50PM -0500, Mark Levedahl wrote:

fail to set the correct file type somewhere. Current cvsnt is fairly 
good at autodetecting and setting text vs binary file type, and enforces 
this across all platforms, so things don't go awry too often. It is in 

There is obviously much sentiment that this should _not_ be the default
(and I agree). But if arbitrary filters are possible, then you can
theoretically write an 'autocrlf' filter which will try to do the right
thing, and you could set it for some or all files:

  echo '*: autocrlf' >.gitattributes

but it would be off by default. If we implement this, everyone has to
"pay" for .gitattributes (even if you don't use it, we have to look it
up to make sure you're not using it!), but nobody has to pay for any
filters they don't use.

-Peff