fast-import problem importing dos format files under cygwin

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when reading,
overreads by that number of bytes and fails when it tries to read the next
command from the middle.

Attached is a test input stream and crash report generated by fast-import
when reading it. In case mail system damages it along the way despite
being attached to prevent that, the file should be in unix format -- that
is what my cygwin perl outputs by default -- and has CRs only on lines 15
and 16. The unix.txt and dos.txt should only differ that the '.'s in
former are replaced by '^M's in the later (so the data commands are
otherwise same).

Note, that when I convert the file to dos format, it is read as intended.
However, that is inconsistent with rest of the cygwin environment which
generated and expects files in unix format. I use binary mounts (not
converting) and CYGWIN environment variable is empty. My git version is
1.6.0.4 from official Cygwin package.

Is this behaviour intentional workaround for something or a bug?

-- 
                                        - Jan Hudec [off-list ref]

On 3 December 2008, 08:51, Jan Hudec wrote:

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when reading,
overreads by that number of bytes and fails when it tries to read the
next command from the middle.

One addition:

I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the
test, as it was, except it didn't like 'refs/heads/master' as branchname
(and accepted bare 'master', but that created '.git/master').

-- 
                                        - Jan Hudec [off-list ref]

Jan Hudec schrieb:

On 3 December 2008, 08:51, Jan Hudec wrote:

quoted

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when reading,
overreads by that number of bytes and fails when it tries to read the
next command from the middle.

One addition:

I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the
test, as it was, except it didn't like 'refs/heads/master' as branchname
(and accepted bare 'master', but that created '.git/master').

With my current version of MinGW git the import is successful after I
edited test1.gfi to match your description (it had CR on all lines; I
removed all except on lines 15 and 16). The repository content is as one
would it expect given the input. master is
b8ad21c3dc271d43a6e43c261909d6be725fa5b8.

Do you happen to have core.autocrlf set in some way and could it make a
difference for fast-import? I have it unset.

-- Hannes

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):

Jan Hudec schrieb:

quoted

On 3 December 2008, 08:51, Jan Hudec wrote:

quoted

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when
reading,
overreads by that number of bytes and fails when it tries to read the
next command from the middle.

Do you happen to have core.autocrlf set in some way and could it make a
difference for fast-import? I have it unset.

I have it set to false explicitly in global config. Tried with not having
it set at all and gives the same problem. Since the previous version of
MSys Git worked for me, I suspect it's somehow cygwin-related.

-- 
                                        - Jan Hudec [off-list ref]

Jan Hudec [off-list ref] wrote:

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):

quoted

Jan Hudec schrieb:

quoted

On 3 December 2008, 08:51, Jan Hudec wrote:

quoted

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when
reading,
overreads by that number of bytes and fails when it tries to read the
next command from the middle.

Do you happen to have core.autocrlf set in some way and could it make a
difference for fast-import? I have it unset.

I have it set to false explicitly in global config. Tried with not having
it set at all and gives the same problem. Since the previous version of
MSys Git worked for me, I suspect it's somehow cygwin-related.

Huh.  So fast-import *never* does auto-CRLF conversion, even if the
property is set.  It just doesn't make those calls internally.
It blindly copies data from the input stream into the pack.
No exceptions.

fast-import under-reading near CRs and getting misaligned on its
input indicates that the stdio library has given us a FILE* for stdin
which is converting CRLF pairs into LFs, even within an fread() call.

My guess here is fast-import's stdin is set in text mode, but it
really needs to be in binary mode.  fast-import.c never attempts
to correct that when it starts, so on DOS based systems we are
probably totally screwed from the beginning...

-- 
Shawn.

Hi,

On Wed, 3 Dec 2008, Shawn O. Pearce wrote:

Jan Hudec [off-list ref] wrote:

quoted

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):

quoted

Jan Hudec schrieb:

quoted

On 3 December 2008, 08:51, Jan Hudec wrote:

quoted

I have been playing with fast-import in cygwin and I have problems 
importing files with CR/LF line-endings. The size in data command 
is calculated including the CRs and than the file is copied binary 
to the fast-import input stream. However fast-import skips the CRs 
when reading, overreads by that number of bytes and fails when it 
tries to read the next command from the middle.

Do you happen to have core.autocrlf set in some way and could it 
make a difference for fast-import? I have it unset.

I have it set to false explicitly in global config. Tried with not 
having it set at all and gives the same problem. Since the previous 
version of MSys Git worked for me, I suspect it's somehow 
cygwin-related.

Huh.  So fast-import *never* does auto-CRLF conversion, even if the 
property is set.  It just doesn't make those calls internally. It 
blindly copies data from the input stream into the pack. No exceptions.

fast-import under-reading near CRs and getting misaligned on its input 
indicates that the stdio library has given us a FILE* for stdin which is 
converting CRLF pairs into LFs, even within an fread() call.

My guess here is fast-import's stdin is set in text mode, but it really 
needs to be in binary mode.  fast-import.c never attempts to correct 
that when it starts, so on DOS based systems we are probably totally 
screwed from the beginning...

I think you need to set the environment variable

	CYGWIN=binmode

Hth,
Dscho

Hi,

On Wed, Dec 03, 2008 at 17:20:54 +0100, Johannes Schindelin wrote:

Hi,
On Wed, 3 Dec 2008, Shawn O. Pearce wrote:

quoted

Jan Hudec [off-list ref] wrote:

quoted

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):

quoted

Jan Hudec schrieb:

quoted

On 3 December 2008, 08:51, Jan Hudec wrote:

quoted

I have been playing with fast-import in cygwin and I have problems 
importing files with CR/LF line-endings. The size in data command 
is calculated including the CRs and than the file is copied binary 
to the fast-import input stream. However fast-import skips the CRs 
when reading, overreads by that number of bytes and fails when it 
tries to read the next command from the middle.

[...] 
fast-import under-reading near CRs and getting misaligned on its input 
indicates that the stdio library has given us a FILE* for stdin which is 
converting CRLF pairs into LFs, even within an fread() call.

My guess here is fast-import's stdin is set in text mode, but it really 
needs to be in binary mode.  fast-import.c never attempts to correct 
that when it starts, so on DOS based systems we are probably totally 
screwed from the beginning...

Yes, it does indeed sound so. Strange thing is why it would be that way, when
it does not seem to be the case for any other process (eg. the shell will
complain loudly if I feed it a DOS formatted script). The standard input is
simple shell redirect from a file on a binary-mounted filesystem. I'll do
some more cross-checks tomorrow.

I think you need to set the environment variable

	CYGWIN=binmode

Will try. Thanks.

-- 
						 Jan 'Bulb' Hudec [off-list ref]