[BUG] git cat-file does not terminate

Hi,

this is some strange behaviour of cat-file:
On a certain file, `git cat-file blob <objectname>` writes an endless 
stream repeating the first 4096 byte of the original file.
cat-file -s and cat-file -t produce correct results.

Even stranger: This only happens with cygwin-git (1.7.4.1).
msysgit (same machine, same repository): works
linux-git (same machine, same repository): works

Even more strange: This only happens with cygwin on a particular machine 
(recent cygwin1.dll 1.7.8) under WinXP/32bit. On another machine, recent 
cygwin, Windows7/64bit it works...

Debugging a bit, I found that the following happens:
In xwrite (wrapper.c), write() is called with the total file size - in 
my case about 87 MB. This call returns -1 and EAGAIN but nevertheless 
writes 4096 byte to the output fd. I don't think that's expected 
behaviour...

I "fixed" it by limiting each write to 64k (thus looping in 
write_in_full) but maybe somebody knows about that cygwin behaviour?

This seems to be the cause of the dreaded "No newline found after blob" 
when running `git svn clone` under cygwin on a repository with large files.

You could argue that this is a cygwin bug but maybe limiting each write 
to a maximum size is a simple workaround.


-Robert

On Fri, Mar 04, 2011 at 02:04:00PM +0100, Robert Wruck wrote:

Hi,

this is some strange behaviour of cat-file:
On a certain file, `git cat-file blob <objectname>` writes an
endless stream repeating the first 4096 byte of the original file.
cat-file -s and cat-file -t produce correct results.

Even stranger: This only happens with cygwin-git (1.7.4.1).
msysgit (same machine, same repository): works
linux-git (same machine, same repository): works

Even more strange: This only happens with cygwin on a particular
machine (recent cygwin1.dll 1.7.8) under WinXP/32bit. On another
machine, recent cygwin, Windows7/64bit it works...

Debugging a bit, I found that the following happens:
In xwrite (wrapper.c), write() is called with the total file size -
in my case about 87 MB. This call returns -1 and EAGAIN but
nevertheless writes 4096 byte to the output fd. I don't think that's
expected behaviour...

I "fixed" it by limiting each write to 64k (thus looping in
write_in_full) but maybe somebody knows about that cygwin behaviour?

This seems to be the cause of the dreaded "No newline found after
blob" when running `git svn clone` under cygwin on a repository with
large files.

You could argue that this is a cygwin bug but maybe limiting each
write to a maximum size is a simple workaround.

Maybe you could post a patch, so everyone can see the technical implications
and discuss the fix?

-Peter

On Fri, Mar 04, 2011 at 04:40:14PM +0100, Peter Baumann wrote:

quoted

I "fixed" it by limiting each write to 64k (thus looping in
write_in_full) but maybe somebody knows about that cygwin behaviour?

This seems to be the cause of the dreaded "No newline found after
blob" when running `git svn clone` under cygwin on a repository with
large files.

You could argue that this is a cygwin bug but maybe limiting each
write to a maximum size is a simple workaround.

Maybe you could post a patch, so everyone can see the technical implications
and discuss the fix?

It would probably look like the patch below, though it really feels like
the right solution is to fix the cygwin bug.

-Peff

---

diff --git a/Makefile b/Makefile
index 4c31d1a..e7d3285 100644
--- a/Makefile
+++ b/Makefile

@@ -167,6 +167,9 @@ all::
 # Define NO_ST_BLOCKS_IN_STRUCT_STAT if your platform does not have st_blocks
 # field that counts the on-disk footprint in 512-byte blocks.
 #
+# Define MAX_WRITE_SIZE to N if your platform has unpredictable results for
+# write() calls larger than N (e.g., cygwin).
+#
 # Define ASCIIDOC7 if you want to format documentation with AsciiDoc 7
 #
 # Define DOCBOOK_XSL_172 if you want to format man pages with DocBook XSL v1.72

@@ -928,6 +931,7 @@ ifeq ($(uname_O),Cygwin)
 	NO_FAST_WORKING_DIRECTORY = UnfortunatelyYes
 	NO_TRUSTABLE_FILEMODE = UnfortunatelyYes
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
+	MAX_WRITE_SIZE=65536
 	# There are conflicting reports about this.
 	# On some boxes NO_MMAP is needed, and not so elsewhere.
 	# Try commenting this out if you suspect MMAP is more efficient

@@ -1495,6 +1499,10 @@ ifdef NO_POSIX_GOODIES
 	BASIC_CFLAGS += -DNO_POSIX_GOODIES
 endif
 
+ifdef MAX_WRITE_SIZE
+	BASIC_CFLAGS += -DMAX_WRITE_SIZE=$(MAX_WRITE_SIZE)
+endif
+
 ifdef BLK_SHA1
 	SHA1_HEADER = "block-sha1/sha1.h"
 	LIB_OBJS += block-sha1/sha1.o

diff --git a/wrapper.c b/wrapper.c
index 056e9d6..a7a2437 100644
--- a/wrapper.c
+++ b/wrapper.c

@@ -133,6 +133,10 @@ ssize_t xread(int fd, void *buf, size_t len)
 ssize_t xwrite(int fd, const void *buf, size_t len)
 {
 	ssize_t nr;
+#ifdef MAX_WRITE_SIZE
+	if (len > MAX_WRITE_SIZE)
+		len = MAX_WRITE_SIZE;
+#endif
 	while (1) {
 		nr = write(fd, buf, len);
 		if ((nr < 0) && (errno == EAGAIN || errno == EINTR))