[PATCH 0/2] 2GB fixes for Windows

On Windows, we can actually have files larger than 2 gigabyte, just not 
using off_t, but off64_t instead.

This came up as issue 194 on the msysGit tracker, and as I first brushed 
the people off saying that it is not an msysGit issue, by way of an 
apology, I started working on this myself.

The first patch adds a test for cloning repositories larger than 2 
gigabyte, which is disabled by default, since it is quite expensive (both 
in terms of time and in terms of space), and since it must fail when the 
underlying filesystem does not allow files larger than 2 gigabyte.

The second patch convinces msysGit (AKA the MinGW port of Git) to make use 
of the 64-bit file offsets.

Johannes Schindelin (2):
  Add an (optional, since expensive) test for >2g clones
  MinGW: 64-bit file offsets

 compat/mingw.c       |    8 +++++---
 compat/mingw.h       |    5 ++++-
 t/t5705-clone-2gb.sh |   45 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 4 deletions(-)
 create mode 100755 t/t5705-clone-2gb.sh

Define GIT_TEST_CLONE_2GB=t if you want the test not to be skipped.

Signed-off-by: Johannes Schindelin <redacted>
---
 t/t5705-clone-2gb.sh |   45 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 45 insertions(+), 0 deletions(-)
 create mode 100755 t/t5705-clone-2gb.sh

diff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh
new file mode 100755
index 0000000..dd4a8ec
--- /dev/null
+++ b/t/t5705-clone-2gb.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+test_description='Test cloning a repository larger than 2 gigabyte'
+. ./test-lib.sh
+
+test -z "$GIT_TEST_CLONE_2GB" &&
+say "Skipping expensive 2GB clone test; enable it with GIT_TEST_CLONE_2GB=t" &&
+test_done &&
+exit
+
+test_expect_success 'setup' '
+
+	git config pack.compression 0 &&
+	git config pack.depth 0 &&
+	blobsize=$((20*1024*1024))
+	blobcount=$((2*1024*1024*1024/$blobsize+1))
+	i=1
+	(while test $i -le $blobcount
+	 do
+		printf "Generating blob $i/$blobcount\r" >&2 &&
+		printf "blob\nmark :$i\ndata $blobsize\n" &&
+		#test-genrandom $i $blobsize &&
+		printf "%-${blobsize}s" $i &&
+		echo "M 100644 :$i $i" >> commit
+		i=$(($i+1)) ||
+		echo $? > exit-status
+	 done &&
+	 echo "commit refs/heads/master" &&
+	 echo "author A U Thor <author@email.com> 123456789 +0000" &&
+	 echo "committer C O Mitter <committer@email.com> 123456789 +0000" &&
+	 echo "data 5" &&
+	 echo ">2gb" &&
+	 cat commit) |
+	git fast-import &&
+	test ! -f exit-status
+
+'
+
+test_expect_success 'clone' '
+
+	git clone --bare --no-hardlinks . clone
+
+'
+
+test_done

-- 
1.6.2.rc1.493.g27ad8

The type 'off_t' should be used everywhere so that the bit-depth of that
type can be adjusted in the standard C library, and you just need to
recompile your program to benefit from the extended precision.

Only that it was not done that way in the MS runtime library.

This patch reroutes off_t to off64_t and provides the other necessary
changes so that finally, clones larger than 2 gigabyte work on Windows
(provided you are on a file system that allows files larger than 2gb).

Initial patch by Sickboy [off-list ref].

Signed-off-by: Johannes Schindelin <redacted>
---
 compat/mingw.c |    8 +++++---
 compat/mingw.h |    5 ++++-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/compat/mingw.c b/compat/mingw.c
index 3dbe6a7..27bcf3f 100644
--- a/compat/mingw.c
+++ b/compat/mingw.c

@@ -46,7 +46,8 @@ static int do_lstat(const char *file_name, struct stat *buf)
 		buf->st_uid = 0;
 		buf->st_nlink = 1;
 		buf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);
-		buf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */
+		buf->st_size = fdata.nFileSizeLow |
+			(((off_t)fdata.nFileSizeHigh)<<32);
 		buf->st_dev = buf->st_rdev = 0; /* not used by Git */
 		buf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));
 		buf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));

@@ -101,7 +102,7 @@ int mingw_fstat(int fd, struct stat *buf)
 	}
 	/* direct non-file handles to MS's fstat() */
 	if (GetFileType(fh) != FILE_TYPE_DISK)
-		return fstat(fd, buf);
+		return _fstati64(fd, buf);
 
 	if (GetFileInformationByHandle(fh, &fdata)) {
 		buf->st_ino = 0;

@@ -109,7 +110,8 @@ int mingw_fstat(int fd, struct stat *buf)
 		buf->st_uid = 0;
 		buf->st_nlink = 1;
 		buf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);
-		buf->st_size = fdata.nFileSizeLow; /* Can't use nFileSizeHigh, since it's not a stat64 */
+		buf->st_size = fdata.nFileSizeLow |
+			(((off_t)fdata.nFileSizeHigh)<<32);
 		buf->st_dev = buf->st_rdev = 0; /* not used by Git */
 		buf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));
 		buf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));

diff --git a/compat/mingw.h b/compat/mingw.h
index a255898..cb9c4d4 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h

@@ -163,11 +163,14 @@ int mingw_rename(const char*, const char*);
 /* Use mingw_lstat() instead of lstat()/stat() and
  * mingw_fstat() instead of fstat() on Windows.
  */
+#define off_t off64_t
+#define stat _stati64
+#define lseek _lseeki64
 int mingw_lstat(const char *file_name, struct stat *buf);
 int mingw_fstat(int fd, struct stat *buf);
 #define fstat mingw_fstat
 #define lstat mingw_lstat
-#define stat(x,y) mingw_lstat(x,y)
+#define stat64(x,y) mingw_lstat(x,y)
 
 int mingw_utime(const char *file_name, const struct utimbuf *times);
 #define utime mingw_utime

-- 
1.6.2.rc1.493.g27ad8

On Thu, Mar 05, 2009 at 05:05:09PM +0100, Johannes Schindelin wrote:

+	(while test $i -le $blobcount
+	 do
+		printf "Generating blob $i/$blobcount\r" >&2 &&
+		printf "blob\nmark :$i\ndata $blobsize\n" &&
+		#test-genrandom $i $blobsize &&
+		printf "%-${blobsize}s" $i &&

Leftover cruft using genrandom? (I'm guessing you tried random at first
to avoid compression, but I think your pack.compression=0 technique is
more sensible).

-Peff

Hi,

On Thu, 5 Mar 2009, Jeff King wrote:

On Thu, Mar 05, 2009 at 05:05:09PM +0100, Johannes Schindelin wrote:

quoted

+	(while test $i -le $blobcount
+	 do
+		printf "Generating blob $i/$blobcount\r" >&2 &&
+		printf "blob\nmark :$i\ndata $blobsize\n" &&
+		#test-genrandom $i $blobsize &&
+		printf "%-${blobsize}s" $i &&

Leftover cruft using genrandom? (I'm guessing you tried random at first
to avoid compression, but I think your pack.compression=0 technique is
more sensible).

Actually, I left it in on purpose.  Yes, it happens to work right now, as 
the packs are built with 0 compression and with 0 deltification.

However, there might be a day when we cannot guarantee anymore that a 
single number padded with spaces to a certain width really makes the pack 
grow by that many bytes.  Then we would need something like test-genrandom 
(which is substantially slower, though).

Interesting random note: it took me quite a while to figure out that both 
pack.compression and pack.depth need to be set to 0.  In hindsight, it is 
obvious, but that does not really help the time I lost due to Windows' 
slowness.

Ciao,
Dscho

On Thu, Mar 05, 2009 at 05:58:23PM +0100, Johannes Schindelin wrote:

quoted

Leftover cruft using genrandom? (I'm guessing you tried random at first
to avoid compression, but I think your pack.compression=0 technique is
more sensible).

Actually, I left it in on purpose.  Yes, it happens to work right now, as 
the packs are built with 0 compression and with 0 deltification.

Fair enough. A comment in the commit message or in the code might have
made that more clear, though...

-Peff

On Donnerstag, 5. März 2009, Johannes Schindelin wrote:

The type 'off_t' should be used everywhere so that the bit-depth of that
type can be adjusted in the standard C library, and you just need to
recompile your program to benefit from the extended precision.

Only that it was not done that way in the MS runtime library.

This patch reroutes off_t to off64_t and provides the other necessary
changes so that finally, clones larger than 2 gigabyte work on Windows
(provided you are on a file system that allows files larger than 2gb).

Initial patch by Sickboy [off-list ref].

Signed-off-by: Johannes Schindelin <redacted>

Acked-by: Johannes Sixt <redacted>

-- Hannes