Re: [PATCH] git-{repack,pack-objects} accept --{pack,blob}-limit to control pack size

Linus Torvalds [off-list ref] wrote:

On Thu, 5 Apr 2007, Shawn O. Pearce wrote:

quoted

For glibc we do try to set _FILE_OFFSET_BITS to 64

I repeat: that's _broken_.

It's in no way portable. It's a glibc horror. It should not be used.

It was a quick hack, but the real way to do it is to use "loff_t" and 
"llseek".

Sure, OK, but that libc function doesn't exist on Mac OS X:

man llseek:

  This function is Linux-specific, and should not be used in programs
  intended to be portable.

So we'd need our own horror to wrap llseek as an lseek fake-alike
anyway.  That's what that glibc horror does, and we didn't have to
write that code.  :-)
 

But there simply isn't any way to do mmap() or pread() portably outside 
the 32-bit area. So there are good reasons why we should just limit 
pack-files to 32-bits on 32-bit architectures.

Not unless your off_t is 64 bits, no.  If it is 64 bits then you
should be able to do a pread or mmap starting past the first 2 GiB.
You just might not be able to ask for a mmap that exceeds 2 GiB in
size, as your size_t may not be that large.  E.g., Darwin/Mac OS X.
Hence the sliding window mmap.

So I think that Dana's approach is just fundamentally correct. Yeah, we 
should probably have a 64-bit index as a *possibility*, but it simply 
isn't a replacement for "keep packs under 2GB in size".

I'm not disagreeing.  Some filesystems (FAT on a USB stick, Dana's
example) just don't want files larger than 2 GiB.  So keeping them
small has a number of advantages.  Plus they are easier to burn on
DVD: 2 packs per DVD.  ;-)

I was simply trying to point out that the mmap code isn't broken
if the off_t is able to handle a file of that size; and if it can't
then other things are broken, like a simple lseek.

-- 
Shawn.