Re: Cygwin can't handle huge packfiles?

Hi,

On Wed, 5 Apr 2006, Kees-Jan Dijkzeul wrote:

On 4/3/06, Linus Torvalds [off-list ref] wrote:
[...]

quoted

That's not hugely fundamental, but I didn't expect people to hit it this
quickly. What kind of project has a 1.5GB pack-file _already_? I hope it's
fifteen years of history (so that we'll have another fifteen years before
we'll have to worry about 4GB pack-files ;)

I'm trying to get Git to manage my companies source tree. We're
writing software for digital TV sets. Anyway, the archive is about 5Gb
in size and contains binaries, zip files, excel sheets meeting minutes
and whatnot. So it doesn't compress very well. The 1.5Gb pack file
hardly contains any history at all (five commits or so). On the flip
side, for now I'll be the only one adding to the archive, so at least
it will not grow that fast ;-)

Anyway, to reconstitute the tree, I need very nearly the entire pack,
so limiting the pack size won't do much good, as git will still try to
allocate a total of 1.5Gb memory (which, unfortunately, isn't there
:-)

Inspired by a patch of Alex Riesen (thanks, Alex), I tried to use the
regular mmap for mapping pack files, only to discover that I compile
without defining "NO_MMAP", so I've been using the stock mmap all
along. So now I'm thinking that the cygwin mmap also does a
malloc-and-read, just like git does with NO_MMAP. So I'll continue to
investigate in that direction.

I think cygwin's mmap() is based on the Win32 API equivalent, which could 
mean that it *is* memory mapped, but in a special area (which is smaller 
than 1.5 gigabyte). In this case, it would make sense to limit the pack 
size, thereby having several packs, and mmap() them as they are needed.

Hth,
Dscho