Re: Cygwin can't handle huge packfiles?

On Wed, Apr 05, 2006 at 05:08:44PM -0400, Christopher Faylor wrote:

On Wed, Apr 05, 2006 at 04:14:20PM +0200, Johannes Schindelin wrote:

quoted

quoted

Inspired by a patch of Alex Riesen (thanks, Alex), I tried to use the
regular mmap for mapping pack files, only to discover that I compile
without defining "NO_MMAP", so I've been using the stock mmap all
along. So now I'm thinking that the cygwin mmap also does a
malloc-and-read, just like git does with NO_MMAP. So I'll continue to
investigate in that direction.

I think cygwin's mmap() is based on the Win32 API equivalent, which could 
mean that it *is* memory mapped, but in a special area (which is smaller 
than 1.5 gigabyte). In this case, it would make sense to limit the pack 
size, thereby having several packs, and mmap() them as they are needed.

Yes, cygwin's mmap uses CreateFileMapping and MapViewOfFile.  IIRC,
Windows might have a 2G limitation lurking under the hood somewhere but
I think that might be tweakable with some registry setting.

Windows places its DLLs criss-cross through the memory space because
every DLL on the system has its own preferred place to be loaded (the
base address). This severely limits the amount of largest contiguous
memory block available, which is needed for one mmap() I think.

Several solutions exist:
  - enlarge the address space with the /3GB boot flag in boot.ini
  - rebase all DLLs with REBASE.EXE (part of platform sdk) .
    Just make them the same and fix them to a low address.
    Problem is rebasing system dlls since those are locked by the system.
  - at start of program before other DLLs are loaded,
    reserve an as large part of the memory as possible with
    VirtualAlloc()

-- 
Rutger Nijlunsing ---------------------------------- eludias ed dse.nl
never attribute to a conspiracy which can be explained by incompetence
----------------------------------------------------------------------