Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff

David Turner [off-list ref] writes:

quoted

I avoided actual files for two reasons

 - disk error rate is higher than memory one, and we might need
trailing SHA-1 back

This only matters if we ever *read* the mmap off disk.  But that will
rarely happen -- usually, everything will stay in memory.

True, and I do not think I'd terribly mind if we decided to use a
on-disk file mmaped with MAP_SHARED as a more portable substitute
for shm_* for this reason.

Also, we
never worry about disk errors for other git objects (e.g. blobs,
commits, or trees), so it seems silly to worry for the index.

The objects are protected by checksums and fsck will notice errors,
so the argument itself is bogus, I'd think, but you do not have to
depend on that argument to support "let's use a mmaped file as a
more portable substitute for shm*" in the first place, I would say.

quoted

 - access is slow (unless cached, but we can't be sure)

We could solve this (and the other problem) with mlock.

Probably you meant madvise(2)?

For something of a size comparable to the index file held by
index-helper-daemon in-core, I'd expect we wouldn't page too
badly.

On Tue, 2016-04-12 at 11:05 -0700, Junio C Hamano wrote:

quoted

quoted

 - access is slow (unless cached, but we can't be sure)

We could solve this (and the other problem) with mlock.

Probably you meant madvise(2)?

For something of a size comparable to the index file held by
index-helper-daemon in-core, I'd expect we wouldn't page too
badly.

No, I meant mlock -- but I guess that's not portable.  madvise with
MADV_WILLNEED will probably do the job.

On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano [off-list ref] wrote:

quoted

quoted

 - access is slow (unless cached, but we can't be sure)

We could solve this (and the other problem) with mlock.

Probably you meant madvise(2)?

For something of a size comparable to the index file held by
index-helper-daemon in-core, I'd expect we wouldn't page too
badly.

I had a look at linux implementation of madvise(MADV_WILLNEED). All it
does is force populating the entire memory region, which is good. But
I suspect when memory is under pressure, some pages may be reclaimed.
index files in monster repo case can go up to a few hundred megabytes,
chances of being reclaimed rise accordingly. But we can reconsider
mlock() later when/if real problems happen.
-- 
Duy