Re: [PATCH 09/16] documentation: add documentation for the bitmap format

On Mon, Jul 1, 2013 at 11:47 AM, Colby Ranger [off-list ref] wrote:

quoted

But I think we are comparing
apples to steaks here, Vincent is (rightfully) concerned about process
startup performance, whereas our timings were assuming the process was
already running.

I did some timing on loading the reverse index for the kernel and it
is pretty slow (~1200ms). I just submitted a fix to do a bucket sort
and reduced that to ~450ms, which is still slow but much better:
https://eclipse.googlesource.com/jgit/jgit/+/6cc532a43cf28403cb623d3df8600a2542a40a43%5E%21/

A reverse index that is hot in RAM would obviously load in about 0ms.
But a cold load of a reverse index that uses only 4 bytes per object
(as Colby did here) for 3.1M objects could take ~590ms to read from
disk, assuming spinning media moving 20 MiB/s. If 8 byte offsets were
also stored this could be more like 1700ms.

Numbers obviously get better if the spinning media can transfer at 40
MiB/s, now its more like 295ms for 4 bytes/object and 885ms for 12
bytes/object.

I think its still reasonable to compute the reverse index on the fly.
But JGit certainly does have the benefit of reusing it across requests
by relying on process memory based caches. C Git needs to rely on the
kernel buffer cache, which requires this data be written out to a file
to be shared.