Re: [PATCH] bitmaps: don't recurse into trees already in the bitmap

On Fri, Jun 18, 2021 at 03:35:05PM +0200, Patrick Steinhardt wrote:

On Fri, Jun 18, 2021 at 08:59:59AM -0400, Jeff King wrote:

quoted

On Wed, Jun 16, 2021 at 02:31:04PM +0200, Patrick Steinhardt wrote:

quoted

quoted

These many-refs scenarios make sense as something that is difficult to
verify using a single fork of an open-source project, but is common in
many closed-source projects that do not use forking to reduce the ref
count per repo.

Agreed. What I typically do to emulate this is to use some version of
following command to create refs for "$n" commits.

    git log --all --format="tformat:create refs/commit/%h %H" |
        shuf | head -n "$n" | git update-ref --stdin

It's obviously not ideal given that resulting refs are distributed at
random. But combined with a sufficiently large repo, it's still helped
me at times to reproduce adverse performance at times.

Yeah, I've done something similar. But I'd caution people into reading
too much into performance tests from something like that. What I've
found over the years is that traversal and bitmap performance can be
somewhat sensitive to tree shape and bitmap placement (which in turn is
often influenced by ref placement, if you are using delta islands or the
recently added pack.preferBitmapTips).

More than once I've spent many head-scratching hours trying to determine
why some real-world repo performs better or worse than expected. :)

I couldn't agree more. I've also had my fair share of weird performance
characteristics in completely unexpected ways. Unfortunately, it has
made me become quite cautious about bitmaps given that they've already
caused their fair share of perfomance regressions.

But your work here actually encourages me to give it another try soonish
and see what kind of repo shapes make them explode this time.

Patrick

Today I've been experimenting with the connectivity check of
git-receive-pack(1) once again to see whether I'm able to get a
performance improvement if the git-rev-list(1) command spawned as part
of the connectivity check uses `--use-bitmap-index`.

Turns out the answer is "no": it has exactly the same performance
characteristics when pushing into a bitmapped repository (linux.git)
compared to the version not using a bitmap index, except for once case
where it performs 70x worse: force-pushing "master~10:master" into a
bitmapped repo takes 11s instead of 0.15s with bitmaps enabled.

Just leaving this here as a note for anybody (or myself) to pick up at a
later point.

Patrick