Re: [PATCH] index-pack: speed up promisor link recording

On 2026-08-02 at 21:33:15, Arijit Banerjee via GitGitGadget wrote:

From: Arijit Banerjee <redacted>

When indexing a promisor pack, index-pack parses every reconstructed
non-blob object into the shared object model to record its outgoing links.
Since parse_object_buffer() runs under read_mutex, worker threads serialize
while allocating persistent tree, commit, and tag structures that are only
needed to enumerate those links.

Read the links directly from the reconstructed object buffers instead. Keep
the strict and fsck paths unchanged, use worker-local typed oidmaps during
normal promisor indexing, and merge them after the workers exit. Transfer
entries during the merge so that it does not temporarily duplicate the
complete link set.

The typed entries preserve checks previously performed as a side effect of
object parsing. Reject malformed commit and tag headers, conflicting
expected types, and targets whose actual type disagrees when the target is
present in the pack. Preserve commit-graft handling and the existing policy
of recording only subtree entries from trees.

With three runs per version on Debian 12, median end-to-end wall-clock time
for a --filter=blob:none clone of linux.git decreased from 156 seconds to
133 seconds (15%). Trace2 attributed the change to the initial index-pack
--promisor phase, whose median duration decreased from 121 seconds to 98
seconds (19%). System CPU time decreased by 46%.

Two paired spot checks against GitHub showed end-to-end reductions of 18%
and 26%. These measurements include network and server variability and are
therefore corroborating rather than controlled results. A third pair was not
interpretable because the baseline request encountered a transport stall.

A full-clone control showed no material change, taking approximately 256
seconds with either version. This is expected because full clones do not
exercise promisor-link recording.

t5302-pack-index.sh passed with both SHA-1 and SHA-256, while
t0410-partial-clone.sh and t5616-partial-clone.sh also passed. New coverage
checks malformed commit headers, conflicting link types, and mismatched tag
target types.

Signed-off-by: Arijit Banerjee <redacted>
---
    index-pack: speed up promisor link recording

    AI assistance: OpenAI Codex was used to identify the bottleneck and
    assist with the implementation, testing, and benchmark analysis. I
    reviewed the resulting change and take responsibility for this
    submission.

I don't think SubmittingPatches really allows more than trivial changes
written by AI:

    The Developer's Certificate of Origin requires contributors to certify
    that they know the origin of their contributions to the project and
    that they have the right to submit it under the project's license.
    It's not yet clear that this can be legally satisfied when submitting
    significant amount of content that has been generated by AI tools.

    [...]

    To avoid these issues, we will reject anything that looks AI
    generated, that sounds overly formal or bloated, that looks like AI
    slop, that looks good on the surface but makes no sense, or that
    senders don’t understand or cannot explain.

This doesn't look like it's a trivial change, so I don't believe this
patch can be accepted.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA