[PATCH 0/3] speeding up on-demand fetch for blobs in partial clone

On Tue, Sep 06, 2022 at 02:38:41PM -0400, Jeff King wrote:

On Mon, Sep 05, 2022 at 11:17:21AM +0000, 程洋 wrote:

quoted

Sorry, I told you the wrong branch. It should be "android-t-preview-1"
git clone --filter=blob:none --no-local -b android-t-preview-1 grade-plugin

Can you try this one?

Yes, I see more slow-down there. There are many more blobs there, but I
don't think it's really the number of them, but their sizes.

The problem is that both upload-pack and pack-objects are keen to call
parse_object() on their inputs. For commits, etc, that is usually
sensible; we have to parse the object to see what it points to. But for
blobs, the only thing we do is inflate a ton of bytes in order to check
the sha1. That's not really productive here; if there is a bit
corruption, the client will notice it on the receiving side.

So here's a cleaned-up series which makes this a lot faster.

The special sauce is in patch 2, along with timings. The first one is
just preparing, and the final one is a small cleanup it enables.

I based these directly on the current tip of master. There will be a
small conflict in the test script when merging with the "rev-list
--verify-objects" series I just sent in:

  https://lore.kernel.org/git/Yxe0k++LA%2FUfFLF%2F@coredump.intra.peff.net/ (local)

The resolution is to just keep the tests added by both sides.

  [1/3]: parse_object(): allow skipping hash check
  [2/3]: upload-pack: skip parse-object re-hashing of "want" objects
  [3/3]: parse_object(): check commit-graph when skip_hash set

 object.c        | 21 ++++++++++++++++++---
 object.h        |  6 ++++++
 revision.c      | 14 +++-----------
 t/t1450-fsck.sh | 20 ++++++++++++++++++++
 upload-pack.c   |  8 ++------
 5 files changed, 49 insertions(+), 20 deletions(-)

-Peff