Re: [PATCH 2/2] index-pack: use streaming interface on large blobs (most of the time)

Nguyễn Thái Ngọc Duy  [off-list ref] writes:

unpack_raw_entry() will not allocate and return decompressed blobs if
they are larger than core.bigFileThreshold.

The blob content is needed by sha1_object() in some cases. When we do
need the blob content, we put it back in core with
get_data_from_pack(). However we rarely need that in pratice.

The first case is when we find an in-repo blob with the same SHA-1. We
need to do collision test, byte-on-byte. Normally (e.g. in
fetch/pull/clone) this does not happen because git avoid to send
objects that client already has.

Perhaps the codepath that performs the byte-for-byte comparison can be
taught to stream from the received pack data and whatever was already
in the repository, using the streaming interface?  That way you do not
have to hold all of the both objects at the same time in core, no?

On Sat, May 19, 2012 at 5:20 AM, Junio C Hamano [off-list ref] wrote:

Nguyễn Thái Ngọc Duy  [off-list ref] writes:

quoted

unpack_raw_entry() will not allocate and return decompressed blobs if
they are larger than core.bigFileThreshold.

The blob content is needed by sha1_object() in some cases. When we do
need the blob content, we put it back in core with
get_data_from_pack(). However we rarely need that in pratice.

The first case is when we find an in-repo blob with the same SHA-1. We
need to do collision test, byte-on-byte. Normally (e.g. in
fetch/pull/clone) this does not happen because git avoid to send
objects that client already has.

Perhaps the codepath that performs the byte-for-byte comparison can be
taught to stream from the received pack data and whatever was already
in the repository, using the streaming interface?  That way you do not
have to hold all of the both objects at the same time in core, no?

Sure. But that does not happen until you are attacked. Maybe later.

verify-pack'ing also runs into this, but that's another issue and
should be fixed separately (hopefully by verify-pack users).
-- 
Duy