Re: [PATCH] Teach "git add" and friends to be paranoid

On Mon, 22 Feb 2010, Junio C Hamano wrote:

Nicolas Pitre [off-list ref] writes:

quoted

We do have to make two separate passes anyway.  The first pass is to 
hash the data only, and if that hash already exists in the object store 
then we call it done and skip over the deflate process which is still 
the dominant cost.  And that happens quite often.

However, with a really large file, then it becomes advantageous to 
simply do the hash and deflate in parallel one chunk at a time, and 
simply discard the newly created objects if it happens to already 
exists.  That's the whole idea behind the newly introduced 
core.bigFileThreshold config variable (but the code to honor it in 
sha1_file.c doesn't exist yet).

The core.bigFileThreshold could be used in sha1_file.c to decide writing
straight into a new packfile; that would avoid both the later repacking
cost and the cross directory rename issue for loose object files.

Yep, that's the plan.  Let's find the time now.


Nicolas