Re: Better value for chunk_size when threaded

On Thu, 6 Dec 2007, Jon Smirl wrote:

On 12/6/07, Nicolas Pitre [off-list ref] wrote:

quoted

On Thu, 6 Dec 2007, Jon Smirl wrote:

quoted

I tried some various ideas out for chunk_size and the best strategy I
found was to simply set it to a constant. How does 20,000 work on
other CPUs?

That depends on the object size.  If you have a repo with big objects
but only 1000 of them for example, then the constant doesn't work.

How about defaulting it to 20,000 and allowing an override? It's not
fatal if we guess wrong, we just want to most common cases to work out
of the box. 20,000 is definitely better than the current window *
1000.

Sure.

... But I think this can be made much better than that with no guessing 
at all.

Say you have 4 threads.  then let's divide the whole object list into 4 
big segments and feed those to each thread.

One thread will always finish before the others.  The idea is to find 
the active thread with the largest amount of remaining objects to 
process at that point, and steal half of them and give that to the 
thread that just finished.  Repeat for each thread that completes its 
segment until everything is done.


Nicolas