Re: [PATCH v5 2/5] read-cache: load cache extensions on a worker thread

On 9/15/2018 6:24 AM, Duy Nguyen wrote:

On Sat, Sep 15, 2018 at 12:22 PM Duy Nguyen [off-list ref] wrote:

quoted

quoted

@@ -1944,6 +1993,26 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)
         istate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));
         istate->initialized = 1;

+       p.istate = istate;
+       p.mmap = mmap;
+       p.mmap_size = mmap_size;
+
+#ifndef NO_PTHREADS
+       nr_threads = git_config_get_index_threads();
+       if (!nr_threads)
+               nr_threads = online_cpus();
+
+       if (nr_threads >= 2) {
+               extension_offset = read_eoie_extension(mmap, mmap_size);
+               if (extension_offset) {

One more thing I forgot. If the extension area is small enough, then
we should not need to create a thread to parse extensions in parallel.
We should know roughly how much work we need because we know the total
size of all extensions.

The only extensions I found to be significant enough to be helped by a separate thread was the cache tree.  Since the size of the cache tree is driven by the number of files in the repo, I think the existing THREAD_COST logic (that comes in the next patch of the series) is a sufficient proxy.  Basically, if you have enough cache entries to be benefited by threading, your extensions (driven by the cache tree) are probably also big enough to warrant a thread.

quoted

quoted

+                       /* create a thread to load the index extensions */

Pointless comment. It's pretty clear from the pthread_create() below
thanks to good function naming. Please remove.

quoted

+                       p.src_offset = extension_offset;
+                       if (pthread_create(&p.pthread, NULL, load_index_extensions, &p))
+                               die(_("unable to create load_index_extensions_thread"));
+               }
+       }
+#endif
+
         if (istate->version == 4) {
                 previous_name = &previous_name_buf;
                 mem_pool_init(&istate->ce_mem_pool,