On 02/25/2016 03:06 AM, Ingo Molnar wrote:

* Jan Kara[off-list ref]  wrote:

quoted

quoted

quoted

quoted

With an exit microbenchmark that creates a large number of threads,
attachs many inodes to them and then exits. The runtimes of that
microbenchmark with 1000 threads before and after the patch on a 4-socket
Intel E7-4820 v3 system (40 cores, 80 threads) were as follows:

   Kernel            Elapsed Time    System Time
   ------            ------------    -----------
   Vanilla 4.5-rc4      65.29s         82m14s
   Patched 4.5-rc4      22.81s         23m03s

Before the patch, spinlock contention at the inode_sb_list_add() function
at the startup phase and the inode_sb_list_del() function at the exit
phase were about 79% and 93% of total CPU time respectively (as measured
by perf). After the patch, the percpu_list_add() function consumed only
about 0.04% of CPU time at startup phase. The percpu_list_del() function
consumed about 0.4% of CPU time at exit phase. There were still some
spinlock contention, but they happened elsewhere.

While looking through this patch, I have noticed that the
list_for_each_entry_safe() iterations in evict_inodes() and
invalidate_inodes() are actually unnecessary. So if you first apply the
attached patch, you don't have to implement safe iteration variants at all.

As a second comment, I'd note that this patch grows struct inode by 1
pointer. It is probably acceptable for large machines given the speedup but
it should be noted in the changelog. Furthermore for UP or even small SMP
systems this is IMHO undesired bloat since the speedup won't be noticeable.

So for these small systems it would be good if per-cpu list magic would just
fall back to single linked list with a spinlock. Do you think that is
reasonably doable?

Even many 'small' systems tend to be SMP these days.

Yes, I know. But my tablet with 4 ARM cores is unlikely to benefit from this
change either. [...]

I'm not sure about that at all, the above numbers are showing a 3x-4x speedup in
system time, which ought to be noticeable on smaller SMP systems as well.

Waiman, could you please post the microbenchmark?

Thanks,

	Ingo

The microbenchmark that I used is attached.

I do agree that performance benefit will decrease as the number of CPUs 
get smaller. The system that I used for testing have 4 sockets with 40 
cores (80 threads). Dave Chinner had run his fstests on a 16-core system 
(probably 2-socket) which showed modest improvement in performance 
(~4m40s vs 4m30s in runtime).

This patch enables parallel insertion and deletion to/from the inode 
list which used to be a serialized operation. So if that list operation 
is a bottleneck, you will see significant improvement. If it is not, we 
may not notice that much of a difference. For a single-socket 4-core 
system, I agree that the performance benefit, if any, will be limited.

Cheers,
Longman