Re: [PATCH 1/2] vhost: allow multiple workers threads

On 02/19/2012 10:41 PM, Michael S. Tsirkin wrote:

On Fri, Feb 17, 2012 at 05:02:05PM -0600, Anthony Liguori wrote:

quoted

quoted

 This patch allows vhost to have multiple worker threads for devices such as
 virtio-net which may have multiple virtqueues.
 
 Since virtqueues are a lockless ring queue, in an ideal world data is being
 produced by the producer as fast as data is being consumed by the consumer.
 These loops will continue to consume data until none is left.
 
 vhost currently multiplexes the consumer side of the queue on a single thread
 by attempting to read from the queue until everything is read or it cannot
 process anymore.  This means that activity on one queue may stall another queue.

There's actually an attempt to address this: look up
VHOST_NET_WEIGHT in the code. I take it, this isn't effective?

quoted

quoted

 This is exacerbated when using any form of polling to read from the queues (as
 we'll introduce in the next patch).  By spawning a thread per-virtqueue, this
 is addressed.
 
 The only problem with this patch right now is how the wake up of the threads is
 done.  It's essentially a broadcast and we have seen lock contention as a
 result.

On which lock?

quoted

quoted

 We've tried some approaches to signal a single thread but I'm not
 confident that that code is correct yet so I'm only sending the broadcast
 version.

Yes, that looks like an obvious question.

quoted

quoted

 Here are some performance results from this change.  There's a modest
 improvement with stream although a fair bit of variability too.
 
 With RR, there's pretty significant improvements as the instance rate drives up.

Interesting. This was actually tested at one time and we saw
a significant performance improvement from using
a single thread especially with a single
stream in the guest. Profiling indicated that
with a single thread we get too many context
switches between TX and RX, since guest networking
tends to run TX and RX processing on the same
guest VCPU.

Maybe we were wrong or maybe this went away
for some reason. I'll see if this can be reproduced.

I've tried a similar test in Jan. The test uses one dedicated vhost 
thread to handle tx requests and another one for rx. Test result shows 
much degradation as the both of the #exits and #irq are increased. There 
are some differences as I test between local host and guest, and the 
guest does not have very recent virtio changes ( unlocked kick and 
exposing index immediately ). I would try the recent kernel.