Re: Network performance with small packets - continued

On Thursday, March 10, 2011 11:16:11 am Tom Lendacky wrote:

On Thursday, March 10, 2011 09:34:22 am Michael S. Tsirkin wrote:

quoted

On Thu, Mar 10, 2011 at 09:23:42AM -0600, Tom Lendacky wrote:

quoted

On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote:

quoted

On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote:

quoted

As for which CPU the interrupt gets pinned to, that doesn't matter
- see below.

So what hurts us the most is that the IRQ jumps between the VCPUs?

Yes, it appears that allowing the IRQ to run on more than one vCPU
hurts. Without the publish last used index patch, vhost keeps
injecting an irq for every received packet until the guest eventually
turns off notifications.

Are you sure you see that? If yes publish used should help a lot.

I definitely see that.  I ran lockstat in the guest and saw the contention
on the lock when the irq was able to run on either vCPU.  Once the irq was
pinned the contention disappeared.  The publish used index patch should
eliminate the extra irq injections and then the pinning or use of
irqbalance shouldn't be required.  I'm getting a kernel oops during boot
with the publish last used patches that I pulled from the mailing list - I
had to make some changes in order to get them to apply and compile and
might not have done the right things.  Can you re-spin that patchset
against kvm.git?

Here are the results for the publish last used index patch (with the baseline 
provided again for reference).

Here is the KVM baseline (average of six runs):
  Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec
  Exits: 148,444.58 Exits/Sec
  TxCPU: 2.40%  RxCPU: 99.35%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 5,154/5,222
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

Using the publish last used index w/o irqbalance (average of six runs):
  Txn Rate: 112,180.10 Txn/Sec, Pkt Rate: 222,878.33 Pkts/Sec
  Exits: 96,280.11 Exits/Sec
  TxCPU: 1.14%  RxCPU: 99.33%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 3,400/3,400
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

About a 29% increase over baseline.

Using the publish last used index w/  irqbalance (average of six runs):
  Txn Rate: 110,891.12 Txn/Sec, Pkt Rate: 220,315.67 Pkts/Sec
  Exits: 97,190.68 Exits/Sec
  TxCPU: 1.10%  RxCPU: 99.38%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 7,040/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

About a 27% increase over baseline.

Here is data from running without the publish last used index patch but with 
irqbalance running (pinning results were near identical):
  Txn Rate: 107,714.53 Txn/Sec, Pkt Rate: 214,006 Pkts/Sec
  Exits: 121,050.45 Exits/Sec
  TxCPU: 9.61%  RxCPU: 99.45%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 13,975/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

The publish last used index patch provides a 3%-4% improvement while reducing 
the exit rate and interrupt rate in the guest as well as reducing the 
transmitting system CPU% quite dramatically.

quoted

quoted

Because the irq injections end up overlapping we get contention on the
irq_desc_lock_class lock. Here are some results using the "baseline"
setup with irqbalance running.

  Txn Rate: 107,714.53 Txn/Sec, Pkt Rate: 214,006 Pkts/Sec
  Exits: 121,050.45 Exits/Sec
  TxCPU: 9.61%  RxCPU: 99.45%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 13,975/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

About a 24% increase over baseline.  Irqbalance essentially pinned the
virtio irq to CPU0 preventing the irq lock contention and resulting in
nice gains.

OK, so we probably want some form of delayed free for TX
on top, and that should get us nice results already.

quoted

quoted

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html