Thread (91 messages) 91 messages, 8 authors, 2011-03-02

Re: SO_REUSEPORT - can it be done in kernel?

From: Thomas Graf <hidden>
Date: 2011-03-01 11:07:14

On Tue, Mar 01, 2011 at 11:33:22AM +0100, Eric Dumazet wrote:
quoted
I retested with net-2.6 w/o Herbert's patch:

named -n 1: 36.9kqps
named:      16.2kqps
Thats better ;)

You could do "cat /proc/net/udp" to check if drops occur on port 53
socket (last column)

But maybe your queryperf is limited to few queries in flight (default is
20 per queryperf instance) 
I tried -q 10, 20, 30, 50, 100. Starting with 20 I see drops, at q=50
queryperf reports 99% drops.

I also tested again on the Intel machine that did ~650kqps using SO_REUSEPORT.

net-2.6: 106.3kqps, 101.2kqps
net-2.6 lockless udp: 251.7kqps, 250.4kqps

I see drops in both test cases occur so I believe the rate supplied by the
clients is sufficient.

The difference is obvious when looking at top and mpstat:

UDP lockless (250kqps):

Cpu0  : 46.4%us, 28.8%sy,  0.0%ni, 24.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  2.0%us,  1.3%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi, 93.6%si,  0.0%st
Cpu2  : 45.9%us, 28.2%sy,  0.0%ni, 25.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 50.0%us, 21.6%sy,  0.0%ni, 28.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 45.4%us, 27.8%sy,  0.0%ni, 26.5%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  : 50.7%us, 23.2%sy,  0.0%ni, 26.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 45.2%us, 28.9%sy,  0.0%ni, 25.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 50.5%us, 22.0%sy,  0.0%ni, 27.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  : 45.3%us, 29.3%sy,  0.0%ni, 25.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  : 50.8%us, 20.8%sy,  0.0%ni, 28.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 : 46.1%us, 27.8%sy,  0.0%ni, 26.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 : 27.2%us, 11.3%sy,  0.0%ni,  3.3%id,  0.0%wa,  0.0%hi, 58.1%si,  0.0%st

05:50:44 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
05:50:44 AM  all   23.86    0.00   13.02    0.22    0.00    6.98    0.00    0.00   55.92
05:50:44 AM    0   26.16    0.00   17.20    0.73    0.00    0.30    0.00    0.00   55.61
05:50:44 AM    1    2.36    0.00    2.11    0.70    0.00   51.97    0.00    0.00   42.87
05:50:44 AM    2   25.90    0.00   16.38    0.32    0.00    0.03    0.00    0.00   57.36
05:50:44 AM    3   28.26    0.00   12.73    0.27    0.00    0.02    0.00    0.00   58.73
05:50:44 AM    4   25.63    0.00   16.04    0.13    0.00    0.03    0.00    0.00   58.17
05:50:44 AM    5   28.19    0.00   12.54    0.17    0.00    0.01    0.00    0.00   59.09
05:50:44 AM    6   25.28    0.00   15.21    0.02    0.00    1.95    0.00    0.00   57.54
05:50:44 AM    7   28.34    0.00   12.40    0.10    0.00    0.01    0.00    0.00   59.14
05:50:44 AM    8   25.70    0.00   15.91    0.01    0.00    0.02    0.00    0.00   58.37
05:50:44 AM    9   28.31    0.00   12.56    0.11    0.00    0.01    0.00    0.00   59.01
05:50:44 AM   10   25.85    0.00   15.65    0.01    0.00    0.02    0.00    0.00   58.47
05:50:44 AM   11   16.11    0.00    7.44    0.10    0.00   29.87    0.00    0.00   46.49

SO_REUSEPORT test (doing 640kqps):

Cpu0  : 57.3%us, 26.5%sy,  0.0%ni,  3.3%id,  0.0%wa,  0.0%hi, 12.9%si,  0.0%st
Cpu1  : 25.7%us, 10.0%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.0%hi, 64.0%si,  0.0%st
Cpu2  : 56.3%us, 28.8%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi, 11.9%si,  0.0%st
Cpu3  : 29.1%us, 10.9%sy,  0.0%ni,  1.3%id,  0.0%wa,  0.0%hi, 58.6%si,  0.0%st
Cpu4  : 57.3%us, 28.5%sy,  0.0%ni,  2.3%id,  0.0%wa,  0.0%hi, 11.9%si,  0.0%st
Cpu5  : 64.8%us, 22.6%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi,  9.6%si,  0.0%st
Cpu6  : 59.0%us, 26.7%sy,  0.0%ni,  2.7%id,  0.0%wa,  0.0%hi, 11.7%si,  0.0%st
Cpu7  : 64.1%us, 22.3%sy,  0.0%ni,  3.7%id,  0.0%wa,  0.0%hi, 10.0%si,  0.0%st
Cpu8  : 57.6%us, 27.5%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi, 11.9%si,  0.0%st
Cpu9  : 65.2%us, 22.2%sy,  0.0%ni,  2.3%id,  0.0%wa,  0.0%hi, 10.3%si,  0.0%st
Cpu10 : 56.9%us, 28.3%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi, 11.8%si,  0.0%st
Cpu11 : 40.2%us, 14.6%sy,  0.0%ni,  2.3%id,  0.0%wa,  0.0%hi, 42.9%si,  0.0%st


Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help