rps testing questions

I do a rps(Receive Packet Steering) testing on centos 5.5 with  kernel 2.6.37.
cpu: 8 core Intel.
ethernet adapter: bnx2x

Problem statement:
enable rps with:
echo "ff" > /sys/class/net/eth2/queues/rx-0/rps_cpus.

running 1 instances of netperf TCP_RR: netperf  -t TCP_RR -H 192.168.0.1 -c -C
without rps: 9963.48(Trans Rate per sec)
with rps:  9387.59(Trans Rate per sec)

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

is there something wrong with my test？

Le lundi 17 janvier 2011 à 17:43 +0800, mi wake a écrit :

I do a rps(Receive Packet Steering) testing on centos 5.5 with  kernel 2.6.37.
cpu: 8 core Intel.
ethernet adapter: bnx2x

Problem statement:
enable rps with:
echo "ff" > /sys/class/net/eth2/queues/rx-0/rps_cpus.

bnx2x with one queue only ?

running 1 instances of netperf TCP_RR: netperf  -t TCP_RR -H 192.168.0.1 -c -C
without rps: 9963.48(Trans Rate per sec)
with rps:  9387.59(Trans Rate per sec)

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

Really ? that seems unlikely with your one flow test, unless you _also_
have hardware IRQS hitting all your cpus. (That would be very bad)

is there something wrong with my test？
--

If you test with one flow, RPS brings nothing at all. Better handle the
packet directly from the cpu handling the hardware IRQ (and NAPI)

You better make sure hardware IRQ are on one cpu, instead of many cpus.

On Mon, 2011-01-17 at 17:43 +0800, mi wake wrote:

I do a rps(Receive Packet Steering) testing on centos 5.5 with  kernel 2.6.37.
cpu: 8 core Intel.
ethernet adapter: bnx2x

Problem statement:
enable rps with:
echo "ff" > /sys/class/net/eth2/queues/rx-0/rps_cpus.

running 1 instances of netperf TCP_RR: netperf  -t TCP_RR -H 192.168.0.1 -c -C
without rps: 9963.48(Trans Rate per sec)
with rps:  9387.59(Trans Rate per sec)

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

is there something wrong with my test？

In addition to what Eric said, check the interrupt moderation settings
(ethtool -c/-C options).  One-way latency for a single request/response
test will be at least the interrupt moderation value.

I haven't tested RPS by itself (Solarflare NICs have plenty of hardware
queues) so I don't know whether it can improve latency.  However, RFS
certainly does when there are many flows.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

2011/1/17 Eric Dumazet [off-list ref]:

Le lundi 17 janvier 2011 à 17:43 +0800, mi wake a écrit :

quoted

I do a rps(Receive Packet Steering) testing on centos 5.5 with  kernel 2.6.37.
cpu: 8 core Intel.
ethernet adapter: bnx2x

Problem statement:
enable rps with:
echo "ff" > /sys/class/net/eth2/queues/rx-0/rps_cpus.

bnx2x with one queue only ?

quoted

running 1 instances of netperf TCP_RR: netperf  -t TCP_RR -H 192.168.0.1 -c -C
without rps: 9963.48(Trans Rate per sec)
with rps:  9387.59(Trans Rate per sec)

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

Really ? that seems unlikely with your one flow test, unless you _also_
have hardware IRQS hitting all your cpus. (That would be very bad)

quoted

is there something wrong with my test？
--

If you test with one flow, RPS brings nothing at all. Better handle the
packet directly from the cpu handling the hardware IRQ (and NAPI)

You better make sure hardware IRQ are on one cpu, instead of many cpus.

 I have checked that bnx2x with one queue only and hardware IRQ are on one cpu.
 I do testing again with more flows.when I use ip range from 192.x.x.1
to 192.x.x.200 to send syn packets.
Server can deal with :
 without rps + rfs : 18M/s
 with     rps +rfs : 21M/s.
Maybe  in previous tests,there are less flow. I will continue testing.
Thank you!

Ben Hutchings wrote:

On Mon, 2011-01-17 at 17:43 +0800, mi wake wrote:

quoted

I do a rps(Receive Packet Steering) testing on centos 5.5 with  kernel 2.6.37.
cpu: 8 core Intel.
ethernet adapter: bnx2x

Problem statement:
enable rps with:
echo "ff" > /sys/class/net/eth2/queues/rx-0/rps_cpus.

running 1 instances of netperf TCP_RR: netperf  -t TCP_RR -H 192.168.0.1 -c -C
without rps: 9963.48(Trans Rate per sec)
with rps:  9387.59(Trans Rate per sec)

Presumably there was an increase in service demand corresponding with the drop in transactions per second.

Also, an unsolicited benchmarking style tip or two.  I find it helpful to either do several discrete runs, or use the confidence intervals (global -i and -I options) with the TCP_RR tests when I am looking to compare two settings.  I find a bit more "variability" in the _RR tests than the _STREAM tests.

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dI_002c-Global-26

Pinning netperf/netserver is also something I tend to do, but combining that with  confidence intervals, RPS is kind of difficult - the successive data connections made while running the iterations of the confidence intervals will have different port numbers and so different hashing.  That would cause RPS to put the connections on different cores in turn, which would, in conjunction with netperf/netserver being pinned to a core cause the relationship between where netperf runs and where netserver runs to change.  That will likely result in cache to cache (processor cache) transfers which will definitely up the service demand and drop the single-stream transactions per second.

In theory :) with RFS that should not be an issue since where netperf/netserver are pinned controls where the inbound processing takes place.

We are in a maze of twisty heuristics... :)

quoted

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

is there something wrong with my test？

In addition to what Eric said, check the interrupt moderation settings
(ethtool -c/-C options).  One-way latency for a single request/response
test will be at least the interrupt moderation value.

I haven't tested RPS by itself (Solarflare NICs have plenty of hardware
queues) so I don't know whether it can improve latency.  However, RFS
certainly does when there are many flows.

Is there actually an expectation that either RPS or RFS would improve *latency*?  Multiple-stream throughput certainly, but with the additional work done to spread things around, I wouldn't expect either to improve latency.

happy benchmarking,

rick jones

On Tue, 2011-01-18 at 10:23 -0800, Rick Jones wrote:

Ben Hutchings wrote:

quoted

On Mon, 2011-01-17 at 17:43 +0800, mi wake wrote:

[...]

quoted

quoted

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

is there something wrong with my test？

In addition to what Eric said, check the interrupt moderation settings
(ethtool -c/-C options).  One-way latency for a single request/response
test will be at least the interrupt moderation value.

I haven't tested RPS by itself (Solarflare NICs have plenty of hardware
queues) so I don't know whether it can improve latency.  However, RFS
certainly does when there are many flows.

Is there actually an expectation that either RPS or RFS would improve *latency*? 
  Multiple-stream throughput certainly, but with the additional work done to 
spread things around, I wouldn't expect either to improve latency.

Yes, it seems to make a big improvement to latency when many flows are
active.  Tom told me that one of his benchmarks was 200 * netperf TCP_RR
in parallel, and I've seen over 40% reduction in latency for that.  That
said, allocating more RX queues might also help (sfc currently defaults
to one per processor package rather than one per processor thread, due
to concerns about CPU efficiency).

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Ben Hutchings wrote:

On Tue, 2011-01-18 at 10:23 -0800, Rick Jones wrote:

quoted

Ben Hutchings wrote:

quoted

On Mon, 2011-01-17 at 17:43 +0800, mi wake wrote:

[...]

quoted

quoted

quoted

I do ab and tbench testing also find there is less tps with enable
rps.but,there is more cpu using when with enable rps.when with enable
rps ，softirqs is blanced  on cpus.

is there something wrong with my test？

In addition to what Eric said, check the interrupt moderation settings
(ethtool -c/-C options).  One-way latency for a single request/response
test will be at least the interrupt moderation value.

I haven't tested RPS by itself (Solarflare NICs have plenty of hardware
queues) so I don't know whether it can improve latency.  However, RFS
certainly does when there are many flows.

Is there actually an expectation that either RPS or RFS would improve *latency*?  Multiple-stream throughput certainly, but with the additional work done to spread things around, I wouldn't expect either to improve latency.

Yes, it seems to make a big improvement to latency when many flows are
active.

OK, you and I were using different definitions.  I was speaking to single-stream latency, but didn't say it explicitly (I may have subconsciously thought it was implicit given the OP used a single instance of netperf :).

happy benchmarking,

rick jones

Tom told me that one of his benchmarks was 200 * netperf TCP_RR
in parallel, and I've seen over 40% reduction in latency for that. That
said, allocating more RX queues might also help (sfc currently defaults
to one per processor package rather than one per processor thread, due
to concerns about CPU efficiency).

Ben.