Re: Accelerated receive flow steering (aRFS) for UDP

On Sat, 2016-10-08 at 12:25 +0000, Chopra, Manish wrote:

quoted

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Saturday, October 08, 2016 5:08 AM
To: Chopra, Manish <redacted>
Cc: netdev@vger.kernel.org; maorg@mellanox.com; tom@herbertland.com
Subject: Re: Accelerated receive flow steering (aRFS) for UDP

On Fri, 2016-10-07 at 22:55 +0000, Chopra, Manish wrote:

quoted

Hello Folks,

I am experimenting aRFS with our NIC devices, and for that I have
kernel 4.8.x installed with below config.

CONFIG_RPS=y
CONFIG_RFS_ACCEL=y

# cat /proc/cpuinfo  | grep processor
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7
processor       : 8
processor       : 9
processor       : 10
processor       : 11
processor       : 12
processor       : 13
processor       : 14
processor       : 15

I configured rps_sock_flow_entries  and our NIC rx queues with below
values

echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 4096 > /sys/class/net/p4p1/queues/rx-0/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-1/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-2/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-3/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-4/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-5/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-6/rps_flow_cnt
echo 4096 > /sys/class/net/p4p1/queues/rx-7/rps_flow_cnt

echo ffff > /sys/class/net/p4p1/queues/rx-0/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-1/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-2/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-3/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-4/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-5/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-6/rps_cpus
echo ffff > /sys/class/net/p4p1/queues/rx-7/rps_cpus

Below is IRQ affinity configuration for NIC irqs used.

# cat /proc/irq/67/smp_affinity_list
8
# cat /proc/irq/68/smp_affinity_list
9
# cat /proc/irq/69/smp_affinity_list
10
# cat /proc/irq/70/smp_affinity_list
11
# cat /proc/irq/71/smp_affinity_list
12
# cat /proc/irq/72/smp_affinity_list
13
# cat /proc/irq/73/smp_affinity_list
14
# cat /proc/irq/74/smp_affinity_list
15

Driver has required feature NETIF_F_NTUPLE set, ndo_rx_flow_steer()
registered and I am running UDP multiple connections stream using
netperf to the host where I am experimenting aRFS.

# netperf -V
Netperf version 2.7.0

netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 8,8 -- -m 1470 -P
5001,48512 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 9,9 -- -m 1470 -P
5001,37990 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 10,10 -- -m 1470 -P
5001,40302 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 11,11 -- -m 1470 -P
5001,39071 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 12,12 -- -m 1470 -P
5001,58994 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 13,13 -- -m 1470 -P
5001,59884 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 14,14 -- -m 1470 -P
5001,40282 &
netperf -H 192.168.200.40 -t UDP_STREAM -l 150 -T 15,15 -- -m 1470 -P
5001,56042 &

I see that our registered callback for ndo_rx_flow_steer() "NEVER"
gets invoked for UDP packets, with TCP_STREAM I do see it gets
invoked.
But while running UDP_STREAM I see it gets invoked for some of TCP
packets as netperf also uses TCP managed connections while running
UDP_STREAM.

My initial investigation suspects that while running UDP_STREAM with
netperf, rps_sock_flow_table doesn't get updated, as packets never
reach to the flow of inet_recvmsg()
where it gets updated using sock_rps_record_flow(). Which might be the
reason it never invokes NIC's flow steering handler ?

Please note that when I run UDP stream using "iperf" - I do see that
our registered callback function for flow steering gets invoked for
"UDP" packets.
I am not sure if I am missing something in configuration or something
else which is I am unware of  ?

I appreciate any help for this.

Make sure you use connected UDP flows


netperf -t UDP_STREAM ... -- -N -n

Otherwise, one UDP socket can be involved in millions of 4-tuples (aka
flows)

Hi Eric, I tried with it but it doesn't help with the said issue. I still don't see that our registered handler for flow steering (ndo_rx_flow_steer()) is getting
invoked for UDP packets [ip->protocol = IPPROTO_UDP] .All it gets invoked for TCP packets only [ip->protocol = IPPROTO_TCP]

netperf -H $1 -t UDP_STREAM -l 150 -T 8,8 -- -N -m 1470 -P 5001,48512 &
netperf -H $1 -t UDP_STREAM -l 150 -T 9,9 -- -N -m 1470 -P 5001,37990 &
netperf -H $1 -t UDP_STREAM -l 150 -T 10,10 -- -N -m 1470 -P 5001,40302 &
netperf -H $1 -t UDP_STREAM -l 150 -T 11,11 -- -N -m 1470 -P 5001,39071 &
netperf -H $1 -t UDP_STREAM -l 150 -T 12,12 -- -N -m 1470 -P 5001,58994 &
netperf -H $1 -t UDP_STREAM -l 150 -T 13,13 -- -N -m 1470 -P 5001,59884 &
netperf -H $1 -t UDP_STREAM -l 150 -T 14,14 -- -N -m 1470 -P 5001,40282 &
netperf -H $1 -t UDP_STREAM -l 150 -T 15,15 -- -N -m 1470 -P 5001,56042 &

Thanks !!

Please carefully read what I wrote, and carefully read netperf
documentation.

When you add -n option to netperf, it _will_ use connected UDP sockets
and your problem will vanish.

You added '-N' only, which is useless for UDP_STREAM.

It might help for UDP_RR, but not UDP_STREAM.