From: Paweł Staszewski <hidden> Date: 2017-08-11 17:52:39
Hi
I made some tests for performance comparison.
Tested HW (FORWARDING HOST):
Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from 172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
Settings used for FORWARDING HOST (changed param. was only number of RSS combined queues + set affinity assignment for them to fit with first numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 4096 tx 1024
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 10
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that without vlans for about 30-40% (wondering if this is some vlan offloading problem and ixgbe)
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number of cores when 7+ cores are reached
perf top:
PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16.32% [kernel] [k] skb_dst_force
16.30% [kernel] [k] dst_release
15.11% [kernel] [k] rt_cache_valid
12.62% [kernel] [k] ipv4_mtu
5.60% [kernel] [k] do_raw_spin_lock
3.03% [kernel] [k] fib_table_lookup
2.70% [kernel] [k] ip_finish_output2
2.10% [kernel] [k] dev_gro_receive
1.89% [kernel] [k] eth_type_trans
1.81% [kernel] [k] ixgbe_poll
1.15% [kernel] [k] ixgbe_xmit_frame_ring
1.06% [kernel] [k] __build_skb
1.04% [kernel] [k] __dev_queue_xmit
0.97% [kernel] [k] ip_rcv
0.78% [kernel] [k] netif_skb_features
0.74% [kernel] [k] ipt_do_table
0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
0.64% [kernel] [k] ip_forward
0.59% [kernel] [k] __netif_receive_skb_core
0.55% [kernel] [k] dev_hard_start_xmit
0.53% [kernel] [k] ip_route_input_rcu
0.53% [kernel] [k] ip_rcv_finish
0.51% [kernel] [k] page_frag_free
0.50% [kernel] [k] kmem_cache_alloc
0.50% [kernel] [k] udp_v4_early_demux
0.44% [kernel] [k] skb_release_data
0.42% [kernel] [k] inet_gro_receive
0.40% [kernel] [k] sch_direct_xmit
0.39% [kernel] [k] __local_bh_enable_ip
0.33% [kernel] [k] netdev_pick_tx
0.33% [kernel] [k] validate_xmit_skb
0.28% [kernel] [k] fib_validate_source
0.27% [kernel] [k] deliver_ptype_list_skb
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] get_dma_ops
0.22% [kernel] [k] skb_network_protocol
0.21% [kernel] [k] ip_output
0.21% [kernel] [k] vlan_dev_hard_start_xmit
0.20% [kernel] [k] ixgbe_alloc_rx_buffers
0.18% [kernel] [k] nf_hook_slow
0.18% [kernel] [k] apic_timer_interrupt
0.18% [kernel] [k] virt_to_head_page
0.18% [kernel] [k] build_skb
0.16% [kernel] [k] swiotlb_map_page
0.16% [kernel] [k] ip_finish_output
0.16% [kernel] [k] udp4_gro_receive
RESULTS:
CSV format - delimeter ";"
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;1;64;1470912;88247040;1470720;85305530
1;1;64;1470912;88285440;1470977;85335110
2;1;64;1470464;88247040;1470402;85290508
3;1;64;1471424;88262400;1471230;85353728
4;1;64;1468736;88166400;1468672;85201652
5;1;64;1470016;88181760;1469949;85234944
6;1;64;1470720;88247040;1470466;85290624
7;1;64;1471232;88277760;1471167;85346246
8;1;64;1469184;88170240;1469249;85216326
9;1;64;1470592;88227840;1470847;85294394
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;2;64;2413120;144802560;2413245;139975924
1;2;64;2415296;144913920;2415356;140098188
2;2;64;2416768;144898560;2416573;140105670
3;2;64;2418176;145056000;2418110;140261806
4;2;64;2416512;144990720;2416509;140172950
5;2;64;2415168;144860160;2414466;140064780
6;2;64;2416960;144983040;2416833;140190930
7;2;64;2413632;144768000;2413568;140001734
8;2;64;2415296;144898560;2414589;140087168
9;2;64;2416576;144963840;2416892;140190930
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;3;64;3419008;205155840;3418882;198239244
1;3;64;3428032;205585920;3427971;198744234
2;3;64;3425472;205536000;3425344;198677260
3;3;64;3425088;205470720;3425156;198603136
4;3;64;3427648;205693440;3426883;198773888
5;3;64;3426880;205670400;3427392;198796044
6;3;64;3429120;205678080;3430140;198848186
7;3;64;3422976;205355520;3423490;198458136
8;3;64;3423168;205336320;3423486;198495372
9;3;64;3424384;205493760;3425538;198617868
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;4;64;4406464;264364800;4405244;255560296
1;4;64;4404672;264349440;4405122;255541504
2;4;64;4402368;264049920;4403326;255188864
3;4;64;4401344;264076800;4400702;255207134
4;4;64;4385536;263074560;4386620;254312716
5;4;64;4386560;263189760;4385404;254379532
6;4;64;4398784;263857920;4399031;255025288
7;4;64;4407232;264445440;4407998;255637900
8;4;64;4413184;264698880;4413758;255875816
9;4;64;4411328;264526080;4411906;255712372
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;5;64;5094464;305871360;5094464;295657262
1;5;64;5090816;305514240;5091201;295274810
2;5;64;5088384;305387520;5089792;295175108
3;5;64;5079296;304869120;5079484;294680368
4;5;64;5092992;305544960;5094207;295349166
5;5;64;5092416;305502720;5093372;295334260
6;5;64;5080896;304896000;5081090;294677004
7;5;64;5085376;305114880;5086401;294933058
8;5;64;5092544;305575680;5092036;295356938
9;5;64;5093056;305652480;5093832;295449506
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;6;64;5705088;342351360;5705784;330965110
1;6;64;5710272;342743040;5707591;331373952
2;6;64;5703424;342182400;5701826;330776552
3;6;64;5708736;342604800;5707963;331147462
4;6;64;5710144;342654720;5712067;331202910
5;6;64;5712064;342777600;5711361;331292288
6;6;64;5710144;342585600;5708607;331144272
7;6;64;5699840;342021120;5697853;330609222
8;6;64;5701184;342124800;5702909;330653592
9;6;64;5711360;342735360;5713283;331247686
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;7;64;6244416;374603520;6243591;362180072
1;7;64;6230912;374016000;6231490;361534126
2;7;64;6244800;374776320;6244866;362224326
3;7;64;6238720;374376960;6238261;361838510
4;7;64;6218816;373079040;6220413;360683962
5;7;64;6224320;373566720;6225086;361017404
6;7;64;6224000;373570560;6221370;360936088
7;7;64;6210048;372741120;6210627;360212654
8;7;64;6231616;374035200;6231537;361445502
9;7;64;6227840;373724160;6228802;361162752
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;8;64;6251840;375144960;6251849;362609678
1;8;64;6250816;375014400;6250881;362547038
2;8;64;6257728;375432960;6257160;362911104
3;8;64;6255552;375325440;6255622;362822074
4;8;64;6243776;374576640;6243270;362120622
5;8;64;6237184;374296320;6237690;361790080
6;8;64;6240960;374415360;6240714;361927366
7;8;64;6222784;373317120;6223746;360854424
8;8;64;6225920;373593600;6227014;361154980
9;8;64;6238528;374304000;6237701;361845238
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;14;64;6486144;389184000;6486135;376236488
1;14;64;6454912;387390720;6454222;374466734
2;14;64;6441152;386480640;6440431;373572780
3;14;64;6450240;386972160;6450870;374070014
4;14;64;6465600;387997440;6467221;375089654
5;14;64;6448384;386860800;6448000;373980230
6;14;64;6452352;387095040;6452148;374168904
7;14;64;6441984;386507520;6443203;373665058
8;14;64;6456704;387340800;6455744;374429092
9;14;64;6464640;387901440;6465218;374949004
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;6939008;416325120;6938696;402411192
1;16;64;6941952;416444160;6941745;402558918
2;16;64;6960576;417584640;6960707;403698718
3;16;64;6940736;416486400;6941820;402503876
4;16;64;6927680;415741440;6927420;401853870
5;16;64;6929792;415687680;6929917;401839196
6;16;64;6950400;416989440;6950661;403026166
7;16;64;6953664;417216000;6953454;403260544
8;16;64;6948480;416851200;6948800;403023266
9;16;64;6924160;415422720;6924092;401542468
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski [off-list ref] wrote:
Hi
I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
Here are many setting above.
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
Is this a NUMA system?
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
From: Paweł Staszewski <hidden> Date: 2017-08-12 17:28:34
Hi and thanks for reply
W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
Hi
I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
This is not only problem/bug report - but some kind of comparision plus some toughts about possible problems :)
And can help somebody when searching the net for possible expectations :)
Also - dono better list where are the smartest people that know what is going in kernel with networking :)
Next time i will place summary on top - sorry :)
Yes this is a good NIC - will have connectx-4 2x100G by monday so will also do some tests
quoted
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
Traffic generator is pktgen so udp flows - better paste parameters from pktgen:
UDP_MIN=9
UDP_MAX=19
pg_set $dev "dst_min 172.16.0.1"
pg_set $dev "dst_max 172.16.0.100"
# Setup random UDP port src range
#pg_set $dev "flag UDPSRC_RND"
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_MIN"
pg_set $dev "udp_src_max $UDP_MAX"
quoted
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
quoted
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
Yes - enabled flow controll is really bad :)
quoted
ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
rx ring 4096 and tx ring 1024
- this is because have best performance then with average packet size from 64 to 1500 bytes
Can be a little better performance for smaller frames like 64 - with rx ring set to 1024
below 1 core/1 RSS queue with rx ring set to 1024
0;1;64;1530112;91772160;1529919;88724208
1;1;64;1531584;91872000;1531520;88813196
2;1;64;1531392;91895040;1531262;88831930
3;1;64;1530880;91875840;1531201;88783558
4;1;64;1530688;91829760;1530688;88768826
5;1;64;1530432;91810560;1530624;88764940
6;1;64;1530880;91868160;1530878;88787328
7;1;64;1530496;91845120;1530560;88765114
8;1;64;1530496;91837440;1530687;88772538
9;1;64;1530176;91795200;1530496;88735360
so from 1.47Mpps to 1.53Mpps
But with bigger packets > 200 performance is better when rx is set to 4096
quoted
ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
Yes cause i'm changing this parameter also to see if any impact on performance we have
quoted
ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
same here - rx-usecs - setting to 10 was kind of compromise to have good performance with big ans small packet sizes
Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256 (1Core/1RSS queue):
0;1;64;1506304;90424320;1506626;87402868
1;1;64;1505536;90343680;1504830;87321088
2;1;64;1506880;90416640;1507522;87388120
3;1;64;1511040;90700800;1511682;87684864
4;1;64;1511040;90681600;1511102;87662476
5;1;64;1511488;90712320;1511614;87673728
6;1;64;1511296;90700800;1511038;87669900
7;1;64;1513344;90773760;1513280;87751680
8;1;64;1513536;90850560;1513470;87807360
9;1;64;1512128;90696960;1512000;87696000
And rx-usecs set to 1
0;1;64;1533632;92037120;1533504;88954368
1;1;64;1533632;92006400;1533570;88943348
2;1;64;1533504;91994880;1533504;88931980
3;1;64;1532864;91979520;1532674;88902516
4;1;64;1533952;92044800;1534080;88961792
5;1;64;1533888;92048640;1534270;88969100
6;1;64;1533952;92037120;1534082;88969216
7;1;64;1533952;92021760;1534208;88969332
8;1;64;1533056;91983360;1532930;88883724
9;1;64;1533760;92021760;1533886;88946828
rx-useck set to 2
0;1;64;1522432;91334400;1522304;88301056
1;1;64;1521920;91330560;1522496;88286208
2;1;64;1522496;91322880;1522432;88304768
3;1;64;1523456;91422720;1523649;88382762
4;1;64;1527680;91676160;1527424;88601728
5;1;64;1527104;91626240;1526912;88572032
6;1;64;1527424;91641600;1527424;88590592
7;1;64;1526336;91572480;1526912;88523776
8;1;64;1527040;91637760;1526912;88579456
9;1;64;1527040;91595520;1526784;88553472
rx-usecs set to 3
0;1;64;1526272;91549440;1526592;88527488
1;1;64;1526528;91560960;1526272;88516352
2;1;64;1525952;91580160;1525888;88527488
3;1;64;1525504;91511040;1524864;88456960
4;1;64;1526272;91568640;1526208;88494080
5;1;64;1525568;91545600;1525312;88494080
6;1;64;1526144;91584000;1526080;88512640
7;1;64;1525376;91530240;1525376;88482944
8;1;64;1526784;91607040;1526592;88549760
9;1;64;1526208;91560960;1526528;88512640
quoted
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
Here are many setting above.
Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc drop filters - and trying to test also tc-offload )
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
Ok lets give a try :)
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple on
rx-usecs 10
1 CPU / 1 RSS QUEUE
0;1;64;1609344;96537600;1609279;93327104
1;1;64;1608320;96514560;1608256;93293812
2;1;64;1608000;96487680;1608125;93267770
3;1;64;1608320;96522240;1608576;93297524
4;1;64;1605888;96387840;1606211;93148986
5;1;64;1601472;96072960;1601600;92870644
6;1;64;1602624;96180480;1602243;92959674
7;1;64;1601728;96107520;1602113;92907764
8;1;64;1602176;96122880;1602176;92933806
9;1;64;1603904;96253440;1603777;93045208
A little better performance 1.6Mpps
But wondering if disabling tso will have no performance impact for tcp traffic ...
Will try to get some pktgen like pktgen-dpdk that can generate also tcp traffic - to compare this.
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
I set this to on cause i have better performance (a little 10kpps for this test)
below same test as above with tx-nocache-copy off
0;1;64;1591552;95496960;1591230;92313654
1;1;64;1596224;95738880;1595842;92555066
2;1;64;1595456;95700480;1595201;92521774
3;1;64;1595456;95723520;1595072;92528966
4;1;64;1595136;95692800;1595457;92503040
5;1;64;1594624;95631360;1594496;92473402
6;1;64;1596224;95761920;1595778;92551180
7;1;64;1595200;95700480;1595331;92521542
8;1;64;1595584;95692800;1595457;92521426
9;1;64;1594624;95662080;1594048;92469574
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
yes I enabled this cause was thinking that can help with traffic on vlans
below same test with ntuple off
so all settings for ixgbe:
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple off
rx-usecs 10
rx-flow-hash udp4 sdfn
0;1;64;1611840;96691200;1611905;93460794
1;1;64;1610688;96645120;1610818;93427328
2;1;64;1610752;96668160;1610497;93442176
3;1;64;1610624;96664320;1610817;93427212
4;1;64;1610752;96652800;1610623;93412480
5;1;64;1610048;96614400;1610112;93404940
6;1;64;1611264;96641280;1611390;93427212
7;1;64;1611008;96691200;1610942;93468160
8;1;64;1610048;96652800;1609984;93408652
9;1;64;1611136;96641280;1610690;93434636
Performance is a little better
and now with tx-nocache-copy on
0;1;64;1597248;95834880;1597311;92644096
1;1;64;1597888;95865600;1597824;92677446
2;1;64;1597952;95834880;1597822;92644038
3;1;64;1597568;95877120;1597375;92685044
4;1;64;1597184;95827200;1597314;92629190
5;1;64;1597696;95842560;1597565;92625652
6;1;64;1597312;95834880;1597376;92644038
7;1;64;1597568;95873280;1597634;92647924
8;1;64;1598400;95919360;1598849;92699602
9;1;64;1597824;95873280;1598208;92684928
That is weird - so enabling tx-nocache-copy with disabled ntuple have bad performance impact - but with enabled ntuple there is no performance impact
quoted
ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
whan used 16 cores / 16 rss queues - there was better load distribution over all cores when sdfn rx-flow-hash enabled
quoted
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
Is this a NUMA system?
This is 2x CPU 6132 - so have two separate pcie access to the nic - need to check what cpu is assigned to pcie where network card is connected to have network card on local cpu where all irq's are binded
quoted
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
Yes - was thinking that with tx/rx vlan offloading there will be not much performance impact when vlans used.
quoted
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
Hi and thanks for reply
W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
quoted
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
Hi
I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
This is not only problem/bug report - but some kind of comparision plus some toughts about possible problems :)
And can help somebody when searching the net for possible expectations :)
Also - dono better list where are the smartest people that know what is going in kernel with networking :)
Next time i will place summary on top - sorry :)
Yes this is a good NIC - will have connectx-4 2x100G by monday so will also do some tests
quoted
quoted
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
Traffic generator is pktgen so udp flows - better paste parameters from pktgen:
UDP_MIN=9
UDP_MAX=19
pg_set $dev "dst_min 172.16.0.1"
pg_set $dev "dst_max 172.16.0.100"
# Setup random UDP port src range
#pg_set $dev "flag UDPSRC_RND"
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_MIN"
pg_set $dev "udp_src_max $UDP_MAX"
quoted
quoted
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
quoted
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
Yes - enabled flow controll is really bad :)
quoted
quoted
ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
rx ring 4096 and tx ring 1024
- this is because have best performance then with average packet size from 64 to 1500 bytes
Can be a little better performance for smaller frames like 64 - with rx ring set to 1024
below 1 core/1 RSS queue with rx ring set to 1024
0;1;64;1530112;91772160;1529919;88724208
1;1;64;1531584;91872000;1531520;88813196
2;1;64;1531392;91895040;1531262;88831930
3;1;64;1530880;91875840;1531201;88783558
4;1;64;1530688;91829760;1530688;88768826
5;1;64;1530432;91810560;1530624;88764940
6;1;64;1530880;91868160;1530878;88787328
7;1;64;1530496;91845120;1530560;88765114
8;1;64;1530496;91837440;1530687;88772538
9;1;64;1530176;91795200;1530496;88735360
so from 1.47Mpps to 1.53Mpps
But with bigger packets > 200 performance is better when rx is set to 4096
quoted
quoted
ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
Yes cause i'm changing this parameter also to see if any impact on performance we have
quoted
quoted
ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
same here - rx-usecs - setting to 10 was kind of compromise to have good performance with big ans small packet sizes
Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256 (1Core/1RSS queue):
0;1;64;1506304;90424320;1506626;87402868
1;1;64;1505536;90343680;1504830;87321088
2;1;64;1506880;90416640;1507522;87388120
3;1;64;1511040;90700800;1511682;87684864
4;1;64;1511040;90681600;1511102;87662476
5;1;64;1511488;90712320;1511614;87673728
6;1;64;1511296;90700800;1511038;87669900
7;1;64;1513344;90773760;1513280;87751680
8;1;64;1513536;90850560;1513470;87807360
9;1;64;1512128;90696960;1512000;87696000
And rx-usecs set to 1
0;1;64;1533632;92037120;1533504;88954368
1;1;64;1533632;92006400;1533570;88943348
2;1;64;1533504;91994880;1533504;88931980
3;1;64;1532864;91979520;1532674;88902516
4;1;64;1533952;92044800;1534080;88961792
5;1;64;1533888;92048640;1534270;88969100
6;1;64;1533952;92037120;1534082;88969216
7;1;64;1533952;92021760;1534208;88969332
8;1;64;1533056;91983360;1532930;88883724
9;1;64;1533760;92021760;1533886;88946828
rx-useck set to 2
0;1;64;1522432;91334400;1522304;88301056
1;1;64;1521920;91330560;1522496;88286208
2;1;64;1522496;91322880;1522432;88304768
3;1;64;1523456;91422720;1523649;88382762
4;1;64;1527680;91676160;1527424;88601728
5;1;64;1527104;91626240;1526912;88572032
6;1;64;1527424;91641600;1527424;88590592
7;1;64;1526336;91572480;1526912;88523776
8;1;64;1527040;91637760;1526912;88579456
9;1;64;1527040;91595520;1526784;88553472
rx-usecs set to 3
0;1;64;1526272;91549440;1526592;88527488
1;1;64;1526528;91560960;1526272;88516352
2;1;64;1525952;91580160;1525888;88527488
3;1;64;1525504;91511040;1524864;88456960
4;1;64;1526272;91568640;1526208;88494080
5;1;64;1525568;91545600;1525312;88494080
6;1;64;1526144;91584000;1526080;88512640
7;1;64;1525376;91530240;1525376;88482944
8;1;64;1526784;91607040;1526592;88549760
9;1;64;1526208;91560960;1526528;88512640
quoted
quoted
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
Here are many setting above.
Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc drop filters - and trying to test also tc-offload )
quoted
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
Ok lets give a try :)
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple on
rx-usecs 10
1 CPU / 1 RSS QUEUE
0;1;64;1609344;96537600;1609279;93327104
1;1;64;1608320;96514560;1608256;93293812
2;1;64;1608000;96487680;1608125;93267770
3;1;64;1608320;96522240;1608576;93297524
4;1;64;1605888;96387840;1606211;93148986
5;1;64;1601472;96072960;1601600;92870644
6;1;64;1602624;96180480;1602243;92959674
7;1;64;1601728;96107520;1602113;92907764
8;1;64;1602176;96122880;1602176;92933806
9;1;64;1603904;96253440;1603777;93045208
A little better performance 1.6Mpps
But wondering if disabling tso will have no performance impact for tcp traffic ...
Will try to get some pktgen like pktgen-dpdk that can generate also tcp traffic - to compare this.
quoted
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
I set this to on cause i have better performance (a little 10kpps for this test)
below same test as above with tx-nocache-copy off
0;1;64;1591552;95496960;1591230;92313654
1;1;64;1596224;95738880;1595842;92555066
2;1;64;1595456;95700480;1595201;92521774
3;1;64;1595456;95723520;1595072;92528966
4;1;64;1595136;95692800;1595457;92503040
5;1;64;1594624;95631360;1594496;92473402
6;1;64;1596224;95761920;1595778;92551180
7;1;64;1595200;95700480;1595331;92521542
8;1;64;1595584;95692800;1595457;92521426
9;1;64;1594624;95662080;1594048;92469574
quoted
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
yes I enabled this cause was thinking that can help with traffic on vlans
below same test with ntuple off
so all settings for ixgbe:
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple off
rx-usecs 10
rx-flow-hash udp4 sdfn
0;1;64;1611840;96691200;1611905;93460794
1;1;64;1610688;96645120;1610818;93427328
2;1;64;1610752;96668160;1610497;93442176
3;1;64;1610624;96664320;1610817;93427212
4;1;64;1610752;96652800;1610623;93412480
5;1;64;1610048;96614400;1610112;93404940
6;1;64;1611264;96641280;1611390;93427212
7;1;64;1611008;96691200;1610942;93468160
8;1;64;1610048;96652800;1609984;93408652
9;1;64;1611136;96641280;1610690;93434636
Performance is a little better
and now with tx-nocache-copy on
0;1;64;1597248;95834880;1597311;92644096
1;1;64;1597888;95865600;1597824;92677446
2;1;64;1597952;95834880;1597822;92644038
3;1;64;1597568;95877120;1597375;92685044
4;1;64;1597184;95827200;1597314;92629190
5;1;64;1597696;95842560;1597565;92625652
6;1;64;1597312;95834880;1597376;92644038
7;1;64;1597568;95873280;1597634;92647924
8;1;64;1598400;95919360;1598849;92699602
9;1;64;1597824;95873280;1598208;92684928
That is weird - so enabling tx-nocache-copy with disabled ntuple have bad performance impact - but with enabled ntuple there is no performance impact
quoted
quoted
ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
whan used 16 cores / 16 rss queues - there was better load distribution over all cores when sdfn rx-flow-hash enabled
quoted
quoted
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
Is this a NUMA system?
This is 2x CPU 6132 - so have two separate pcie access to the nic - need to check what cpu is assigned to pcie where network card is connected to have network card on local cpu where all irq's are binded
quoted
quoted
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
Yes - was thinking that with tx/rx vlan offloading there will be not much performance impact when vlans used.
quoted
quoted
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
From: Alexander Duyck <hidden> Date: 2017-08-14 00:07:29
On Sat, Aug 12, 2017 at 10:27 AM, Paweł Staszewski
[off-list ref] wrote:
Hi and thanks for reply
W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
quoted
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
[off-list ref] wrote:
quoted
Hi
I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
So I had read through most of this before I realized what it was you
were reporting. As far as the behavior there are a few things going
on. I have some additional comments below but they are mostly based on
what I had read up to that point.
As far as possible issues for item 1. The VLAN adds 4 bytes of data of
the payload, when it is stripped it can result in a packet that is 56
bytes. These missing 8 bytes can cause issues as it forces the CPU to
do a read/modify/write every time the device writes to the 64B cache
line instead of just doing it as a single write. This can be very
expensive and hurt performance. In addition it adds 4 bytes on the
wire, so if you are sending the same 64B packets over the VLAN
interface it is bumping them up to 68B to make room for the VLAN tag.
I am suspecting you are encountering one of these type of issues. You
might try tweaking the packet sizes in increments of 4 to see if there
is a sweet spot that you might be falling out of or into.
Item 2 is a known issue with the NICs supported by ixgbe, at least for
anything 82599 and later. The issue here is that there isn't really an
Rx descriptor cache so to try and optimize performance the hardware
will try to write back as many descriptors it has ready for the ring
requesting writeback. The problem is as you add more rings it means
the writes get smaller as they are triggering more often. So what you
end up seeing is that for each additional ring you add the performance
starts dropping as soon as the rings are no longer being fully
saturated. You can tell this has happened when the CPUs in use
suddenly all stop reporting 100% softirq use. So for example to
perform at line rate with 64B packets you would need something like
XDP and to keep the ring count small, like maybe 2 rings. Any more
than that and the performance will start to drop as you hit PCIe
bottlenecks.
This is not only problem/bug report - but some kind of comparision plus
some toughts about possible problems :)
And can help somebody when searching the net for possible expectations :)
Also - dono better list where are the smartest people that know what is
going in kernel with networking :)
Next time i will place summary on top - sorry :)
Yes this is a good NIC - will have connectx-4 2x100G by monday so will also
do some tests
quoted
quoted
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
Traffic generator is pktgen so udp flows - better paste parameters from
pktgen:
UDP_MIN=9
UDP_MAX=19
pg_set $dev "dst_min 172.16.0.1"
pg_set $dev "dst_max 172.16.0.100"
# Setup random UDP port src range
#pg_set $dev "flag UDPSRC_RND"
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_MIN"
pg_set $dev "udp_src_max $UDP_MAX"
quoted
quoted
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
quoted
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
Yes - enabled flow controll is really bad :)
quoted
quoted
ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
rx ring 4096 and tx ring 1024
- this is because have best performance then with average packet size from
64 to 1500 bytes
The problem is this has huge negative effects on the CPU caches.
Generally less is more. When I perform tests I will usually drop the
ring size for Tx to 128 and Rx to 256. That reduces the descriptor
caches per ring to 1 page each for the Tx and Rx. With an increased
interrupt rate you should be able to service this optimally without
too much issue.
Also for these type of tests the Tx ring never really gets over 64
packets anyway since a single Tx ring is always populated by a single
Rx ring so as long as there isn't any flow control in play the Tx
queue should always be empty when the Rx clean-up begins and it will
only be populated with up to NAPI poll weight worth of packets.
Can be a little better performance for smaller frames like 64 - with rx ring
set to 1024
below 1 core/1 RSS queue with rx ring set to 1024
0;1;64;1530112;91772160;1529919;88724208
1;1;64;1531584;91872000;1531520;88813196
2;1;64;1531392;91895040;1531262;88831930
3;1;64;1530880;91875840;1531201;88783558
4;1;64;1530688;91829760;1530688;88768826
5;1;64;1530432;91810560;1530624;88764940
6;1;64;1530880;91868160;1530878;88787328
7;1;64;1530496;91845120;1530560;88765114
8;1;64;1530496;91837440;1530687;88772538
9;1;64;1530176;91795200;1530496;88735360
so from 1.47Mpps to 1.53Mpps
But with bigger packets > 200 performance is better when rx is set to 4096
This is likely due to the interrupt moderation on the adapter. Instead
of adjusting the ring size up you might try pushing the time between
interrupts down. I have generally found around 25 usecs is best. You
can change the rx-usecs value via ethtool -C to get the rate you want.
You should find that it will perform better that way since you put
less stress on the CPU caches.
quoted
quoted
ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
Yes cause i'm changing this parameter also to see if any impact on
performance we have
quoted
quoted
ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
same here - rx-usecs - setting to 10 was kind of compromise to have good
performance with big ans small packet sizes
From my personal experience I can say that 10 is probably too
aggressive. The logic for trying to find an ideal interrupt rate for
these kind of tests is actually pretty simple. What you want to do is
have the updates coming fast enough that you never hit the point of
descriptor starvation, but at the same time you don't want them coming
too quickly otherwise you limit how many descriptors can be coalesced
into a single PCI DMA write since the descriptors have to be flushed
when an interrupt is triggered.
Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
(1Core/1RSS queue):
0;1;64;1506304;90424320;1506626;87402868
1;1;64;1505536;90343680;1504830;87321088
2;1;64;1506880;90416640;1507522;87388120
3;1;64;1511040;90700800;1511682;87684864
4;1;64;1511040;90681600;1511102;87662476
5;1;64;1511488;90712320;1511614;87673728
6;1;64;1511296;90700800;1511038;87669900
7;1;64;1513344;90773760;1513280;87751680
8;1;64;1513536;90850560;1513470;87807360
9;1;64;1512128;90696960;1512000;87696000
And rx-usecs set to 1
0;1;64;1533632;92037120;1533504;88954368
1;1;64;1533632;92006400;1533570;88943348
2;1;64;1533504;91994880;1533504;88931980
3;1;64;1532864;91979520;1532674;88902516
4;1;64;1533952;92044800;1534080;88961792
5;1;64;1533888;92048640;1534270;88969100
6;1;64;1533952;92037120;1534082;88969216
7;1;64;1533952;92021760;1534208;88969332
8;1;64;1533056;91983360;1532930;88883724
9;1;64;1533760;92021760;1533886;88946828
rx-useck set to 2
0;1;64;1522432;91334400;1522304;88301056
1;1;64;1521920;91330560;1522496;88286208
2;1;64;1522496;91322880;1522432;88304768
3;1;64;1523456;91422720;1523649;88382762
4;1;64;1527680;91676160;1527424;88601728
5;1;64;1527104;91626240;1526912;88572032
6;1;64;1527424;91641600;1527424;88590592
7;1;64;1526336;91572480;1526912;88523776
8;1;64;1527040;91637760;1526912;88579456
9;1;64;1527040;91595520;1526784;88553472
rx-usecs set to 3
0;1;64;1526272;91549440;1526592;88527488
1;1;64;1526528;91560960;1526272;88516352
2;1;64;1525952;91580160;1525888;88527488
3;1;64;1525504;91511040;1524864;88456960
4;1;64;1526272;91568640;1526208;88494080
5;1;64;1525568;91545600;1525312;88494080
6;1;64;1526144;91584000;1526080;88512640
7;1;64;1525376;91530240;1525376;88482944
8;1;64;1526784;91607040;1526592;88549760
9;1;64;1526208;91560960;1526528;88512640
quoted
quoted
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
Here are many setting above.
Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc
drop filters - and trying to test also tc-offload )
quoted
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
Ok lets give a try :)
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple
on
rx-usecs 10
1 CPU / 1 RSS QUEUE
0;1;64;1609344;96537600;1609279;93327104
1;1;64;1608320;96514560;1608256;93293812
2;1;64;1608000;96487680;1608125;93267770
3;1;64;1608320;96522240;1608576;93297524
4;1;64;1605888;96387840;1606211;93148986
5;1;64;1601472;96072960;1601600;92870644
6;1;64;1602624;96180480;1602243;92959674
7;1;64;1601728;96107520;1602113;92907764
8;1;64;1602176;96122880;1602176;92933806
9;1;64;1603904;96253440;1603777;93045208
A little better performance 1.6Mpps
But wondering if disabling tso will have no performance impact for tcp
traffic ...
If you were passing TCP traffic through the router GRO/TSO would
impact things, but for UDP it just adds overhead.
Will try to get some pktgen like pktgen-dpdk that can generate also tcp
traffic - to compare this.
quoted
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
I set this to on cause i have better performance (a little 10kpps for this
test)
below same test as above with tx-nocache-copy off
0;1;64;1591552;95496960;1591230;92313654
1;1;64;1596224;95738880;1595842;92555066
2;1;64;1595456;95700480;1595201;92521774
3;1;64;1595456;95723520;1595072;92528966
4;1;64;1595136;95692800;1595457;92503040
5;1;64;1594624;95631360;1594496;92473402
6;1;64;1596224;95761920;1595778;92551180
7;1;64;1595200;95700480;1595331;92521542
8;1;64;1595584;95692800;1595457;92521426
9;1;64;1594624;95662080;1594048;92469574
If I recall it should have no actual impact one way or the other. The
tx-nocache-copy option should only impact socket traffic, not routing
since if I recall correctly it only impacts copies from userspace.
quoted
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
yes I enabled this cause was thinking that can help with traffic on vlans
below same test with ntuple off
so all settings for ixgbe:
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple
off
rx-usecs 10
rx-flow-hash udp4 sdfn
0;1;64;1611840;96691200;1611905;93460794
1;1;64;1610688;96645120;1610818;93427328
2;1;64;1610752;96668160;1610497;93442176
3;1;64;1610624;96664320;1610817;93427212
4;1;64;1610752;96652800;1610623;93412480
5;1;64;1610048;96614400;1610112;93404940
6;1;64;1611264;96641280;1611390;93427212
7;1;64;1611008;96691200;1610942;93468160
8;1;64;1610048;96652800;1609984;93408652
9;1;64;1611136;96641280;1610690;93434636
Performance is a little better
and now with tx-nocache-copy on
0;1;64;1597248;95834880;1597311;92644096
1;1;64;1597888;95865600;1597824;92677446
2;1;64;1597952;95834880;1597822;92644038
3;1;64;1597568;95877120;1597375;92685044
4;1;64;1597184;95827200;1597314;92629190
5;1;64;1597696;95842560;1597565;92625652
6;1;64;1597312;95834880;1597376;92644038
7;1;64;1597568;95873280;1597634;92647924
8;1;64;1598400;95919360;1598849;92699602
9;1;64;1597824;95873280;1598208;92684928
That is weird - so enabling tx-nocache-copy with disabled ntuple have bad
performance impact - but with enabled ntuple there is no performance impact
I would leave the ntuple feature enabled if you are routing simply
because that disables the ixgbe feature ATR which can have a negative
impact on routing tests (causes reordering).
quoted
quoted
ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
whan used 16 cores / 16 rss queues - there was better load distribution over
all cores when sdfn rx-flow-hash enabled
That is to be expected. The default hash will only has on IPv4
addresses. Enabling the use of UDP ports would allow for more entropy.
If you want similar performance without resorting to hashing on ports
you would have to change the source/destination IP addresses.
quoted
quoted
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
Is this a NUMA system?
This is 2x CPU 6132 - so have two separate pcie access to the nic - need to
check what cpu is assigned to pcie where network card is connected to have
network card on local cpu where all irq's are binded
quoted
quoted
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
Yes - was thinking that with tx/rx vlan offloading there will be not much
performance impact when vlans used.
What is the rate difference? Also did you account for the header size
when noticing that there is a difference in rates? I just want to make
sure we aren't seen an issue where you are expecting a rate of
14.88Mpps when VLAN tags drop the rate due to header overhead down to
something like 14.2Mpps if I recall correctly.
quoted
quoted
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
I would suspect that it may be more than likely the case. One thing
you might look at doing is CPU pinning the interrupts for the NIC in a
1:1 fashion so that the queues are all bound to separate cores without
them sharing between Hyper-threads.
Yes that is always a trade off. Also the ixgbe is limited in terms of
PCIe bus bandwidth. The more queues you add the worse the descriptor
overhead will be. Generally I have found that about 6 queues is ideal.
As you start getting to more than 8 the performance for 64B packets
will start to drop off as each additional queue will hurt the
descriptor cache performance as it starts writing back fewer and fewer
descriptors per write which will increase the PCIe bus overhead for
the writes.
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
From: Paweł Staszewski <hidden> Date: 2017-08-14 15:14:06
W dniu 2017-08-14 o 02:07, Alexander Duyck pisze:
On Sat, Aug 12, 2017 at 10:27 AM, Paweł Staszewski
[off-list ref] wrote:
quoted
Hi and thanks for reply
W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
quoted
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
[off-list ref] wrote:
quoted
Hi
I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
So I had read through most of this before I realized what it was you
were reporting. As far as the behavior there are a few things going
on. I have some additional comments below but they are mostly based on
what I had read up to that point.
As far as possible issues for item 1. The VLAN adds 4 bytes of data of
the payload, when it is stripped it can result in a packet that is 56
bytes. These missing 8 bytes can cause issues as it forces the CPU to
do a read/modify/write every time the device writes to the 64B cache
line instead of just doing it as a single write. This can be very
expensive and hurt performance. In addition it adds 4 bytes on the
wire, so if you are sending the same 64B packets over the VLAN
interface it is bumping them up to 68B to make room for the VLAN tag.
I am suspecting you are encountering one of these type of issues. You
might try tweaking the packet sizes in increments of 4 to see if there
is a sweet spot that you might be falling out of or into.
No this is not a problem with 4byte header or soo
Cause topology is like this
TX generator (pktgen) physical interface no vlan -> RX physical interface (no vlan) [ FORWARDING HOST ] TX vlan interface binded to physical interface -> SINK
below data for packet size 70 (pktgen PKT_SIZE: 70)
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;70;7246720;434749440;7245917;420269856
1;16;70;7249152;434872320;7248885;420434344
2;16;70;7249024;434926080;7249225;420401400
3;16;70;7249984;434952960;7249448;420435736
4;16;70;7251200;435064320;7250990;420495244
5;16;70;7241408;434592000;7241781;420068074
6;16;70;7229696;433689600;7229750;419268196
7;16;70;7236032;434127360;7236133;419669092
8;16;70;7236608;434161920;7236274;419695830
9;16;70;7226496;433578240;7227501;419107826
100% cpu load on all 16 cores
the difference vlan/no vlan currently on this host varries from 40 to even 50% (but cant check if can reach 50% performance degradation cause pktgen can give me only 10Mpps with 70% of cpu load for forwarding host (soo still place to forward maybee at line rate 14Mpps)
Item 2 is a known issue with the NICs supported by ixgbe, at least for
anything 82599 and later. The issue here is that there isn't really an
Rx descriptor cache so to try and optimize performance the hardware
will try to write back as many descriptors it has ready for the ring
requesting writeback. The problem is as you add more rings it means
the writes get smaller as they are triggering more often. So what you
end up seeing is that for each additional ring you add the performance
starts dropping as soon as the rings are no longer being fully
saturated. You can tell this has happened when the CPUs in use
suddenly all stop reporting 100% softirq use. So for example to
perform at line rate with 64B packets you would need something like
XDP and to keep the ring count small, like maybe 2 rings. Any more
than that and the performance will start to drop as you hit PCIe
bottlenecks.
quoted
This is not only problem/bug report - but some kind of comparision plus
some toughts about possible problems :)
And can help somebody when searching the net for possible expectations :)
Also - dono better list where are the smartest people that know what is
going in kernel with networking :)
Next time i will place summary on top - sorry :)
Yes this is a good NIC - will have connectx-4 2x100G by monday so will also
do some tests
quoted
quoted
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
Traffic generator is pktgen so udp flows - better paste parameters from
pktgen:
UDP_MIN=9
UDP_MAX=19
pg_set $dev "dst_min 172.16.0.1"
pg_set $dev "dst_max 172.16.0.100"
# Setup random UDP port src range
#pg_set $dev "flag UDPSRC_RND"
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_MIN"
pg_set $dev "udp_src_max $UDP_MAX"
quoted
quoted
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
quoted
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
Yes - enabled flow controll is really bad :)
quoted
quoted
ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
rx ring 4096 and tx ring 1024
- this is because have best performance then with average packet size from
64 to 1500 bytes
The problem is this has huge negative effects on the CPU caches.
Generally less is more. When I perform tests I will usually drop the
ring size for Tx to 128 and Rx to 256. That reduces the descriptor
caches per ring to 1 page each for the Tx and Rx. With an increased
interrupt rate you should be able to service this optimally without
too much issue.
Also for these type of tests the Tx ring never really gets over 64
packets anyway since a single Tx ring is always populated by a single
Rx ring so as long as there isn't any flow control in play the Tx
queue should always be empty when the Rx clean-up begins and it will
only be populated with up to NAPI poll weight worth of packets.
Will check different ring sizes also to compare this.
quoted
Can be a little better performance for smaller frames like 64 - with rx ring
set to 1024
below 1 core/1 RSS queue with rx ring set to 1024
0;1;64;1530112;91772160;1529919;88724208
1;1;64;1531584;91872000;1531520;88813196
2;1;64;1531392;91895040;1531262;88831930
3;1;64;1530880;91875840;1531201;88783558
4;1;64;1530688;91829760;1530688;88768826
5;1;64;1530432;91810560;1530624;88764940
6;1;64;1530880;91868160;1530878;88787328
7;1;64;1530496;91845120;1530560;88765114
8;1;64;1530496;91837440;1530687;88772538
9;1;64;1530176;91795200;1530496;88735360
so from 1.47Mpps to 1.53Mpps
But with bigger packets > 200 performance is better when rx is set to 4096
This is likely due to the interrupt moderation on the adapter. Instead
of adjusting the ring size up you might try pushing the time between
interrupts down. I have generally found around 25 usecs is best. You
can change the rx-usecs value via ethtool -C to get the rate you want.
You should find that it will perform better that way since you put
less stress on the CPU caches.
quoted
quoted
quoted
ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
Yes cause i'm changing this parameter also to see if any impact on
performance we have
quoted
quoted
ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
same here - rx-usecs - setting to 10 was kind of compromise to have good
performance with big ans small packet sizes
From my personal experience I can say that 10 is probably too
aggressive. The logic for trying to find an ideal interrupt rate for
these kind of tests is actually pretty simple. What you want to do is
have the updates coming fast enough that you never hit the point of
descriptor starvation, but at the same time you don't want them coming
too quickly otherwise you limit how many descriptors can be coalesced
into a single PCI DMA write since the descriptors have to be flushed
when an interrupt is triggered.
quoted
Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
(1Core/1RSS queue):
0;1;64;1506304;90424320;1506626;87402868
1;1;64;1505536;90343680;1504830;87321088
2;1;64;1506880;90416640;1507522;87388120
3;1;64;1511040;90700800;1511682;87684864
4;1;64;1511040;90681600;1511102;87662476
5;1;64;1511488;90712320;1511614;87673728
6;1;64;1511296;90700800;1511038;87669900
7;1;64;1513344;90773760;1513280;87751680
8;1;64;1513536;90850560;1513470;87807360
9;1;64;1512128;90696960;1512000;87696000
And rx-usecs set to 1
0;1;64;1533632;92037120;1533504;88954368
1;1;64;1533632;92006400;1533570;88943348
2;1;64;1533504;91994880;1533504;88931980
3;1;64;1532864;91979520;1532674;88902516
4;1;64;1533952;92044800;1534080;88961792
5;1;64;1533888;92048640;1534270;88969100
6;1;64;1533952;92037120;1534082;88969216
7;1;64;1533952;92021760;1534208;88969332
8;1;64;1533056;91983360;1532930;88883724
9;1;64;1533760;92021760;1533886;88946828
rx-useck set to 2
0;1;64;1522432;91334400;1522304;88301056
1;1;64;1521920;91330560;1522496;88286208
2;1;64;1522496;91322880;1522432;88304768
3;1;64;1523456;91422720;1523649;88382762
4;1;64;1527680;91676160;1527424;88601728
5;1;64;1527104;91626240;1526912;88572032
6;1;64;1527424;91641600;1527424;88590592
7;1;64;1526336;91572480;1526912;88523776
8;1;64;1527040;91637760;1526912;88579456
9;1;64;1527040;91595520;1526784;88553472
rx-usecs set to 3
0;1;64;1526272;91549440;1526592;88527488
1;1;64;1526528;91560960;1526272;88516352
2;1;64;1525952;91580160;1525888;88527488
3;1;64;1525504;91511040;1524864;88456960
4;1;64;1526272;91568640;1526208;88494080
5;1;64;1525568;91545600;1525312;88494080
6;1;64;1526144;91584000;1526080;88512640
7;1;64;1525376;91530240;1525376;88482944
8;1;64;1526784;91607040;1526592;88549760
9;1;64;1526208;91560960;1526528;88512640
quoted
quoted
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
Here are many setting above.
Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc
drop filters - and trying to test also tc-offload )
quoted
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
Ok lets give a try :)
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple
on
rx-usecs 10
1 CPU / 1 RSS QUEUE
0;1;64;1609344;96537600;1609279;93327104
1;1;64;1608320;96514560;1608256;93293812
2;1;64;1608000;96487680;1608125;93267770
3;1;64;1608320;96522240;1608576;93297524
4;1;64;1605888;96387840;1606211;93148986
5;1;64;1601472;96072960;1601600;92870644
6;1;64;1602624;96180480;1602243;92959674
7;1;64;1601728;96107520;1602113;92907764
8;1;64;1602176;96122880;1602176;92933806
9;1;64;1603904;96253440;1603777;93045208
A little better performance 1.6Mpps
But wondering if disabling tso will have no performance impact for tcp
traffic ...
If you were passing TCP traffic through the router GRO/TSO would
impact things, but for UDP it just adds overhead.
quoted
Will try to get some pktgen like pktgen-dpdk that can generate also tcp
traffic - to compare this.
quoted
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
I set this to on cause i have better performance (a little 10kpps for this
test)
below same test as above with tx-nocache-copy off
0;1;64;1591552;95496960;1591230;92313654
1;1;64;1596224;95738880;1595842;92555066
2;1;64;1595456;95700480;1595201;92521774
3;1;64;1595456;95723520;1595072;92528966
4;1;64;1595136;95692800;1595457;92503040
5;1;64;1594624;95631360;1594496;92473402
6;1;64;1596224;95761920;1595778;92551180
7;1;64;1595200;95700480;1595331;92521542
8;1;64;1595584;95692800;1595457;92521426
9;1;64;1594624;95662080;1594048;92469574
If I recall it should have no actual impact one way or the other. The
tx-nocache-copy option should only impact socket traffic, not routing
since if I recall correctly it only impacts copies from userspace.
quoted
quoted
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
yes I enabled this cause was thinking that can help with traffic on vlans
below same test with ntuple off
so all settings for ixgbe:
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple
off
rx-usecs 10
rx-flow-hash udp4 sdfn
0;1;64;1611840;96691200;1611905;93460794
1;1;64;1610688;96645120;1610818;93427328
2;1;64;1610752;96668160;1610497;93442176
3;1;64;1610624;96664320;1610817;93427212
4;1;64;1610752;96652800;1610623;93412480
5;1;64;1610048;96614400;1610112;93404940
6;1;64;1611264;96641280;1611390;93427212
7;1;64;1611008;96691200;1610942;93468160
8;1;64;1610048;96652800;1609984;93408652
9;1;64;1611136;96641280;1610690;93434636
Performance is a little better
and now with tx-nocache-copy on
0;1;64;1597248;95834880;1597311;92644096
1;1;64;1597888;95865600;1597824;92677446
2;1;64;1597952;95834880;1597822;92644038
3;1;64;1597568;95877120;1597375;92685044
4;1;64;1597184;95827200;1597314;92629190
5;1;64;1597696;95842560;1597565;92625652
6;1;64;1597312;95834880;1597376;92644038
7;1;64;1597568;95873280;1597634;92647924
8;1;64;1598400;95919360;1598849;92699602
9;1;64;1597824;95873280;1598208;92684928
That is weird - so enabling tx-nocache-copy with disabled ntuple have bad
performance impact - but with enabled ntuple there is no performance impact
I would leave the ntuple feature enabled if you are routing simply
because that disables the ixgbe feature ATR which can have a negative
impact on routing tests (causes reordering).
quoted
quoted
quoted
ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
whan used 16 cores / 16 rss queues - there was better load distribution over
all cores when sdfn rx-flow-hash enabled
That is to be expected. The default hash will only has on IPv4
addresses. Enabling the use of UDP ports would allow for more entropy.
If you want similar performance without resorting to hashing on ports
you would have to change the source/destination IP addresses.
quoted
quoted
quoted
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
Is this a NUMA system?
This is 2x CPU 6132 - so have two separate pcie access to the nic - need to
check what cpu is assigned to pcie where network card is connected to have
network card on local cpu where all irq's are binded
quoted
quoted
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
Yes - was thinking that with tx/rx vlan offloading there will be not much
performance impact when vlans used.
What is the rate difference? Also did you account for the header size
when noticing that there is a difference in rates? I just want to make
sure we aren't seen an issue where you are expecting a rate of
14.88Mpps when VLAN tags drop the rate due to header overhead down to
something like 14.2Mpps if I recall correctly.
As reply above - the difference is
With vlan: 7Mpps (100%cpu on 16Cores with 16RSS queues)
Without vlan: 10Mpps (70% cpu load on 16Cores with 16RSS queues)
So this is really big difference.
quoted
quoted
quoted
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
I would suspect that it may be more than likely the case. One thing
you might look at doing is CPU pinning the interrupts for the NIC in a
1:1 fashion so that the queues are all bound to separate cores without
them sharing between Hyper-threads.
Yes that is always a trade off. Also the ixgbe is limited in terms of
PCIe bus bandwidth. The more queues you add the worse the descriptor
overhead will be. Generally I have found that about 6 queues is ideal.
As you start getting to more than 8 the performance for 64B packets
will start to drop off as each additional queue will hurt the
descriptor cache performance as it starts writing back fewer and fewer
descriptors per write which will increase the PCIe bus overhead for
the writes.
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
- -
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
Extracted part of attached perf output:
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
From: Eric Dumazet <hidden> Date: 2017-08-14 16:33:09
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
From: Paolo Abeni <pabeni@redhat.com> Date: 2017-08-14 16:57:56
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
I think that the high impact of the above code in this specific test is
mostly due to the following:
- ingress packets with different RSS rx hash lands on different CPUs
- but they use the same dst entry, since the destination IPs belong to
the same subnet
- the dst refcnt cacheline is contented between all the CPUs
Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
flag for vlan if the underlaying device does not have (relevant)
classifier attached? (and clearing it as needed)
Paolo
From: Paweł Staszewski <hidden> Date: 2017-08-15 00:40:07
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
and perf top:
PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
14.25% [kernel] [k] dst_release
14.17% [kernel] [k] skb_dst_force
13.41% [kernel] [k] rt_cache_valid
11.47% [kernel] [k] ip_finish_output2
7.01% [kernel] [k] do_raw_spin_lock
5.07% [kernel] [k] page_frag_free
3.47% [mlx5_core] [k] mlx5e_xmit
2.88% [kernel] [k] fib_table_lookup
2.43% [mlx5_core] [k] skb_from_cqe.isra.32
1.97% [kernel] [k] virt_to_head_page
1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
0.93% [kernel] [k] __dev_queue_xmit
0.87% [kernel] [k] __build_skb
0.84% [kernel] [k] ipt_do_table
0.79% [kernel] [k] ip_rcv
0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
0.78% [kernel] [k] netif_skb_features
0.73% [kernel] [k] __netif_receive_skb_core
0.52% [kernel] [k] dev_hard_start_xmit
0.52% [kernel] [k] build_skb
0.51% [kernel] [k] ip_route_input_rcu
0.50% [kernel] [k] skb_unref
0.49% [kernel] [k] ip_forward
0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.44% [kernel] [k] udp_v4_early_demux
0.41% [kernel] [k] napi_consume_skb
0.40% [kernel] [k] __local_bh_enable_ip
0.39% [kernel] [k] ip_rcv_finish
0.39% [kernel] [k] kmem_cache_alloc
0.38% [kernel] [k] sch_direct_xmit
0.33% [kernel] [k] validate_xmit_skb
0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.29% [kernel] [k] netdev_pick_tx
0.28% [mlx5_core] [k] mlx5e_build_rx_skb
0.27% [kernel] [k] deliver_ptype_list_skb
0.26% [kernel] [k] fib_validate_source
0.26% [mlx5_core] [k] mlx5e_napi_poll
0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.26% [mlx5_core] [k] mlx5e_rx_cache_get
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] skb_network_protocol
0.20% [kernel] [k] nf_hook_slow
0.20% [kernel] [k] vlan_passthru_hard_header
0.20% [kernel] [k] vlan_dev_hard_start_xmit
0.19% [kernel] [k] swiotlb_map_page
0.18% [kernel] [k] compound_head
0.18% [kernel] [k] neigh_connected_output
0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.18% [kernel] [k] ip_output
0.17% [kernel] [k] prefetch_freepointer.isra.70
0.17% [kernel] [k] __slab_free
0.16% [kernel] [k] eth_type_vlan
0.16% [kernel] [k] ip_finish_output
0.15% [kernel] [k] kmem_cache_free_bulk
0.14% [kernel] [k] netif_receive_skb_internal
wondering why this:
1.97% [kernel] [k] virt_to_head_page
is in top...
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
- -
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
Extracted part of attached perf output:
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
From: Paweł Staszewski <hidden> Date: 2017-08-15 00:46:13
W dniu 2017-08-14 o 18:57, Paolo Abeni pisze:
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
quoted
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
I think that the high impact of the above code in this specific test is
mostly due to the following:
- ingress packets with different RSS rx hash lands on different CPUs
yes but isn't this normal ?
everybody that want to ballance load over cores will try tu use as many as possible :)
With some limitations ... best are 6 to 7 RSS queues - so need to use 6 to 7 cpu cores
- but they use the same dst entry, since the destination IPs belong to
the same subnet
typical for ddos - many sources one destination
- the dst refcnt cacheline is contented between all the CPUs
Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
flag for vlan if the underlaying device does not have (relevant)
classifier attached? (and clearing it as needed)
Paolo
From: Eric Dumazet <hidden> Date: 2017-08-15 01:07:10
On Tue, 2017-08-15 at 02:45 +0200, Paweł Staszewski wrote:
W dniu 2017-08-14 o 18:57, Paolo Abeni pisze:
quoted
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
quoted
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
I think that the high impact of the above code in this specific test is
mostly due to the following:
- ingress packets with different RSS rx hash lands on different CPUs
yes but isn't this normal ?
everybody that want to ballance load over cores will try tu use as many
as possible :)
With some limitations ... best are 6 to 7 RSS queues - so need to use 6
to 7 cpu cores
quoted
- but they use the same dst entry, since the destination IPs belong to
the same subnet
typical for ddos - many sources one destination
Nobody hit this issue yet.
We usually change the kernel, given typical workloads.
In this case, we might need per cpu nh_rth_input
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
From: Paweł Staszewski <hidden> Date: 2017-08-15 09:13:06
Hi
Yes it helped - now there is almost no difference when using vlans or not:
10.5Mpps - with vlan
11Mpps - without vlan
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
quoted hunk
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
quoted
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
Hi
Yes it helped - now there is almost no difference when using vlans or not:
10.5Mpps - with vlan
11Mpps - without vlan
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
quoted
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
quoted
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski [off-list ref] wrote:
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
quoted
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
Thanks for being explicit about what you setup is :-)
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
Extracted part of attached perf output:
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
From: Paweł Staszewski <hidden> Date: 2017-08-15 09:31:52
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
quoted
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
quoted
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
Thanks for being explicit about what you setup is :-)
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
Extracted part of attached perf output:
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
On Mon, 14 Aug 2017 18:57:50 +0200
Paolo Abeni [off-list ref] wrote:
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
quoted
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
I think that the high impact of the above code in this specific test is
mostly due to the following:
- ingress packets with different RSS rx hash lands on different CPUs
- but they use the same dst entry, since the destination IPs belong to
the same subnet
- the dst refcnt cacheline is contented between all the CPUs
Good point and explanation Paolo :-)
I changed my pktgen setup to be closer to Pawel's to provoke this
situation some more, and I get closer to provoke this although not as
clearly as Pawel.
A perf diff does show, that the overhead in the VLAN cause originates
from the routing "dst_release" code. Diff Baseline==non-vlan case.
[jbrouer@canyon ~]$ sudo ~/perf diff
# Event 'cycles'
#
# Baseline Delta Abs Shared Object Symbol
# ........ ......... ................ .........................................
#
3.23% +4.32% [kernel.vmlinux] [k] __dev_queue_xmit
+3.43% [kernel.vmlinux] [k] dst_release
13.54% -3.17% [kernel.vmlinux] [k] fib_table_lookup
9.33% -2.73% [kernel.vmlinux] [k] _raw_spin_lock
7.91% -1.75% [ixgbe] [k] ixgbe_poll
+1.64% [8021q] [k] vlan_dev_hard_start_xmit
7.23% -1.26% [ixgbe] [k] ixgbe_xmit_frame_ring
3.34% -1.10% [kernel.vmlinux] [k] eth_type_trans
5.20% +0.97% [kernel.vmlinux] [k] ip_route_input_rcu
1.13% +0.95% [kernel.vmlinux] [k] ip_rcv_finish
2.49% -0.82% [kernel.vmlinux] [k] ip_forward
3.05% -0.80% [kernel.vmlinux] [k] __build_skb
0.44% +0.74% [kernel.vmlinux] [k] __netif_receive_skb
+0.71% [kernel.vmlinux] [k] neigh_connected_output
1.70% +0.68% [kernel.vmlinux] [k] validate_xmit_skb
1.42% +0.67% [kernel.vmlinux] [k] dev_hard_start_xmit
0.49% +0.66% [kernel.vmlinux] [k] netif_receive_skb_internal
+0.62% [kernel.vmlinux] [k] eth_header
+0.57% [ixgbe] [k] ixgbe_tx_ctxtdesc
1.19% -0.55% [kernel.vmlinux] [k] __netdev_pick_tx
2.54% -0.48% [kernel.vmlinux] [k] fib_validate_source
2.83% +0.46% [kernel.vmlinux] [k] ip_finish_output2
1.45% +0.45% [kernel.vmlinux] [k] netif_skb_features
1.66% -0.45% [kernel.vmlinux] [k] napi_gro_receive
0.90% -0.40% [kernel.vmlinux] [k] validate_xmit_skb_list
1.45% -0.39% [kernel.vmlinux] [k] ip_finish_output
+0.36% [8021q] [k] vlan_passthru_hard_header
1.28% -0.33% [kernel.vmlinux] [k] netdev_pick_tx
Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
flag for vlan if the underlaying device does not have (relevant)
classifier attached? (and clearing it as needed)
On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski [off-list ref] wrote:
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
quoted
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
quoted
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
It looks like the mlx5 page recycle mechanism works:
230413 ( 230,413) <= rx7_cache_reuse /sec
+ 926376 ( 926,376) <= rx7_page_reuse /sec
=1156789 (230413+926376)
-1156775 ( 1,156,775) <= rx7_packets /sec
= 14
You can also determine this as there are no counters for:
rx_cache_full or
rx_cache_empty or
rx1_cache_empty
rx1_cache_busy
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
quoted
quoted
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
quoted
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
Thanks for being explicit about what you setup is :-)
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
From: Paweł Staszewski <hidden> Date: 2017-08-15 10:03:16
W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
quoted
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
quoted
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
It looks like the mlx5 page recycle mechanism works:
230413 ( 230,413) <= rx7_cache_reuse /sec
+ 926376 ( 926,376) <= rx7_page_reuse /sec
=1156789 (230413+926376)
-1156775 ( 1,156,775) <= rx7_packets /sec
= 14
You can also determine this as there are no counters for:
rx_cache_full or
rx_cache_empty or
rx1_cache_empty
rx1_cache_busy
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
quoted
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
Thanks for being explicit about what you setup is :-)
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
From: Paweł Staszewski <hidden> Date: 2017-08-15 10:06:47
W dniu 2017-08-15 o 12:02, Paweł Staszewski pisze:
W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
quoted
On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
quoted
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski [off-list ref] wrote:
quoted
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
quoted
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski [off-list ref] wrote:
quoted
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
It looks like the mlx5 page recycle mechanism works:
230413 ( 230,413) <= rx7_cache_reuse /sec
+ 926376 ( 926,376) <= rx7_page_reuse /sec
=1156789 (230413+926376)
-1156775 ( 1,156,775) <= rx7_packets /sec
= 14
You can also determine this as there are no counters for:
rx_cache_full or
rx_cache_empty or
rx1_cache_empty
rx1_cache_busy
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
quoted
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
Thanks for being explicit about what you setup is :-)
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
It seems a little strange that these 4 functions are on the top
I don't see these in my test.
quoted
quoted
quoted
quoted
5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
[...]
So, it doesn't look like there is any backlog queue. Although, this
can be difficult to measure/see this way (as the kernel empty the queue
quickly via bulk deq), also given the small amount of xmit_more which
indicate that the queue was likely very small.
There is a "dropped" counter, which indicate that you likely had a
setup (earlier) where you managed to overflow the qdisc queues.
just see that after changing RSS on nics did't deleted qdisc and added
again:
Here situation with qdisc del / add
tc -s -d qdisc show dev enp175s0f1
qdisc mq 1: root
Sent 43738523966 bytes 683414438 pkt (dropped 0, overlimits 0 requeues 1886)
backlog 0b 0p requeues 1886
qdisc pfifo_fast 0: parent 1:10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 2585011904 bytes 40390811 pkt (dropped 0, overlimits 0 requeues 110)
backlog 0b 0p requeues 110
qdisc pfifo_fast 0: parent 1:f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 2602068416 bytes 40657319 pkt (dropped 0, overlimits 0 requeues 121)
backlog 0b 0p requeues 121
[...]
Exactly as you indicated above, these "dropped" stats came from another
(earlier) test case. (Great that you caught this yourself)
While trying to reproduce you case, I also managed to cause a situation
with qdisc overload. This caused some weird behavior, where I saw
RX=8Mpps and TX only 4Mpps. (I didn't figure out the exact tuning that
caused this, and cannot reproduce it now).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
From: Paweł Staszewski <hidden> Date: 2017-09-09 09:05:10
Hi
Are there any plans to have this fix normally in kernel ?
Or it is mostly only hack - not longterm fix and need to be different ?
All tests that was done shows that without this patch there is about 20-30% network forwarding performance degradation when using vlan interfaces
Thanks
Paweł
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
quoted hunk
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
quoted
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
Hi
Are there any plans to have this fix normally in kernel ?
Or it is mostly only hack - not longterm fix and need to be different ?
All tests that was done shows that without this patch there is about 20-30% network forwarding performance degradation when using vlan interfaces
Thanks
Paweł
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
quoted
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
quoted
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
Hi
Are there any plans to have this fix normally in kernel ?
Or it is mostly only hack - not longterm fix and need to be different ?
All tests that was done shows that without this patch there is about 20-30% network forwarding performance degradation when using vlan interfaces
Thanks
Paweł
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
quoted
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
quoted
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
Any plans for this patch to go normal into the kernel ?
So far im using it for about 3 weeks on all my linux based routers - and
no problems.
Yes, I was about to submit it, as I mentioned it few hours ago to you ;)
Yes i saw Your point 2) in previous emails :)
But there was no patch in previous reply for this so was thinking that maybee too many things to do and You forgot about it :)
Thanks
Paweł
Any plans for this patch to go normal into the kernel ?
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
--
Florian
Any plans for this patch to go normal into the kernel ?
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
Some devices libe bond have it.
Just maybee when there was first patch vlans were not taken into account.
Did not checked all :)
But I know Eric will do :)
From: Eric Dumazet <hidden> Date: 2017-09-21 21:54:57
On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
quoted
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
We support bonding and team already.
Right, so that seems to mostly leave us with DSA at least. What about
other devices that also have IFF_NO_QUEUE set?
--
Florian
From: Eric Dumazet <hidden> Date: 2017-09-22 00:37:17
On Thu, 2017-09-21 at 15:07 -0700, Florian Fainelli wrote:
On 09/21/2017 02:54 PM, Eric Dumazet wrote:
quoted
On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
quoted
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
We support bonding and team already.
Right, so that seems to mostly leave us with DSA at least. What about
other devices that also have IFF_NO_QUEUE set?
It wont work.
loopback has IFF_NO_QUEUE, but you need to keep dst on skbs...
Any plans for this patch to go normal into the kernel ?
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
How far it is from applying this to the kernel ?
So far im using this on all my servers from about 3 months now without problems
From: Eric Dumazet <hidden> Date: 2017-10-18 21:54:08
On Wed, 2017-10-18 at 23:49 +0200, Paweł Staszewski wrote:
How far it is from applying this to the kernel ?
So far im using this on all my servers from about 3 months now without
problems
It is a hack, and does not support properly bonding/team.
( If the real_dev->privflags IFF_XMIT_DST_RELEASE bit changes,
we want to update all the vlans at the same time )
We need something more sophisticated, and I had no time to spend on
this topic recently.
From: Paweł Staszewski <hidden> Date: 2017-10-18 22:45:57
W dniu 2017-10-18 o 23:54, Eric Dumazet pisze:
On Wed, 2017-10-18 at 23:49 +0200, Paweł Staszewski wrote:
quoted
How far it is from applying this to the kernel ?
So far im using this on all my servers from about 3 months now without
problems
It is a hack, and does not support properly bonding/team.
( If the real_dev->privflags IFF_XMIT_DST_RELEASE bit changes,
we want to update all the vlans at the same time )
We need something more sophisticated, and I had no time to spend on
this topic recently.