Hi,
On Broadcom STB chips using bcmsysport.c and bcm_sf2.c we have an out of
band HW mechanism (not using per-flow pause frames) where we can have
the integrated network switch backpressure the CPU Ethernet controller
which translates in completing TX packets interrupts at the appropriate
pace and therefore get flow control applied end-to-end from the host CPU
port towards any downstream port. At least that is the premise and this
works reasonably well.
This has a few drawbacks in that each of the bcmsysport TX queues need
to semi-statically map to their switch port output queues such that the
switch can calculate buffer occupancy and report congestion status,
which prompted this email [1] but this is tangential and is a policy not
a mechanism issue.
[1]: https://www.spinics.net/lists/netdev/msg448153.html
This is useful when your CPU / integrated switch links up at 1Gbits/sec
internally, and tries to push 1Gbits/sec worth of UDP traffic to e.g: a
downstream port linking at 100Mbits/sec, which could happen depending on
what you have connected to this device.
Now the problem that I am facing, is the following:
- net.core.wmem_default = 160KB (default value)
- using iperf -b 800M -u towards an iperf UDP server with the physical
link to that server established at 100Mbits/sec
- iperf does synchronous write(2) AFAICT so this gives it flow control
- using the default duration of 10s, you can barely see any packet loss
from one run to another
- the longer the run, the higher you are going to see some packet loss,
usually in the range of ~0.15% top
The transmit flow looks like this:
gphy (net/dsa/slave.c::dsa_slave_xmit, IFF_NO_QUEUE device)
-> eth0 (drivers/net/ethernet/broadcom/bcmsysport.c, "regular" network
device)
I can clearly see that the network stack pushed N UDP packets (Udp and
Ip counters in /proc/net/snmp concur) however what the driver
transmitted and what the switch transmistted is N - M, and matches the
packet loss reported by the UDP server. I don't measure any SndbufErrors
which is not making sense yet.
If I reduce the default socket size to say, 10x less than 160KB, 16KB,
then I either don't see any packet loss at 100Mbits/sec for 5 minutes or
more, or just very very little, down to 0.001%. Now if I repeat the
experiment with the physical link at 10Mbits/sec, same thing, the 16KB
wmem_default setting is no longer working and we need to lower the
socket write buffer size again.
So what I am wondering is:
- do I have an obvious flow control problem in my network driver that
usually does not lead to packet loss, but may sometime happen?
- why would lowering the socket write size appear to masquerade or solve
this problem?
I can consistently reproduce this across several kernel versions, 4.1,
4.9 and latest net-next and therefore can also test patches.
Thanks for reading thus far!
--
Florian
From: Eric Dumazet <hidden> Date: 2017-08-23 22:26:52
On Wed, 2017-08-23 at 13:02 -0700, Florian Fainelli wrote:
Hi,
On Broadcom STB chips using bcmsysport.c and bcm_sf2.c we have an out of
band HW mechanism (not using per-flow pause frames) where we can have
the integrated network switch backpressure the CPU Ethernet controller
which translates in completing TX packets interrupts at the appropriate
pace and therefore get flow control applied end-to-end from the host CPU
port towards any downstream port. At least that is the premise and this
works reasonably well.
This has a few drawbacks in that each of the bcmsysport TX queues need
to semi-statically map to their switch port output queues such that the
switch can calculate buffer occupancy and report congestion status,
which prompted this email [1] but this is tangential and is a policy not
a mechanism issue.
[1]: https://www.spinics.net/lists/netdev/msg448153.html
This is useful when your CPU / integrated switch links up at 1Gbits/sec
internally, and tries to push 1Gbits/sec worth of UDP traffic to e.g: a
downstream port linking at 100Mbits/sec, which could happen depending on
what you have connected to this device.
Now the problem that I am facing, is the following:
- net.core.wmem_default = 160KB (default value)
- using iperf -b 800M -u towards an iperf UDP server with the physical
link to that server established at 100Mbits/sec
- iperf does synchronous write(2) AFAICT so this gives it flow control
- using the default duration of 10s, you can barely see any packet loss
from one run to another
- the longer the run, the higher you are going to see some packet loss,
usually in the range of ~0.15% top
The transmit flow looks like this:
gphy (net/dsa/slave.c::dsa_slave_xmit, IFF_NO_QUEUE device)
-> eth0 (drivers/net/ethernet/broadcom/bcmsysport.c, "regular" network
device)
I can clearly see that the network stack pushed N UDP packets (Udp and
Ip counters in /proc/net/snmp concur) however what the driver
transmitted and what the switch transmistted is N - M, and matches the
packet loss reported by the UDP server. I don't measure any SndbufErrors
which is not making sense yet.
If I reduce the default socket size to say, 10x less than 160KB, 16KB,
then I either don't see any packet loss at 100Mbits/sec for 5 minutes or
more, or just very very little, down to 0.001%. Now if I repeat the
experiment with the physical link at 10Mbits/sec, same thing, the 16KB
wmem_default setting is no longer working and we need to lower the
socket write buffer size again.
So what I am wondering is:
- do I have an obvious flow control problem in my network driver that
usually does not lead to packet loss, but may sometime happen?
- why would lowering the socket write size appear to masquerade or solve
this problem?
I can consistently reproduce this across several kernel versions, 4.1,
4.9 and latest net-next and therefore can also test patches.
Thanks for reading thus far!
Have you checked qdisc counters ? Maybe drops happen there.
tc -s qdisc show
On Wed, 2017-08-23 at 13:02 -0700, Florian Fainelli wrote:
quoted
Hi,
On Broadcom STB chips using bcmsysport.c and bcm_sf2.c we have an out of
band HW mechanism (not using per-flow pause frames) where we can have
the integrated network switch backpressure the CPU Ethernet controller
which translates in completing TX packets interrupts at the appropriate
pace and therefore get flow control applied end-to-end from the host CPU
port towards any downstream port. At least that is the premise and this
works reasonably well.
This has a few drawbacks in that each of the bcmsysport TX queues need
to semi-statically map to their switch port output queues such that the
switch can calculate buffer occupancy and report congestion status,
which prompted this email [1] but this is tangential and is a policy not
a mechanism issue.
[1]: https://www.spinics.net/lists/netdev/msg448153.html
This is useful when your CPU / integrated switch links up at 1Gbits/sec
internally, and tries to push 1Gbits/sec worth of UDP traffic to e.g: a
downstream port linking at 100Mbits/sec, which could happen depending on
what you have connected to this device.
Now the problem that I am facing, is the following:
- net.core.wmem_default = 160KB (default value)
- using iperf -b 800M -u towards an iperf UDP server with the physical
link to that server established at 100Mbits/sec
- iperf does synchronous write(2) AFAICT so this gives it flow control
- using the default duration of 10s, you can barely see any packet loss
from one run to another
- the longer the run, the higher you are going to see some packet loss,
usually in the range of ~0.15% top
The transmit flow looks like this:
gphy (net/dsa/slave.c::dsa_slave_xmit, IFF_NO_QUEUE device)
-> eth0 (drivers/net/ethernet/broadcom/bcmsysport.c, "regular" network
device)
I can clearly see that the network stack pushed N UDP packets (Udp and
Ip counters in /proc/net/snmp concur) however what the driver
transmitted and what the switch transmistted is N - M, and matches the
packet loss reported by the UDP server. I don't measure any SndbufErrors
which is not making sense yet.
If I reduce the default socket size to say, 10x less than 160KB, 16KB,
then I either don't see any packet loss at 100Mbits/sec for 5 minutes or
more, or just very very little, down to 0.001%. Now if I repeat the
experiment with the physical link at 10Mbits/sec, same thing, the 16KB
wmem_default setting is no longer working and we need to lower the
socket write buffer size again.
So what I am wondering is:
- do I have an obvious flow control problem in my network driver that
usually does not lead to packet loss, but may sometime happen?
- why would lowering the socket write size appear to masquerade or solve
this problem?
I can consistently reproduce this across several kernel versions, 4.1,
4.9 and latest net-next and therefore can also test patches.
Thanks for reading thus far!
Have you checked qdisc counters ? Maybe drops happen there.
From: Eric Dumazet <hidden> Date: 2017-08-23 23:04:39
On Wed, 2017-08-23 at 15:49 -0700, Florian Fainelli wrote:
On 08/23/2017 03:26 PM, Eric Dumazet wrote:
quoted
On Wed, 2017-08-23 at 13:02 -0700, Florian Fainelli wrote:
quoted
Hi,
On Broadcom STB chips using bcmsysport.c and bcm_sf2.c we have an out of
band HW mechanism (not using per-flow pause frames) where we can have
the integrated network switch backpressure the CPU Ethernet controller
which translates in completing TX packets interrupts at the appropriate
pace and therefore get flow control applied end-to-end from the host CPU
port towards any downstream port. At least that is the premise and this
works reasonably well.
This has a few drawbacks in that each of the bcmsysport TX queues need
to semi-statically map to their switch port output queues such that the
switch can calculate buffer occupancy and report congestion status,
which prompted this email [1] but this is tangential and is a policy not
a mechanism issue.
[1]: https://www.spinics.net/lists/netdev/msg448153.html
This is useful when your CPU / integrated switch links up at 1Gbits/sec
internally, and tries to push 1Gbits/sec worth of UDP traffic to e.g: a
downstream port linking at 100Mbits/sec, which could happen depending on
what you have connected to this device.
Now the problem that I am facing, is the following:
- net.core.wmem_default = 160KB (default value)
- using iperf -b 800M -u towards an iperf UDP server with the physical
link to that server established at 100Mbits/sec
- iperf does synchronous write(2) AFAICT so this gives it flow control
- using the default duration of 10s, you can barely see any packet loss
from one run to another
- the longer the run, the higher you are going to see some packet loss,
usually in the range of ~0.15% top
The transmit flow looks like this:
gphy (net/dsa/slave.c::dsa_slave_xmit, IFF_NO_QUEUE device)
-> eth0 (drivers/net/ethernet/broadcom/bcmsysport.c, "regular" network
device)
I can clearly see that the network stack pushed N UDP packets (Udp and
Ip counters in /proc/net/snmp concur) however what the driver
transmitted and what the switch transmistted is N - M, and matches the
packet loss reported by the UDP server. I don't measure any SndbufErrors
which is not making sense yet.
If I reduce the default socket size to say, 10x less than 160KB, 16KB,
then I either don't see any packet loss at 100Mbits/sec for 5 minutes or
more, or just very very little, down to 0.001%. Now if I repeat the
experiment with the physical link at 10Mbits/sec, same thing, the 16KB
wmem_default setting is no longer working and we need to lower the
socket write buffer size again.
So what I am wondering is:
- do I have an obvious flow control problem in my network driver that
usually does not lead to packet loss, but may sometime happen?
- why would lowering the socket write size appear to masquerade or solve
this problem?
I can consistently reproduce this across several kernel versions, 4.1,
4.9 and latest net-next and therefore can also test patches.
Thanks for reading thus far!
Have you checked qdisc counters ? Maybe drops happen there.
CONFIG_NET_SCHED is actually disabled in this kernel configuration.
But even with that enabled, I don't see any drops being reported at the
qdisc level, see below.
You might try
perf record -a -g -e skb:kfree_skb sleep 10
perf report
On Wed, 2017-08-23 at 15:49 -0700, Florian Fainelli wrote:
quoted
On 08/23/2017 03:26 PM, Eric Dumazet wrote:
quoted
On Wed, 2017-08-23 at 13:02 -0700, Florian Fainelli wrote:
quoted
Hi,
On Broadcom STB chips using bcmsysport.c and bcm_sf2.c we have an out of
band HW mechanism (not using per-flow pause frames) where we can have
the integrated network switch backpressure the CPU Ethernet controller
which translates in completing TX packets interrupts at the appropriate
pace and therefore get flow control applied end-to-end from the host CPU
port towards any downstream port. At least that is the premise and this
works reasonably well.
This has a few drawbacks in that each of the bcmsysport TX queues need
to semi-statically map to their switch port output queues such that the
switch can calculate buffer occupancy and report congestion status,
which prompted this email [1] but this is tangential and is a policy not
a mechanism issue.
[1]: https://www.spinics.net/lists/netdev/msg448153.html
This is useful when your CPU / integrated switch links up at 1Gbits/sec
internally, and tries to push 1Gbits/sec worth of UDP traffic to e.g: a
downstream port linking at 100Mbits/sec, which could happen depending on
what you have connected to this device.
Now the problem that I am facing, is the following:
- net.core.wmem_default = 160KB (default value)
- using iperf -b 800M -u towards an iperf UDP server with the physical
link to that server established at 100Mbits/sec
- iperf does synchronous write(2) AFAICT so this gives it flow control
- using the default duration of 10s, you can barely see any packet loss
from one run to another
- the longer the run, the higher you are going to see some packet loss,
usually in the range of ~0.15% top
The transmit flow looks like this:
gphy (net/dsa/slave.c::dsa_slave_xmit, IFF_NO_QUEUE device)
-> eth0 (drivers/net/ethernet/broadcom/bcmsysport.c, "regular" network
device)
I can clearly see that the network stack pushed N UDP packets (Udp and
Ip counters in /proc/net/snmp concur) however what the driver
transmitted and what the switch transmistted is N - M, and matches the
packet loss reported by the UDP server. I don't measure any SndbufErrors
which is not making sense yet.
If I reduce the default socket size to say, 10x less than 160KB, 16KB,
then I either don't see any packet loss at 100Mbits/sec for 5 minutes or
more, or just very very little, down to 0.001%. Now if I repeat the
experiment with the physical link at 10Mbits/sec, same thing, the 16KB
wmem_default setting is no longer working and we need to lower the
socket write buffer size again.
So what I am wondering is:
- do I have an obvious flow control problem in my network driver that
usually does not lead to packet loss, but may sometime happen?
- why would lowering the socket write size appear to masquerade or solve
this problem?
I can consistently reproduce this across several kernel versions, 4.1,
4.9 and latest net-next and therefore can also test patches.
Thanks for reading thus far!
Have you checked qdisc counters ? Maybe drops happen there.
CONFIG_NET_SCHED is actually disabled in this kernel configuration.
But even with that enabled, I don't see any drops being reported at the
qdisc level, see below.
You might try
perf record -a -g -e skb:kfree_skb sleep 10
perf report
Attached is the perf report --stdio of:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 800M -t 60
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 36169 connected with 192.168.1.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 685 MBytes 95.8 Mbits/sec
[ 4] Sent 488633 datagrams
[ 4] Server Report:
[ 4] 0.0-74.4 sec 685 MBytes 77.2 Mbits/sec 0.187 ms 300/488632
(0.061%)
[ 4] 0.0-74.4 sec 58 datagrams received out-of-order
It measured 488644 events which is greater than the number of packets
transmitted by iperf but I only count 8 non-IP packets being sent
(protocl=2054 -> ETH_P_ARP) so I am not sure what the other 4 were and
why there are not accounted for.
Almost all events came from net_tx_action() except 2 that came from
net/core/neighbour.c::neigh_probe() and 65 that came from
arch/arm/include/asm/irqflags.h::arch_local_irq_save() ?!?
I did eventually run what you suggested:
# perf record -a -g -e skb:kfree_skb sleep 10
# perf report --stdio
# To display the perf.data header info, please use
--header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 3 of event 'skb:kfree_skb'
# Event count (approx.): 3
#
# Children Self Trace output
# ........ ........ ....................................................
#
33.33% 33.33% skbaddr=0xef207240 protocol=2048 location=0xc087f684
|
---0xe9fa0360
kfree_skb
33.33% 33.33% skbaddr=0xef207300 protocol=4 location=0xc086cdac
|
---0xe9fa0360
kfree_skb
33.33% 33.33% skbaddr=0xef207b40 protocol=4 location=0xc086cdac
|
---0xe9fa0360
kfree_skb
Florian
From: Eric Dumazet <hidden> Date: 2017-08-24 00:43:49
On Wed, 2017-08-23 at 17:03 -0700, Florian Fainelli wrote:
Attached is the perf report --stdio of:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 800M -t 60
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 36169 connected with 192.168.1.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 685 MBytes 95.8 Mbits/sec
[ 4] Sent 488633 datagrams
[ 4] Server Report:
[ 4] 0.0-74.4 sec 685 MBytes 77.2 Mbits/sec 0.187 ms 300/488632
(0.061%)
[ 4] 0.0-74.4 sec 58 datagrams received out-of-order
It measured 488644 events which is greater than the number of packets
transmitted by iperf but I only count 8 non-IP packets being sent
(protocl=2054 -> ETH_P_ARP) so I am not sure what the other 4 were and
why there are not accounted for.
Almost all events came from net_tx_action() except 2 that came from
net/core/neighbour.c::neigh_probe() and 65 that came from
arch/arm/include/asm/irqflags.h::arch_local_irq_save() ?!?
On Wed, 2017-08-23 at 17:03 -0700, Florian Fainelli wrote:
quoted
Attached is the perf report --stdio of:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 800M -t 60
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 36169 connected with 192.168.1.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 685 MBytes 95.8 Mbits/sec
[ 4] Sent 488633 datagrams
[ 4] Server Report:
[ 4] 0.0-74.4 sec 685 MBytes 77.2 Mbits/sec 0.187 ms 300/488632
(0.061%)
[ 4] 0.0-74.4 sec 58 datagrams received out-of-order
It measured 488644 events which is greater than the number of packets
transmitted by iperf but I only count 8 non-IP packets being sent
(protocl=2054 -> ETH_P_ARP) so I am not sure what the other 4 were and
why there are not accounted for.
Almost all events came from net_tx_action() except 2 that came from
net/core/neighbour.c::neigh_probe() and 65 that came from
arch/arm/include/asm/irqflags.h::arch_local_irq_save() ?!?
@@ -597,7 +597,7 @@ static int bcm_sysport_set_coalesce(struct net_device *dev,staticvoidbcm_sysport_free_cb(structbcm_sysport_cb*cb){-dev_kfree_skb_any(cb->skb);+dev_consume_skb_any(cb->skb);cb->skb=NULL;dma_unmap_addr_set(cb,dma_addr,0);}
Yay, now I am getting some sensible information, thanks!
iperf reports 143 packets lost, and perf report gets me 146 entries for
kfree_skb() looking like that:
#
2.74% 2.74% skbaddr=0xee5f30c0 protocol=2048 location=0xc0855cdc
|
---0x2d
write
0xe9fc2368
kfree_skb
What is annoying is that 0xc0855cdc is resolved to
arch/arm/include/asm/irqflags.h::arch_local_irq_save() which does not
really help telling me where exactly in the stack the drop is happening,
although now I know it is somewhere down the path from write(2)... of
course it is :)
Now what is disturbing too is that iperf does have its write() system
call get an error returned, write happily returned the number of bytes
written even though the perf trace indicates there was packet drops down
the road..
I will keep investigating, thanks a lot Eric!
--
Florian
On Wed, 2017-08-23 at 17:03 -0700, Florian Fainelli wrote:
quoted
Attached is the perf report --stdio of:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 800M -t 60
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 36169 connected with 192.168.1.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 685 MBytes 95.8 Mbits/sec
[ 4] Sent 488633 datagrams
[ 4] Server Report:
[ 4] 0.0-74.4 sec 685 MBytes 77.2 Mbits/sec 0.187 ms 300/488632
(0.061%)
[ 4] 0.0-74.4 sec 58 datagrams received out-of-order
It measured 488644 events which is greater than the number of packets
transmitted by iperf but I only count 8 non-IP packets being sent
(protocl=2054 -> ETH_P_ARP) so I am not sure what the other 4 were and
why there are not accounted for.
Almost all events came from net_tx_action() except 2 that came from
net/core/neighbour.c::neigh_probe() and 65 that came from
arch/arm/include/asm/irqflags.h::arch_local_irq_save() ?!?
@@ -597,7 +597,7 @@ static int bcm_sysport_set_coalesce(struct net_device *dev,staticvoidbcm_sysport_free_cb(structbcm_sysport_cb*cb){-dev_kfree_skb_any(cb->skb);+dev_consume_skb_any(cb->skb);cb->skb=NULL;dma_unmap_addr_set(cb,dma_addr,0);}
Yay, now I am getting some sensible information, thanks!
iperf reports 143 packets lost, and perf report gets me 146 entries for
kfree_skb() looking like that:
#
2.74% 2.74% skbaddr=0xee5f30c0 protocol=2048 location=0xc0855cdc
|
---0x2d
write
0xe9fc2368
kfree_skb
What is annoying is that 0xc0855cdc is resolved to
arch/arm/include/asm/irqflags.h::arch_local_irq_save() which does not
really help telling me where exactly in the stack the drop is happening,
although now I know it is somewhere down the path from write(2)... of
course it is :)
Now what is disturbing too is that iperf does have its write() system
call get an error returned, write happily returned the number of bytes
written even though the perf trace indicates there was packet drops down
the road..
Because perf was not able to track down exactly where the location was
with builtin_return_address(0) and always pointed to
arch_local_irq_save(), I added a WARN() in kfree_skb(), and of course
this made it impossible to see packet loss under the same conditions
anymore. After increasing dramatically wmem_{default,max} I could start
seeing significant drop made largely worse by the printks, so not very
conclusive either.
I took another approach and recorded net_dev_queue() events, and I can
see only 976704 (gphy + eth0 devices) / 2 = 488352 dev_queue_events()
were recorded. The total sent by iperf was 488898 and we lost 558
packets according to the server 488352 + 558 > 488898 which makes sense
because neigh_probe() and others still run. This still confirms there is
a drop occurring between UDP socket sendmsg and the SKB enqueuing...
So my take away is that the CPU has a bursty write(2) behavior, and the
larger the socket write buffer size, the more datagrams you can burst
into and the more of these SKBs can be flat out being dropped when the
transmit queue is congested. Adding a WARN() is enough of a slow down
for the CPU that we are not able to reliably reproduce this burst.
Eric, are there areas of the stack where we are allowed to drop packets,
not propagate that back to write(2) and also not increment any counter
either, or maybe I am not looking where I should...
Thanks!
--
Florian
From: Eric Dumazet <hidden> Date: 2017-08-25 23:57:22
On Fri, 2017-08-25 at 16:18 -0700, Florian Fainelli wrote:
Eric, are there areas of the stack where we are allowed to drop packets,
not propagate that back to write(2) and also not increment any counter
either, or maybe I am not looking where I should...
What happens if you increase these sysctls ?
grep . `find /proc/sys|grep unres_qlen`
unres_qlen_bytes -> 2000000
unres_qlen -> 10000
On Fri, 2017-08-25 at 16:18 -0700, Florian Fainelli wrote:
quoted
Eric, are there areas of the stack where we are allowed to drop packets,
not propagate that back to write(2) and also not increment any counter
either, or maybe I am not looking where I should...
What happens if you increase these sysctls ?
I don't see packet loss after I tweak these two sysctls according to
your suggestions.
Tweaking eth0's sysctls did not change anything, but tweaking gphy's
sysctl resolved the loss. This was a little surprising considering that
gphy is an IFF_NO_QUEUE interface and eth0 is the conduit interface that
does the real transmission.
Does that make sense with respect to what I reported earlier? Should I
try to dump the neigh stats?
Thanks!
From: Eric Dumazet <hidden> Date: 2017-08-26 01:52:06
On Fri, 2017-08-25 at 18:17 -0700, Florian Fainelli wrote:
On 08/25/2017 04:57 PM, Eric Dumazet wrote:
quoted
On Fri, 2017-08-25 at 16:18 -0700, Florian Fainelli wrote:
quoted
Eric, are there areas of the stack where we are allowed to drop packets,
not propagate that back to write(2) and also not increment any counter
either, or maybe I am not looking where I should...
What happens if you increase these sysctls ?
I don't see packet loss after I tweak these two sysctls according to
your suggestions.
Tweaking eth0's sysctls did not change anything, but tweaking gphy's
sysctl resolved the loss. This was a little surprising considering that
gphy is an IFF_NO_QUEUE interface and eth0 is the conduit interface that
does the real transmission.
Does that make sense with respect to what I reported earlier? Should I
try to dump the neigh stats?
Note that if you had TCP traffic, the neighbour would be constantly
confirmed and no losses would happen.
I guess we should an SNMP counter for packets dropped in neigh queues.
On Fri, 2017-08-25 at 18:17 -0700, Florian Fainelli wrote:
quoted
On 08/25/2017 04:57 PM, Eric Dumazet wrote:
quoted
On Fri, 2017-08-25 at 16:18 -0700, Florian Fainelli wrote:
quoted
Eric, are there areas of the stack where we are allowed to drop packets,
not propagate that back to write(2) and also not increment any counter
either, or maybe I am not looking where I should...
What happens if you increase these sysctls ?
I don't see packet loss after I tweak these two sysctls according to
your suggestions.
Tweaking eth0's sysctls did not change anything, but tweaking gphy's
sysctl resolved the loss. This was a little surprising considering that
gphy is an IFF_NO_QUEUE interface and eth0 is the conduit interface that
does the real transmission.
Does that make sense with respect to what I reported earlier? Should I
try to dump the neigh stats?
Note that if you had TCP traffic, the neighbour would be constantly
confirmed and no losses would happen.
OK, that still sounds like quite a lot for a not so long UDP session (60
seconds).
I was finally able to get a better capture by switching to an ARM64
kernel, and as confirmed this is all coming from the neighbour code:
# Event count (approx.): 1970
#
# Children Self Trace output
# ........ ........
....................................................................
#
3.10% 3.10% skbaddr=0xffffffc2fa22a800 protocol=2048
location=0xffffff80086e53f4
|
---write
el0_svc_naked
sys_write
vfs_write
__vfs_write
sock_write_iter
sock_sendmsg
inet_sendmsg
udp_sendmsg
udp_send_skb
ip_send_skb
ip_local_out
ip_output
ip_finish_output
ip_finish_output2
neigh_resolve_output
__neigh_event_send
kfree_skb
kfree_skb
3.10% 3.10% skbaddr=0xffffffc2fa22a900 protocol=2048
location=0xffffff80086e53f4
|
---write
el0_svc_naked
sys_write
vfs_write
__vfs_write
sock_write_iter
sock_sendmsg
inet_sendmsg
udp_sendmsg
udp_send_skb
ip_send_skb
ip_local_out
ip_output
ip_finish_output
ip_finish_output2
neigh_resolve_output
__neigh_event_send
kfree_skb
kfree_skb
3.10% 3.10% skbaddr=0xffffffc2fa22aa00 protocol=2048
location=0xffffff80086e53f4
|
---write
el0_svc_naked
sys_write
vfs_write
__vfs_write
sock_write_iter
sock_sendmsg
inet_sendmsg
udp_sendmsg
udp_send_skb
ip_send_skb
ip_local_out
ip_output
ip_finish_output
ip_finish_output2
neigh_resolve_output
__neigh_event_send
kfree_skb
kfree_skb
I guess we should an SNMP counter for packets dropped in neigh queues.
It would. Since the call trace involves udp_send_skb() how come we are
not returning an error to write(2)? are there other code paths where the
neighbor code can do drops like these?
--
Florian
From: Eric Dumazet <hidden> Date: 2017-08-26 03:40:46
On Fri, 2017-08-25 at 20:25 -0700, Florian Fainelli wrote:
It would. Since the call trace involves udp_send_skb() how come we are
not returning an error to write(2)? are there other code paths where the
neighbor code can do drops like these?
Are you suggesting write(2) should block until ARP resolution is
done ? :)
What about non blocking writes ?
Honestly UDP is not a protocol for which we must absolutely be sure
packets are sent or not.
It would. Since the call trace involves udp_send_skb() how come we are
not returning an error to write(2)? are there other code paths where the
neighbor code can do drops like these?
Keep in mind that the neighbour code isn't dropping the current 'skb'
coming from the IP stack, it's dropping the oldest packet in the
resolution queue.
From: David Miller <davem@davemloft.net> Date: 2017-08-26 04:19:06
From: Eric Dumazet <redacted>
Date: Fri, 25 Aug 2017 20:40:44 -0700
On Fri, 2017-08-25 at 20:25 -0700, Florian Fainelli wrote:
quoted
It would. Since the call trace involves udp_send_skb() how come we are
not returning an error to write(2)? are there other code paths where the
neighbor code can do drops like these?
Are you suggesting write(2) should block until ARP resolution is
done ? :)
What about non blocking writes ?
Honestly UDP is not a protocol for which we must absolutely be sure
packets are sent or not.
Agreed, but the ARP resolution queue really needs to scale it's backlog
to the physical technology it is attached to.
From: Eric Dumazet <hidden> Date: 2017-08-26 12:47:54
On Fri, 2017-08-25 at 21:19 -0700, David Miller wrote:
Agreed, but the ARP resolution queue really needs to scale it's backlog
to the physical technology it is attached to.
Yes, last time (in 2011) we increased the old limit of 3 packets :/
We probably should match sysctl_wmem_max so that a single socket
provider would hit its sk_sndbuf limit
Something like :
@@ -109,7 +109,8 @@ neigh/default/unres_qlen_bytes - INTEGER queued for each unresolved address by other network layers. (added in linux 3.3) Setting negative value is meaningless and will return error.- Default: 65536 Bytes(64KB)+ Default: SK_WMEM_MAX, enough to store 256 packets of medium size+ (less than 256 bytes per packet) neigh/default/unres_qlen - INTEGER The maximum number of packets which may be queued for each
@@ -2368,6 +2368,16 @@ bool sk_net_capable(const struct sock *sk, int cap);voidsk_get_meminfo(conststructsock*sk,u32*meminfo);+/* Take into consideration the size of the struct sk_buff overhead in the+*determinationofthesevalues,sincethatisnon-constantacross+*platforms.Thismakessocketqueueingbehaviorandperformance+*notdependuponsuchdifferences.+*/+#define _SK_MEM_PACKETS 256+#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)+#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+extern__u32sysctl_wmem_max;extern__u32sysctl_rmem_max;
@@ -307,16 +307,6 @@ static struct lock_class_key af_wlock_keys[AF_MAX];staticstructlock_class_keyaf_elock_keys[AF_MAX];staticstructlock_class_keyaf_kern_callback_keys[AF_MAX];-/* Take into consideration the size of the struct sk_buff overhead in the-*determinationofthesevalues,sincethatisnon-constantacross-*platforms.Thismakessocketqueueingbehaviorandperformance-*notdependuponsuchdifferences.-*/-#define _SK_MEM_PACKETS 256-#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)-#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-/* Run time adjustable parameters. */__u32sysctl_wmem_max__read_mostly=SK_WMEM_MAX;EXPORT_SYMBOL(sysctl_wmem_max);
On Fri, 2017-08-25 at 21:19 -0700, David Miller wrote:
quoted
Agreed, but the ARP resolution queue really needs to scale it's backlog
to the physical technology it is attached to.
Yes, last time (in 2011) we increased the old limit of 3 packets :/
We probably should match sysctl_wmem_max so that a single socket
provider would hit its sk_sndbuf limit
Before:
/proc/sys/net/ipv4/neigh/eth0/unres_qlen:34
/proc/sys/net/ipv4/neigh/eth0/unres_qlen_bytes:65536
/proc/sys/net/ipv4/neigh/gphy/unres_qlen:34
/proc/sys/net/ipv4/neigh/gphy/unres_qlen_bytes:65536
After:
/proc/sys/net/ipv4/neigh/eth0/unres_qlen:106
/proc/sys/net/ipv4/neigh/eth0/unres_qlen_bytes:229376
/proc/sys/net/ipv4/neigh/gphy/unres_qlen:106
/proc/sys/net/ipv4/neigh/gphy/unres_qlen_bytes:229376
and this does help a lot with the test case reported over an hour, only
2 packets lost:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 900M -t
3600 -u
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams, IPG target: 13.07 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 48209 connected with 192.168.1.23 port 5001
write failed: Invalid argument
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-404.9 sec 4.51 GBytes 95.7 Mbits/sec
[ 4] Sent 3294727 datagrams
[ 4] Server Report:
[ 4] 0.0-405.1 sec 4.51 GBytes 95.6 Mbits/sec 14.979 ms
2/3294728 (6.1e-05%)
Thanks Eric!
@@ -109,7 +109,8 @@ neigh/default/unres_qlen_bytes - INTEGER queued for each unresolved address by other network layers. (added in linux 3.3) Setting negative value is meaningless and will return error.- Default: 65536 Bytes(64KB)+ Default: SK_WMEM_MAX, enough to store 256 packets of medium size+ (less than 256 bytes per packet) neigh/default/unres_qlen - INTEGER The maximum number of packets which may be queued for each
@@ -2368,6 +2368,16 @@ bool sk_net_capable(const struct sock *sk, int cap);voidsk_get_meminfo(conststructsock*sk,u32*meminfo);+/* Take into consideration the size of the struct sk_buff overhead in the+*determinationofthesevalues,sincethatisnon-constantacross+*platforms.Thismakessocketqueueingbehaviorandperformance+*notdependuponsuchdifferences.+*/+#define _SK_MEM_PACKETS 256+#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)+#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+extern__u32sysctl_wmem_max;extern__u32sysctl_rmem_max;
@@ -307,16 +307,6 @@ static struct lock_class_key af_wlock_keys[AF_MAX];staticstructlock_class_keyaf_elock_keys[AF_MAX];staticstructlock_class_keyaf_kern_callback_keys[AF_MAX];-/* Take into consideration the size of the struct sk_buff overhead in the-*determinationofthesevalues,sincethatisnon-constantacross-*platforms.Thismakessocketqueueingbehaviorandperformance-*notdependuponsuchdifferences.-*/-#define _SK_MEM_PACKETS 256-#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)-#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-/* Run time adjustable parameters. */__u32sysctl_wmem_max__read_mostly=SK_WMEM_MAX;EXPORT_SYMBOL(sysctl_wmem_max);
On Fri, 2017-08-25 at 21:19 -0700, David Miller wrote:
quoted
Agreed, but the ARP resolution queue really needs to scale it's backlog
to the physical technology it is attached to.
Yes, last time (in 2011) we increased the old limit of 3 packets :/
We probably should match sysctl_wmem_max so that a single socket
provider would hit its sk_sndbuf limit
Eric, do you want to post this as a formal patch? I don't think I
understand these tunables enough to provide a good commit message
anyways. Thanks!
Before:
/proc/sys/net/ipv4/neigh/eth0/unres_qlen:34
/proc/sys/net/ipv4/neigh/eth0/unres_qlen_bytes:65536
/proc/sys/net/ipv4/neigh/gphy/unres_qlen:34
/proc/sys/net/ipv4/neigh/gphy/unres_qlen_bytes:65536
After:
/proc/sys/net/ipv4/neigh/eth0/unres_qlen:106
/proc/sys/net/ipv4/neigh/eth0/unres_qlen_bytes:229376
/proc/sys/net/ipv4/neigh/gphy/unres_qlen:106
/proc/sys/net/ipv4/neigh/gphy/unres_qlen_bytes:229376
and this does help a lot with the test case reported over an hour, only
2 packets lost:
# perf record -a -g -e skb:kfree_skb iperf -c 192.168.1.23 -b 900M -t
3600 -u
------------------------------------------------------------
Client connecting to 192.168.1.23, UDP port 5001
Sending 1470 byte datagrams, IPG target: 13.07 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 48209 connected with 192.168.1.23 port 5001
write failed: Invalid argument
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-404.9 sec 4.51 GBytes 95.7 Mbits/sec
[ 4] Sent 3294727 datagrams
[ 4] Server Report:
[ 4] 0.0-405.1 sec 4.51 GBytes 95.6 Mbits/sec 14.979 ms
2/3294728 (6.1e-05%)
Thanks Eric!
@@ -109,7 +109,8 @@ neigh/default/unres_qlen_bytes - INTEGER queued for each unresolved address by other network layers. (added in linux 3.3) Setting negative value is meaningless and will return error.- Default: 65536 Bytes(64KB)+ Default: SK_WMEM_MAX, enough to store 256 packets of medium size+ (less than 256 bytes per packet) neigh/default/unres_qlen - INTEGER The maximum number of packets which may be queued for each
@@ -2368,6 +2368,16 @@ bool sk_net_capable(const struct sock *sk, int cap);voidsk_get_meminfo(conststructsock*sk,u32*meminfo);+/* Take into consideration the size of the struct sk_buff overhead in the+*determinationofthesevalues,sincethatisnon-constantacross+*platforms.Thismakessocketqueueingbehaviorandperformance+*notdependuponsuchdifferences.+*/+#define _SK_MEM_PACKETS 256+#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)+#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+extern__u32sysctl_wmem_max;extern__u32sysctl_rmem_max;
@@ -307,16 +307,6 @@ static struct lock_class_key af_wlock_keys[AF_MAX];staticstructlock_class_keyaf_elock_keys[AF_MAX];staticstructlock_class_keyaf_kern_callback_keys[AF_MAX];-/* Take into consideration the size of the struct sk_buff overhead in the-*determinationofthesevalues,sincethatisnon-constantacross-*platforms.Thismakessocketqueueingbehaviorandperformance-*notdependuponsuchdifferences.-*/-#define _SK_MEM_PACKETS 256-#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)-#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-/* Run time adjustable parameters. */__u32sysctl_wmem_max__read_mostly=SK_WMEM_MAX;EXPORT_SYMBOL(sysctl_wmem_max);
From: Eric Dumazet <hidden> Date: 2017-08-29 18:01:32
On Tue, 2017-08-29 at 10:53 -0700, Florian Fainelli wrote:
On 08/26/2017 11:56 AM, Florian Fainelli wrote:
quoted
On 08/26/2017 05:47 AM, Eric Dumazet wrote:
quoted
On Fri, 2017-08-25 at 21:19 -0700, David Miller wrote:
quoted
Agreed, but the ARP resolution queue really needs to scale it's backlog
to the physical technology it is attached to.
Yes, last time (in 2011) we increased the old limit of 3 packets :/
We probably should match sysctl_wmem_max so that a single socket
provider would hit its sk_sndbuf limit
Eric, do you want to post this as a formal patch? I don't think I
understand these tunables enough to provide a good commit message
anyways. Thanks!
I will post it today.
I was out of the office yesterday, rafting on the south fork of American
River ;)
This will target net-next.
Thanks.
From: Eric Dumazet <hidden> Date: 2017-08-29 22:16:04
From: Eric Dumazet <redacted>
Florian reported UDP xmit drops that could be root caused to the
too small neigh limit.
Current limit is 64 KB, meaning that even a single UDP socket would hit
it, since its default sk_sndbuf comes from net.core.wmem_default
(~212992 bytes on 64bit arches).
Once ARP/ND resolution is in progress, we should allow a little more
packets to be queued, at least for one producer.
Once neigh arp_queue is filled, a rogue socket should hit its sk_sndbuf
limit and either block in sendmsg() or return -EAGAIN.
Signed-off-by: Eric Dumazet <redacted>
Reported-by: Florian Fainelli <f.fainelli@gmail.com>
---
Documentation/networking/ip-sysctl.txt | 7 +++++--
include/net/sock.h | 10 ++++++++++
net/core/sock.c | 10 ----------
net/decnet/dn_neigh.c | 2 +-
net/ipv4/arp.c | 2 +-
net/ipv4/tcp_input.c | 2 +-
net/ipv6/ndisc.c | 2 +-
7 files changed, 19 insertions(+), 16 deletions(-)
@@ -109,7 +109,10 @@ neigh/default/unres_qlen_bytes - INTEGER queued for each unresolved address by other network layers. (added in linux 3.3) Setting negative value is meaningless and will return error.- Default: 65536 Bytes(64KB)+ Default: SK_WMEM_MAX, (same as net.core.wmem_default).+ Exact value depends on architecture and kernel options,+ but should be enough to allow queuing 256 packets+ of medium size. neigh/default/unres_qlen - INTEGER The maximum number of packets which may be queued for each
@@ -119,7 +122,7 @@ neigh/default/unres_qlen - INTEGER unexpected packet loss. The current default value is calculated according to default value of unres_qlen_bytes and true size of packet.- Default: 31+ Default: 101 mtu_expires - INTEGER Time, in seconds, that cached PMTU information is kept.
@@ -2368,6 +2368,16 @@ bool sk_net_capable(const struct sock *sk, int cap);voidsk_get_meminfo(conststructsock*sk,u32*meminfo);+/* Take into consideration the size of the struct sk_buff overhead in the+*determinationofthesevalues,sincethatisnon-constantacross+*platforms.Thismakessocketqueueingbehaviorandperformance+*notdependuponsuchdifferences.+*/+#define _SK_MEM_PACKETS 256+#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)+#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)+extern__u32sysctl_wmem_max;extern__u32sysctl_rmem_max;
@@ -307,16 +307,6 @@ static struct lock_class_key af_wlock_keys[AF_MAX];staticstructlock_class_keyaf_elock_keys[AF_MAX];staticstructlock_class_keyaf_kern_callback_keys[AF_MAX];-/* Take into consideration the size of the struct sk_buff overhead in the-*determinationofthesevalues,sincethatisnon-constantacross-*platforms.Thismakessocketqueueingbehaviorandperformance-*notdependuponsuchdifferences.-*/-#define _SK_MEM_PACKETS 256-#define _SK_MEM_OVERHEAD SKB_TRUESIZE(256)-#define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-#define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS)-/* Run time adjustable parameters. */__u32sysctl_wmem_max__read_mostly=SK_WMEM_MAX;EXPORT_SYMBOL(sysctl_wmem_max);
@@ -6086,9 +6086,9 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,structtcp_sock*tp=tcp_sk(sk);structnet*net=sock_net(sk);structsock*fastopen_sk=NULL;-structdst_entry*dst=NULL;structrequest_sock*req;boolwant_cookie=false;+structdst_entry*dst;structflowifl;/* TW buckets are converted to open requests without
From: David Miller <davem@davemloft.net> Date: 2017-08-29 23:11:06
From: Eric Dumazet <redacted>
Date: Tue, 29 Aug 2017 15:16:01 -0700
From: Eric Dumazet <redacted>
Florian reported UDP xmit drops that could be root caused to the
too small neigh limit.
Current limit is 64 KB, meaning that even a single UDP socket would hit
it, since its default sk_sndbuf comes from net.core.wmem_default
(~212992 bytes on 64bit arches).
Once ARP/ND resolution is in progress, we should allow a little more
packets to be queued, at least for one producer.
Once neigh arp_queue is filled, a rogue socket should hit its sk_sndbuf
limit and either block in sendmsg() or return -EAGAIN.
Signed-off-by: Eric Dumazet <redacted>
Reported-by: Florian Fainelli <f.fainelli@gmail.com>