Thread (4 messages) 4 messages, 2 authors, 2014-12-29

Re: Marvell Kirkwood - MV643XX: near 100% UDP RX packet loss

From: Bruno Prémont <bonbons@linux-vserver.org>
Date: 2014-12-27 11:17:33

On Thu, 25 December 2014 Rick Jones [off-list ref] wrote:
quoted
Why are so many packets being discarded?
You should also check the netstat statistics, particularly UDP on the 
receiving side.  Look before and after the test and see how they change, 
if at all.
Here they go.

Summary of numbers:
iperf UDP run, 5 seconds @ 1Gb/s
 lost 71216/71776 packets

                                  before      after      delta
ethtool:
 rx_packets:                      420001     688424     268423
 rx_bytes:                     433251917  809803463  376551546
 rx_errors:                            0          0          0
 rx_dropped:                           0          0          0
 bad_octets_received:                  0          0          0
 bad_frames_received:                  0          0          0
 rx_discard:                      159691     323123     163432
 rx_overrun:                           0          0          0
netstat, udp:
 packets received                  15559      16137        578
 packets to unknown port received     18         18          0
 packet receive errors             41599      83890      42291
 packets sent                      34697      34770         73
 receive buffer errors                 0          0          0
 send buffer errors                    0          0          0



Details of numbers:

ethtool -S eth0; netstat -ns; iperf3 -s -p 3740; netstat -ns; ethtool -S eth0                                                             
NIC statistics:
     rx_packets: 420001
     tx_packets: 158940
     rx_bytes: 433251917
     tx_bytes: 51728533
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     good_octets_received: 652309262
     bad_octets_received: 0
     internal_mac_transmit_err: 0
     good_frames_received: 579692
     bad_frames_received: 0
     broadcast_frames_received: 3916
     multicast_frames_received: 4369
     frames_64_octets: 14938
     frames_65_to_127_octets: 133791
     frames_128_to_255_octets: 106501
     frames_256_to_511_octets: 2974
     frames_512_to_1023_octets: 80262
     frames_1024_to_max_octets: 400166
     good_octets_sent: 52557543
     good_frames_sent: 158940
     excessive_collision: 0
     multicast_frames_sent: 428
     broadcast_frames_sent: 9352
     unrec_mac_control_received: 0
     fc_sent: 0
     good_fc_received: 0
     bad_fc_received: 0
     undersize_received: 0
     fragments_received: 0
     oversize_received: 0
     jabber_received: 0
     mac_receive_error: 0
     bad_crc_event: 0
     collision: 0
     late_collision: 0
     rx_discard: 159691
     rx_overrun: 0
Ip:
    416522 total packets received
    8 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    193528 incoming packets delivered
    164801 requests sent out
    2 outgoing packets dropped
    512 fragments dropped after timeout
    265053 reassemblies required
    42444 packets reassembled ok
    3518 packet reassembles failed
    2 fragments received ok
    4 fragments created
Icmp:
    64870 ICMP messages received
    1730 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 21090
        echo requests: 4912
        echo replies: 38868
    70216 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 21090
        time exceeded: 6
        echo request: 44208
        echo replies: 4912
IcmpMsg:
        InType0: 38868
        InType3: 21090
        InType8: 4912
        OutType0: 4912
        OutType3: 21090
        OutType8: 44208
        OutType11: 6
Tcp:
    7 active connections openings
    4 passive connection openings
    0 failed connection attempts
    0 connection resets received
    5 connections established
    20099 segments received
    20688 segments send out
    0 segments retransmited
    0 bad segments received.
    0 resets sent
Udp:
    15559 packets received
    18 packets to unknown port received.
    41599 packet receive errors
    34697 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    4 TCP sockets finished time wait in fast timer
    7 delayed acks sent
    5 delayed acks further delayed because of locked socket
    3 packets directly queued to recvmsg prequeue.
    254 bytes directly received in process context from prequeue
    7990 packet headers predicted
    3 packets header predicted and directly queued to user
    301 acknowledgments not containing data payload received
    10132 predicted acknowledgments
    TCPRcvCoalesce: 7843
    TCPOrigDataSent: 11810
IpExt:
    InMcastPkts: 78
    InBcastPkts: 303
    InOctets: 421982031
    OutOctets: 20785506
    InMcastOctets: 2184
    InBcastOctets: 43372
    InNoECTPkts: 416523
-----------------------------------------------------------
Server listening on 3740
-----------------------------------------------------------
Accepted connection from 192.168.0.139, port 36178
[  5] local 192.168.0.70 port 3740 connected to 192.168.0.139 port 55831
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.02   sec   736 KBytes  5.93 Mbits/sec  7.017 ms  11481/11573 (99%)  
[  5]   1.02-2.02   sec   368 KBytes  3.01 Mbits/sec  9.887 ms  14867/14913 (1e+02%)  
[  5]   2.02-3.02   sec   560 KBytes  4.59 Mbits/sec  5.791 ms  14528/14598 (1e+02%)  
[  5]   3.02-4.02   sec   984 KBytes  8.06 Mbits/sec  7.613 ms  13972/14095 (99%)  
[  5]   4.02-5.04   sec  1.66 MBytes  13.7 Mbits/sec  4.602 ms  15842/16055 (99%)  
[  5]   5.04-5.25   sec   128 KBytes  4.80 Mbits/sec  3.809 ms  526/542 (97%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-5.25   sec   562 MBytes   897 Mbits/sec  3.809 ms  71216/71776 (99%)  
-----------------------------------------------------------
Server listening on 3740
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
Ip:
    684863 total packets received
    8 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    236661 incoming packets delivered
    165154 requests sent out
    2 outgoing packets dropped
    512 fragments dropped after timeout
    533112 reassemblies required
    85295 packets reassembled ok
    6421 packet reassembles failed
    2 fragments received ok
    4 fragments created
Icmp:
    65021 ICMP messages received
    1737 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 21144
        echo requests: 4923
        echo replies: 38954
    70380 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 21144
        time exceeded: 6
        echo request: 44307
        echo replies: 4923
IcmpMsg:
        InType0: 38954
        InType3: 21144
        InType8: 4923
        OutType0: 4923
        OutType3: 21144
        OutType8: 44307
        OutType11: 6
Tcp:
    7 active connections openings
    5 passive connection openings
    0 failed connection attempts
    0 connection resets received
    5 connections established
    20203 segments received
    20803 segments send out
    0 segments retransmited
    0 bad segments received.
    0 resets sent
Udp:
    16137 packets received
    18 packets to unknown port received.
    83890 packet receive errors
    34770 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    4 TCP sockets finished time wait in fast timer
    9 delayed acks sent
    5 delayed acks further delayed because of locked socket
    6 packets directly queued to recvmsg prequeue.
    510 bytes directly received in process context from prequeue
    8008 packet headers predicted
    6 packets header predicted and directly queued to user
    304 acknowledgments not containing data payload received
    10211 predicted acknowledgments
    TCPRcvCoalesce: 7859
    TCPOrigDataSent: 11904
IpExt:
    InMcastPkts: 78
    InBcastPkts: 303
    InOctets: 793681544
    OutOctets: 20825320
    InMcastOctets: 2184
    InBcastOctets: 43372
    InNoECTPkts: 684865
NIC statistics:
     rx_packets: 688424
     tx_packets: 159342
     rx_bytes: 809803463
     tx_bytes: 51782065
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     good_octets_received: 1258435196
     bad_octets_received: 0
     internal_mac_transmit_err: 0
     good_frames_received: 1011547
     bad_frames_received: 0
     broadcast_frames_received: 3926
     multicast_frames_received: 4384
     frames_64_octets: 14978
     frames_65_to_127_octets: 134201
     frames_128_to_255_octets: 106794
     frames_256_to_511_octets: 3006
     frames_512_to_1023_octets: 152189
     frames_1024_to_max_octets: 759721
     good_octets_sent: 52613183
     good_frames_sent: 159342
     excessive_collision: 0
     multicast_frames_sent: 428
     broadcast_frames_sent: 9378
     unrec_mac_control_received: 0
     fc_sent: 0
     good_fc_received: 0
     bad_fc_received: 0
     undersize_received: 0
     fragments_received: 0
     oversize_received: 0
     jabber_received: 0
     mac_receive_error: 0
     bad_crc_event: 0
     collision: 0
     late_collision: 0
     rx_discard: 323123
     rx_overrun: 0



iperf on client side:
iperf3 -p 3740 -t 5 -b 0 -u -4 -c sheevaplug
Connecting to host sheevaplug, port 3740
[  4] local 192.168.0.139 port 55831 connected to 192.168.0.70 port 3740
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   114 MBytes   959 Mbits/sec  14630  
[  4]   1.00-2.00   sec   114 MBytes   959 Mbits/sec  14630  
[  4]   2.00-3.00   sec   114 MBytes   954 Mbits/sec  14560  
[  4]   3.00-4.00   sec   110 MBytes   926 Mbits/sec  14130  
[  4]   4.00-5.00   sec   109 MBytes   915 Mbits/sec  13960  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-5.00   sec   562 MBytes   943 Mbits/sec  3.809 ms  71216/71776 (99%)  
[  4] Sent 71776 datagrams

iperf Done.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help