Hi list,
I am using a NC553i ethernet card connected on a HP 10GbE Flex-10.
I am sending UDP multicast packets from one blade to another (HP
ProLiant BL460c G7) which has stricly the same HW.
I have lots of packet loss from Tx to Rx, and I can't understand why.
I suspected TX coalescing but since 3.4 I can't set this parameter
(and adaptive-tx is on by default).
I have tried the same test with a debian lenny (2.6.26 kernel and HP
drivers) and it works very well (adaptive-tx is off).
Here is the netstat (from Tx point of view) :
$> netstat -s eth1 > before ; sleep 10 ; netstat -s eth1 > after
$> beforeafter before after
Ip:
280769 total packets received
4 with invalid addresses
0 forwarded
0 incoming packets discarded
275063 incoming packets delivered
305430 requests sent out
0 dropped because of missing route
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 0
echo requests: 0
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 0
echo replies: 0
IcmpMsg:
InType3: 0
InType8: 0
OutType0: 0
OutType3: 0
Tcp:
18 active connections openings
18 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
3681 segments received
3650 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
12626 packets received
0 packets to unknown port received.
0 packet receive errors
259025 packets sent
UdpLite:
TcpExt:
0 invalid SYN cookies received
0 packets pruned from receive queue because of socket buffer overrun
14 TCP sockets finished time wait in fast timer
0 packets rejects in established connections because of timestamp
61 delayed acks sent
0 delayed acks further delayed because of locked socket
Quick ack mode was activated 0 times
2924 packets directly queued to recvmsg prequeue.
32 bytes directly in process context from backlog
48684 bytes directly received in process context from prequeue
232 packet headers predicted
1991 packets header predicted and directly queued to user
132 acknowledgments not containing data payload received
2230 predicted acknowledgments
0 times recovered from packet loss by selective acknowledgements
0 congestion windows recovered without slow start after partial ack
0 TCP data loss events
0 timeouts after SACK recovery
0 fast retransmits
0 forward retransmits
0 retransmits in slow start
0 other TCP timeouts
1 times receiver scheduled too late for direct processing
0 packets collapsed in receive queue due to low socket buffer
0 DSACKs sent for old packets
0 DSACKs received
0 connections reset due to unexpected data
0 connections reset due to early user close
0 connections aborted due to timeout
0 times unabled to send RST due to no memory
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPDeferAcceptDrop: 0
IpExt:
InMcastPkts: -652745397
OutMcastPkts: 301498
InBcastPkts: 13
InOctets: -2004227752
OutOctets: -2096666083
InMcastOctets: 1058181285
OutMcastOctets: -1510963815
InBcastOctets: 1014
And ethtool diff :
$> ethtool -S eth1 > before ; sleep 10 ; ethtool -S eth1 > after
$> beforeafter before after
NIC statistics:
rx_crc_errors: 0
rx_alignment_symbol_errors: 0
rx_pause_frames: 0
rx_control_frames: 0
rx_in_range_errors: 0
rx_out_range_errors: 0
rx_frame_too_long: 0
rx_address_mismatch_drops: 6
rx_dropped_too_small: 0
rx_dropped_too_short: 0
rx_dropped_header_too_small: 0
rx_dropped_tcp_length: 0
rx_dropped_runt: 0
rxpp_fifo_overflow_drop: 0
rx_input_fifo_overflow_drop: 0
rx_ip_checksum_errs: 0
rx_tcp_checksum_errs: 0
rx_udp_checksum_errs: 0
tx_pauseframes: 0
tx_controlframes: 0
rx_priority_pause_frames: 0
pmem_fifo_overflow_drop: 0
jabber_events: 0
rx_drops_no_pbuf: 0
rx_drops_no_erx_descr: 0
rx_drops_no_tpre_descr: 0
rx_drops_too_many_frags: 0
forwarded_packets: 0
rx_drops_mtu: 0
eth_red_drops: 0
be_on_die_temperature: 0
rxq0: rx_bytes: 0
rxq0: rx_pkts: 0
rxq0: rx_compl: 0
rxq0: rx_mcast_pkts: 0
rxq0: rx_post_fail: 0
rxq0: rx_drops_no_skbs: 0
rxq0: rx_drops_no_frags: 0
txq0: tx_compl: 257113
txq0: tx_bytes: 1038623935
txq0: tx_pkts: 257113
txq0: tx_reqs: 257113
txq0: tx_wrbs: 514226
txq0: tx_stops: 10
As you can see, there is 10 tx_stops in 10 seconds (it varies, can be 3 to 15).
Any thoughts ?
Regards,
JM
Hi list,
I am using a NC553i ethernet card connected on a HP 10GbE Flex-10.
I am sending UDP multicast packets from one blade to another (HP
ProLiant BL460c G7) which has stricly the same HW.
I have lots of packet loss from Tx to Rx, and I can't understand why.
I suspected TX coalescing but since 3.4 I can't set this parameter
(and adaptive-tx is on by default).
I have tried the same test with a debian lenny (2.6.26 kernel and HP
drivers) and it works very well (adaptive-tx is off).
Here is the netstat (from Tx point of view) :
$> netstat -s eth1 > before ; sleep 10 ; netstat -s eth1 > after
$> beforeafter before after
Ip:
280769 total packets received
4 with invalid addresses
0 forwarded
0 incoming packets discarded
275063 incoming packets delivered
305430 requests sent out
0 dropped because of missing route
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 0
echo requests: 0
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 0
echo replies: 0
IcmpMsg:
InType3: 0
InType8: 0
OutType0: 0
OutType3: 0
Tcp:
18 active connections openings
18 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
3681 segments received
3650 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
12626 packets received
0 packets to unknown port received.
0 packet receive errors
259025 packets sent
UdpLite:
TcpExt:
0 invalid SYN cookies received
0 packets pruned from receive queue because of socket buffer overrun
14 TCP sockets finished time wait in fast timer
0 packets rejects in established connections because of timestamp
61 delayed acks sent
0 delayed acks further delayed because of locked socket
Quick ack mode was activated 0 times
2924 packets directly queued to recvmsg prequeue.
32 bytes directly in process context from backlog
48684 bytes directly received in process context from prequeue
232 packet headers predicted
1991 packets header predicted and directly queued to user
132 acknowledgments not containing data payload received
2230 predicted acknowledgments
0 times recovered from packet loss by selective acknowledgements
0 congestion windows recovered without slow start after partial ack
0 TCP data loss events
0 timeouts after SACK recovery
0 fast retransmits
0 forward retransmits
0 retransmits in slow start
0 other TCP timeouts
1 times receiver scheduled too late for direct processing
0 packets collapsed in receive queue due to low socket buffer
0 DSACKs sent for old packets
0 DSACKs received
0 connections reset due to unexpected data
0 connections reset due to early user close
0 connections aborted due to timeout
0 times unabled to send RST due to no memory
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPDeferAcceptDrop: 0
IpExt:
InMcastPkts: -652745397
OutMcastPkts: 301498
InBcastPkts: 13
InOctets: -2004227752
OutOctets: -2096666083
InMcastOctets: 1058181285
OutMcastOctets: -1510963815
InBcastOctets: 1014
And ethtool diff :
$> ethtool -S eth1 > before ; sleep 10 ; ethtool -S eth1 > after
$> beforeafter before after
NIC statistics:
rx_crc_errors: 0
rx_alignment_symbol_errors: 0
rx_pause_frames: 0
rx_control_frames: 0
rx_in_range_errors: 0
rx_out_range_errors: 0
rx_frame_too_long: 0
rx_address_mismatch_drops: 6
rx_dropped_too_small: 0
rx_dropped_too_short: 0
rx_dropped_header_too_small: 0
rx_dropped_tcp_length: 0
rx_dropped_runt: 0
rxpp_fifo_overflow_drop: 0
rx_input_fifo_overflow_drop: 0
rx_ip_checksum_errs: 0
rx_tcp_checksum_errs: 0
rx_udp_checksum_errs: 0
tx_pauseframes: 0
tx_controlframes: 0
rx_priority_pause_frames: 0
pmem_fifo_overflow_drop: 0
jabber_events: 0
rx_drops_no_pbuf: 0
rx_drops_no_erx_descr: 0
rx_drops_no_tpre_descr: 0
rx_drops_too_many_frags: 0
forwarded_packets: 0
rx_drops_mtu: 0
eth_red_drops: 0
be_on_die_temperature: 0
rxq0: rx_bytes: 0
rxq0: rx_pkts: 0
rxq0: rx_compl: 0
rxq0: rx_mcast_pkts: 0
rxq0: rx_post_fail: 0
rxq0: rx_drops_no_skbs: 0
rxq0: rx_drops_no_frags: 0
txq0: tx_compl: 257113
txq0: tx_bytes: 1038623935
txq0: tx_pkts: 257113
txq0: tx_reqs: 257113
txq0: tx_wrbs: 514226
txq0: tx_stops: 10
As you can see, there is 10 tx_stops in 10 seconds (it varies, can be 3 to 15).
Any thoughts ?
Regards,
JM
If this can help, setting tx queue length to 5000 seems to make the
problem disappear.
I didn't specified it : MTU is 4096, UDP packets are 4000 bytes.
JM
On Wed, 2012-05-30 at 08:28 +0200, Jean-Michel Hautbois wrote:
quoted
If this can help, setting tx queue length to 5000 seems to make the
problem disappear.
Then you should have drops at Qdisc layer (before your change to 5000)
tc -s -d qdisc
quoted
I didn't specified it : MTU is 4096, UDP packets are 4000 bytes.
Yes :
qdisc mq 0: dev eth1 root
Sent 5710049154383 bytes 1413544639 pkt (dropped 73078, overlimits 0
requeues 281540)
backlog 0b 0p requeues 281540
Why ? With a 2.6.26 kernel it works well with a tx queue length of 1000.
From: Eric Dumazet <hidden> Date: 2012-05-30 07:06:32
On Wed, 2012-05-30 at 08:51 +0200, Jean-Michel Hautbois wrote:
2012/5/30 Eric Dumazet [off-list ref]:
quoted
On Wed, 2012-05-30 at 08:28 +0200, Jean-Michel Hautbois wrote:
quoted
If this can help, setting tx queue length to 5000 seems to make the
problem disappear.
Then you should have drops at Qdisc layer (before your change to 5000)
tc -s -d qdisc
quoted
I didn't specified it : MTU is 4096, UDP packets are 4000 bytes.
Yes :
qdisc mq 0: dev eth1 root
Sent 5710049154383 bytes 1413544639 pkt (dropped 73078, overlimits 0
requeues 281540)
backlog 0b 0p requeues 281540
Why ? With a 2.6.26 kernel it works well with a tx queue length of 1000.
If you send big bursts of packets, then you need a large enough queue.
Maybe your kernel is now faster than before and queue fills faster, or
TX ring is smaller ?
ethtool -g eth0
Note that everybody try to reduce dumb queue sizes because of latencies.
On Wed, 2012-05-30 at 08:51 +0200, Jean-Michel Hautbois wrote:
quoted
2012/5/30 Eric Dumazet [off-list ref]:
quoted
On Wed, 2012-05-30 at 08:28 +0200, Jean-Michel Hautbois wrote:
quoted
If this can help, setting tx queue length to 5000 seems to make the
problem disappear.
Then you should have drops at Qdisc layer (before your change to 5000)
tc -s -d qdisc
quoted
I didn't specified it : MTU is 4096, UDP packets are 4000 bytes.
Yes :
qdisc mq 0: dev eth1 root
Sent 5710049154383 bytes 1413544639 pkt (dropped 73078, overlimits 0
requeues 281540)
backlog 0b 0p requeues 281540
Why ? With a 2.6.26 kernel it works well with a tx queue length of 1000.
If you send big bursts of packets, then you need a large enough queue.
Maybe your kernel is now faster than before and queue fills faster, or
TX ring is smaller ?
ethtool -g eth0
Note that everybody try to reduce dumb queue sizes because of latencies.
TX ring is not the same :
On 3.2 :
$> ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
On 2.6.26 :
$>ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
Current hardware settings:
RX: 1003
RX Mini: 0
RX Jumbo: 0
TX: 0
I can't set TX ring using ethtool -G eth1 tx N : operation not supported
I am not really impacted by latency, but the lower the better.
JM
On Wed, 2012-05-30 at 08:51 +0200, Jean-Michel Hautbois wrote:
quoted
2012/5/30 Eric Dumazet [off-list ref]:
quoted
On Wed, 2012-05-30 at 08:28 +0200, Jean-Michel Hautbois wrote:
quoted
If this can help, setting tx queue length to 5000 seems to make the
problem disappear.
Then you should have drops at Qdisc layer (before your change to 5000)
tc -s -d qdisc
quoted
I didn't specified it : MTU is 4096, UDP packets are 4000 bytes.
Yes :
qdisc mq 0: dev eth1 root
Sent 5710049154383 bytes 1413544639 pkt (dropped 73078, overlimits 0
requeues 281540)
backlog 0b 0p requeues 281540
Why ? With a 2.6.26 kernel it works well with a tx queue length of 1000.
If you send big bursts of packets, then you need a large enough queue.
Maybe your kernel is now faster than before and queue fills faster, or
TX ring is smaller ?
ethtool -g eth0
Note that everybody try to reduce dumb queue sizes because of latencies.
TX ring is not the same :
On 3.2 :
$> ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
On 2.6.26 :
$>ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 2048
Current hardware settings:
RX: 1003
RX Mini: 0
RX Jumbo: 0
TX: 0
I can't set TX ring using ethtool -G eth1 tx N : operation not supported
I am not really impacted by latency, but the lower the better.
JM
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
JM
From: Eric Dumazet <hidden> Date: 2012-05-30 09:56:18
On Wed, 2012-05-30 at 11:40 +0200, Jean-Michel Hautbois wrote:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Might be different coalescing params :
ethtool -c eth1
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
Behalf Of Jean-Michel Hautbois
2012/5/30 Jean-Michel Hautbois [off-list ref]:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Yes, there is an issue with be2net interrupt mitigation in the recent code with
RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
mitigated.
I have a fix and will send it out soon..
thanks,
-Sathya
On Wed, 2012-05-30 at 11:40 +0200, Jean-Michel Hautbois wrote:
quoted
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Might be different coalescing params :
ethtool -c eth1
Yes, as stated in my first e-mail, this is different, in 2.6.26 the
adaptive-tx coalescing is off, while it is on for 3.4 (sorry, I said
3.2 before but it is 3.4).
But I can't change this setting since commit 10ef9ab...
JM
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
Behalf Of Jean-Michel Hautbois
2012/5/30 Jean-Michel Hautbois [off-list ref]:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Yes, there is an issue with be2net interrupt mitigation in the recent code with
RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
mitigated.
I have a fix and will send it out soon..
thanks,
-Sathya
Hi Sathya !
Thanks for this information !
I had the correct diagnostic :). I am waiting for your fix.
Regards,
JM
From: Eric Dumazet <hidden> Date: 2012-05-30 10:31:02
On Wed, 2012-05-30 at 03:04 -0700, Sathya.Perla@Emulex.Com wrote:
quoted
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
Behalf Of Jean-Michel Hautbois
2012/5/30 Jean-Michel Hautbois [off-list ref]:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Yes, there is an issue with be2net interrupt mitigation in the recent code with
RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
mitigated.
I have a fix and will send it out soon..
I also have a benet fix for non GRO :
Pulling 64 bytes in skb head is too much for TCP IPv4 with no
timestamps, as this makes splice() or TCP coalescing less effective.
(Having tcp payload in linear part of the skb disables various optims)
Could you please test it ?
Thanks
@@ -1202,15 +1202,19 @@ static void skb_fill_rx_data(struct be_rx_obj *rxo, struct sk_buff *skb,/* Copy data in the first descriptor of this completion */curr_frag_len=min(rxcp->pkt_size,rx_frag_size);-/* Copy the header portion into skb_data */-hdr_len=min(BE_HDR_LEN,curr_frag_len);+/* If frame is small enough to fit in skb->head, pull it completely.+*Ifnot,onlypullethernetheadersothatsplice()orTCPcoalesce+*aremoreefficient.+*/+hdr_len=(curr_frag_len<=skb_tailroom(skb))?+curr_frag_len:ETH_HLEN;+memcpy(skb->data,start,hdr_len);skb->len=curr_frag_len;-if(curr_frag_len<=BE_HDR_LEN){/* tiny packet */+skb->tail+=hdr_len;+if(hdr_len==curr_frag_len){/* tiny packet *//* Complete packet has now been moved to data */put_page(page_info->page);-skb->data_len=0;-skb->tail+=curr_frag_len;}else{skb_shinfo(skb)->nr_frags=1;skb_frag_set_page(skb,0,page_info->page);
-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
I also have a benet fix for non GRO :
Pulling 64 bytes in skb head is too much for TCP IPv4 with no
timestamps, as this makes splice() or TCP coalescing less effective.
(Having tcp payload in linear part of the skb disables various optims)
Could you please test it ?
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
Behalf Of Jean-Michel Hautbois
2012/5/30 Jean-Michel Hautbois [off-list ref]:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Yes, there is an issue with be2net interrupt mitigation in the recent code with
RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
mitigated.
I have a fix and will send it out soon..
thanks,
-Sathya
It seems this issue exist with mellanox mlx4 too...
I have a bond between eth1 (be2net) and eth9 (mlx4_en) and I can
switch from one to the other using ifenslave.
Setting a queue of 4000 on be2net works well in my use case, when I
switch to mlx4 based NIC which has a default queue len of 1000, I have
dropped packets too (less than be2net, but about 30-50 per second).
Setting a queue len of 4000 on mlx4 works too, but the number of
interrupts is similar.
The use case is the same : Lots of TX, no RX.
JM
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
Behalf Of Jean-Michel Hautbois
2012/5/30 Jean-Michel Hautbois [off-list ref]:
I used vmstat in order to see the differences between the two kernels.
The main difference is the number of interrupts per second.
I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
I suspect the be2net driver to be the main cause, and I checkes the
/proc/interrupts file in order to be sure.
I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
whereas there is eth1-rx0 and eth1-tx on 2.6.26.
Yes, there is an issue with be2net interrupt mitigation in the recent code with
RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
mitigated.
I have a fix and will send it out soon..
thanks,
-Sathya
Hi Sathya !
Thanks for this information !
I had the correct diagnostic :). I am waiting for your fix.
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
JM
From: Eric Dumazet <hidden> Date: 2012-06-06 11:01:31
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
Yep.
This driver wants to limit number of tx completions, thats just wrong.
Fix and dirty patch:
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
quoted
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
Yep.
This driver wants to limit number of tx completions, thats just wrong.
Fix and dirty patch:
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
quoted
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
Yep.
This driver wants to limit number of tx completions, thats just wrong.
Fix and dirty patch:
I will try that in a few minutes.
I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
a broadcom (bnx2x).
And it is not really better, still need about 18000 at 2.4Gbps in
order to avoid drops...
I really think there is something in the networking stack or in my
configuration (DMA ? Something else ?)...
As it doesn't seem to be driver related as I said...
JM
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
quoted
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
Yep.
This driver wants to limit number of tx completions, thats just wrong.
Fix and dirty patch:
I will try that in a few minutes.
I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
a broadcom (bnx2x).
And it is not really better, still need about 18000 at 2.4Gbps in
order to avoid drops...
I really think there is something in the networking stack or in my
configuration (DMA ? Something else ?)...
As it doesn't seem to be driver related as I said...
If it can help, on a 3.0 kernel a txqueuelen of 9000 is sufficient in
order to get this bandwith on TX.
JM
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
quoted
Well, well, well, after having tested several configurations, several
drivers, I have a big difference between an old 2.6.26 kernel and a
newer one (I tried 3.2 and 3.4).
Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
set to 4096. I am sending packets only, nothing on RX.
I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
kernel, but a lot of drops with a newer kernel.
So, I don't know if I missed something in my kernel configuration, but
I have used the 2.6.26 one as a reference, in order to set the same
options (DMA related, etc).
I easily reproduce this problem and setting a bigger txqueuelen solves
it partially.
1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
If you have any idea, I am interested, as this is a big issue for my use case.
Yep.
This driver wants to limit number of tx completions, thats just wrong.
Fix and dirty patch:
I will try that in a few minutes.
I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
a broadcom (bnx2x).
And it is not really better, still need about 18000 at 2.4Gbps in
order to avoid drops...
I really think there is something in the networking stack or in my
configuration (DMA ? Something else ?)...
As it doesn't seem to be driver related as I said...
If it can help, on a 3.0 kernel a txqueuelen of 9000 is sufficient in
order to get this bandwith on TX.
JM
All,
I made some tests, and I didn't mention it : I am using the bonding
driver over my ethernet drivers (be2net/mlx4 etc.).
When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
When I disable bonding, and use directly the NIC then I don't see any
drops in qdisc and it works well.
So, I think there is something between 2.6.26 and 3.0 in the bonding
driver which causes this issue.
Any help would be appreciated :).
Regards,
JM
From: Eric Dumazet <hidden> Date: 2012-06-07 12:31:34
On Thu, 2012-06-07 at 14:27 +0200, Jean-Michel Hautbois wrote:
I made some tests, and I didn't mention it : I am using the bonding
driver over my ethernet drivers (be2net/mlx4 etc.).
When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
When I disable bonding, and use directly the NIC then I don't see any
drops in qdisc and it works well.
So, I think there is something between 2.6.26 and 3.0 in the bonding
driver which causes this issue.
What your bond configuration looks like ?
cat /proc/net/bonding/bond0
ifconfig -a
tc -s -d qdisc
On Thu, 2012-06-07 at 14:27 +0200, Jean-Michel Hautbois wrote:
quoted
I made some tests, and I didn't mention it : I am using the bonding
driver over my ethernet drivers (be2net/mlx4 etc.).
When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
When I disable bonding, and use directly the NIC then I don't see any
drops in qdisc and it works well.
So, I think there is something between 2.6.26 and 3.0 in the bonding
driver which causes this issue.
What your bond configuration looks like ?
cat /proc/net/bonding/bond0
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 50
Up Delay (ms): 100
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Speed: 4000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:b5:99:b9:8d:d4
Slave queue ID: 0
Slave Interface: eth9
MII Status: up
Speed: 4000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 78:e7:d1:68:bb:38
Slave queue ID: 0