Re: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit time")
From: Arjun V <hidden>
Date: 2016-06-27 07:19:18
On Friday, June 06/24/16, 2016 at 21:55:07 +0530, Eric Dumazet wrote:
quoted hunk ↗ jump to hunk
On Fri, 2016-06-24 at 07:25 -0700, Eric Dumazet wrote:quoted
Please do not top post on netdev mailing list On Fri, Jun 24, 2016 at 4:38 AM, Arjun V. [off-list ref] wrote:quoted
Eric, We are seeing skb's with length(skb->len) greater than 65536 coming into our ndo_start_xmit() callback routine. We can add a check in our eth_xmit() routine to skip those packets, but it will be better if you fix this in kernel. I have attached pcap file obtained from tcpdump. In the pcap file there are 2 such packets(I used tcpdump filter to extract out those packets). Let us know if you need anything else.Beats me really. At retransmit time, we only can eventually reduce packet sizes (assuming GSO is used, because we might coalesce sub-mss packets in tcp_retrans_try_collapse()) So why are you seeing too big packets at retransmit, I really have no idea. MIght be some bug related to MSS computation, overflowing somehow ?Could you try this ? Thanks !diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index d39e9e47a26e55ad2b8f775bf9ea9dfb5b12aee5..27013056bcfb9aa49601806bb3aa55a1ac664873 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c@@ -1177,6 +1177,7 @@ int inet_sk_rebuild_header(struct sock *sk) /* Routing failed... */ sk->sk_route_caps = 0; + sk->sk_gso_max_segs = 1; /* * Other protocols have to map its equivalent state to TCP_SYN_SENT. * DCCP maps its DCCP_REQUESTING state to TCP_SYN_SENT. -acmediff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c index 2076c21107d07e4e78a0a29f1d374c3414b8e1bd..ecc0281acfb702b138c68ac51e3a0518052785b0 100644 --- a/net/ipv6/af_inet6.c +++ b/net/ipv6/af_inet6.c@@ -685,6 +685,7 @@ int inet6_sk_rebuild_header(struct sock *sk) dst = ip6_dst_lookup_flow(sk, &fl6, final_p); if (IS_ERR(dst)) { sk->sk_route_caps = 0; + sk->sk_gso_max_segs = 1; sk->sk_err_soft = -PTR_ERR(dst); return PTR_ERR(dst); }
Eric, Thanks for the quick patch. It didn't help much. We still see packets larger than 65536 with the patch. Below is the dump_stack() trace I am seeing for packets larger than 65536 in our xmit routine: [ 619.094366] CPU: 5 PID: 19322 Comm: netperf Not tainted 4.7.0-rc2+ #85 [ 619.094373] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 1.0b 04/21/2015 [ 619.094379] 0000000000000001 ffff88067fd42c58 ffffffff81534635 ffff88065b59c2e0 [ 619.094393] ffff88065b520000 ffff8806570e1ae0 ffff88065b520000 ffff88067fd42d68 [ 619.094404] ffffffffa001cbee 000005f27fd42ca8 ffff880600000005 ffff880600000300 [ 619.094417] Call Trace: [ 619.094421] <IRQ> [<ffffffff81534635>] dump_stack+0xa5/0xe0 [ 619.094445] [<ffffffffa001cbee>] t4_eth_xmit+0x43e/0xd90 [cxgb4] [ 619.094455] [<ffffffff81187fa9>] ? trace_clock_local+0x9/0x10 [ 619.094461] [<ffffffff81198672>] ? rb_reserve_next_event+0x642/0x750 [ 619.094469] [<ffffffff81aee5ed>] dev_hard_start_xmit+0x15d/0x460 [ 619.094475] [<ffffffff81aedb80>] ? __netdev_pick_tx+0x170/0x1f0 [ 619.094482] [<ffffffff81af58ff>] ? validate_xmit_skb_list+0x4f/0xc0 [ 619.094489] [<ffffffff81b2b861>] sch_direct_xmit+0x221/0x2b0 [ 619.094497] [<ffffffff81af602c>] __dev_queue_xmit+0x6bc/0x9a0 [ 619.094504] [<ffffffff81bf79bf>] ? ipt_do_table+0x30f/0x620 [ 619.094512] [<ffffffff81af6330>] dev_queue_xmit+0x10/0x20 [ 619.094518] [<ffffffff81b774f9>] ip_finish_output2+0x459/0x570 [ 619.094525] [<ffffffff81b426c4>] ? nf_iterate+0xa4/0x110 [ 619.094531] [<ffffffff81b426c4>] ? nf_iterate+0xa4/0x110 [ 619.094538] [<ffffffff81b7973b>] ip_finish_output+0x1eb/0x330 [ 619.094545] [<ffffffff81b798f4>] ip_output+0x74/0x110 [ 619.094552] [<ffffffff81b78802>] ? __ip_local_out+0xb2/0x100 [ 619.094557] [<ffffffff81b79550>] ? ip_queue_xmit+0x4f0/0x4f0 [ 619.094564] [<ffffffff81b788d1>] ip_local_out+0x81/0xa0 [ 619.094571] [<ffffffff81b792b1>] ip_queue_xmit+0x251/0x4f0 [ 619.094579] [<ffffffff81b9bf19>] ? tcp_options_write+0x279/0x350 [ 619.094585] [<ffffffff81b9c7bb>] tcp_transmit_skb+0x7cb/0xe70 [ 619.094592] [<ffffffff81b9ef03>] ? tcp_current_mss+0x53/0xc0 [ 619.094599] [<ffffffff81b9f815>] __tcp_retransmit_skb+0x2e5/0x9c0 [ 619.094607] [<ffffffff810cb4a6>] ? get_nohz_timer_target+0x26/0x140 [ 619.094613] [<ffffffff81b9ff07>] tcp_retransmit_skb+0x17/0x100 [ 619.094620] [<ffffffff81ba046e>] tcp_xmit_retransmit_queue+0x47e/0x670 [ 619.094626] [<ffffffff81b8e753>] tcp_xmit_recovery+0xa3/0xb0 [ 619.094631] [<ffffffff81b98897>] tcp_ack+0xd67/0x1e70 [ 619.094639] [<ffffffff81b9634f>] ? tcp_validate_incoming+0x7f/0x560 [ 619.094646] [<ffffffff81b99af2>] tcp_rcv_established+0x152/0x780 [ 619.094654] [<ffffffff81baa13e>] tcp_v4_do_rcv+0x10e/0x3a0 [ 619.094661] [<ffffffff81bab24d>] tcp_v4_rcv+0xe7d/0xf20 [ 619.094667] [<ffffffff81bf4f93>] ? ipv4_confirm+0x133/0x1d0 [ 619.094673] [<ffffffff81b426c4>] ? nf_iterate+0xa4/0x110 [ 619.094680] [<ffffffff81b70138>] ip_local_deliver_finish+0xc8/0x2c0 [ 619.094686] [<ffffffff81b703f8>] ip_local_deliver+0xc8/0x140 [ 619.094692] [<ffffffff81ba9f02>] ? tcp_v4_early_demux+0x162/0x290 [ 619.094697] [<ffffffff81b70070>] ? ip_call_ra_chain+0x220/0x220 [ 619.094703] [<ffffffff81b6fc5f>] ip_rcv_finish+0x3ef/0x5e0 [ 619.094709] [<ffffffff81b4294a>] ? nf_hook_slow+0x4a/0x1b0 [ 619.094714] [<ffffffff81b70abb>] ip_rcv+0x64b/0x750 [ 619.094721] [<ffffffff81b6f870>] ? inet_add_protocol+0x60/0x60 [ 619.094727] [<ffffffff81af1e93>] __netif_receive_skb_core+0xd53/0x13e0 [ 619.094735] [<ffffffff81bb325f>] ? tcp4_gro_receive+0xef/0x2d0 [ 619.094743] [<ffffffff81bcd256>] ? inet_gro_receive+0x3b6/0x4e0 [ 619.094749] [<ffffffff81ae794a>] ? gro_pull_from_frag0+0x16a/0x190 [ 619.094755] [<ffffffff81af25a5>] __netif_receive_skb+0x85/0xe0 [ 619.094762] [<ffffffff81af2928>] netif_receive_skb_internal+0xf8/0x160 [ 619.094768] [<ffffffff81ae789d>] ? gro_pull_from_frag0+0xbd/0x190 [ 619.094774] [<ffffffff81b29f6c>] ? eth_type_trans+0x14c/0x280 [ 619.094780] [<ffffffff81af44d5>] napi_gro_frags+0x3e5/0x400 [ 619.094792] [<ffffffffa001bd85>] t4_ethrx_handler+0x8e5/0x9f0 [cxgb4] [ 619.094799] [<ffffffff81b796c9>] ? ip_finish_output+0x179/0x330 [ 619.094809] [<ffffffffa001af55>] process_responses+0x735/0x8b0 [cxgb4] [ 619.094815] [<ffffffff81b798f4>] ? ip_output+0x74/0x110 [ 619.094822] [<ffffffff81b788d1>] ? ip_local_out+0x81/0xa0 [ 619.094828] [<ffffffff8103e532>] ? native_sched_clock+0x82/0x1a0 [ 619.094834] [<ffffffff81187fa9>] ? trace_clock_local+0x9/0x10 [ 619.094840] [<ffffffff81198672>] ? rb_reserve_next_event+0x642/0x750 [ 619.094849] [<ffffffff81193e82>] ? rb_set_commit_to_write+0x212/0x350 [ 619.094861] [<ffffffffa001b186>] napi_rx_handler+0xb6/0x260 [cxgb4] [ 619.094868] [<ffffffff81af3b4c>] napi_poll+0x1fc/0x3d0 [ 619.094873] [<ffffffff81198985>] ? ring_buffer_lock_reserve+0x205/0x260 [ 619.094881] [<ffffffff811933b3>] ? rb_event_data+0x53/0xa0 [ 619.094886] [<ffffffff81af3e1e>] net_rx_action+0xfe/0x2f0 [ 619.094893] [<ffffffff811b88e9>] ? trace_event_buffer_commit+0x179/0x2b0 [ 619.094901] [<ffffffff8109b225>] ? trace_event_raw_event_softirq+0x115/0x150 [ 619.094909] [<ffffffff81deed4d>] __do_softirq+0x1cd/0x4e1 [ 619.094916] [<ffffffff8109a868>] ? irq_exit+0x58/0x110 [ 619.094925] [<ffffffff81dedf1c>] do_softirq_own_stack+0x1c/0x30 [ 619.094929] <EOI> [<ffffffff8109a458>] do_softirq+0x68/0x70 [ 619.094940] [<ffffffff8109a629>] __local_bh_enable_ip+0xf9/0x160 [ 619.094946] [<ffffffff81bf79bf>] ipt_do_table+0x30f/0x620 [ 619.094954] [<ffffffff81bfae3f>] iptable_mangle_hook+0xef/0x1d0 [ 619.094960] [<ffffffff81b426c4>] nf_iterate+0xa4/0x110 [ 619.094966] [<ffffffff81b4294a>] nf_hook_slow+0x4a/0x1b0 [ 619.094972] [<ffffffff81b7884b>] __ip_local_out+0xfb/0x100 [ 619.094978] [<ffffffff81b74f00>] ? neigh_key_eq32+0x20/0x20 [ 619.094984] [<ffffffff81b78882>] ip_local_out+0x32/0xa0 [ 619.094990] [<ffffffff81b792b1>] ip_queue_xmit+0x251/0x4f0 [ 619.094997] [<ffffffff81b9bf19>] ? tcp_options_write+0x279/0x350 [ 619.095003] [<ffffffff81b9c7bb>] tcp_transmit_skb+0x7cb/0xe70 [ 619.095010] [<ffffffff81b9e942>] tcp_send_ack+0x182/0x1c0 [ 619.095016] [<ffffffff81b96093>] __tcp_ack_snd_check+0x63/0x100 [ 619.095023] [<ffffffff81b99d9f>] tcp_rcv_established+0x3ff/0x780 [ 619.095029] [<ffffffff81baa13e>] tcp_v4_do_rcv+0x10e/0x3a0 [ 619.095035] [<ffffffff81b86969>] tcp_prequeue_process+0x169/0x190 [ 619.095042] [<ffffffff81b84e32>] ? tcp_cleanup_rbuf+0x72/0x280 [ 619.095049] [<ffffffff81b8b6ed>] tcp_recvmsg+0xfed/0x12d0 [ 619.095057] [<ffffffff812bb776>] ? __fget_light+0x96/0x100 [ 619.095064] [<ffffffff81bcc4b9>] inet_recvmsg+0xa9/0xe0 [ 619.095071] [<ffffffff81ac13da>] sock_recvmsg+0x4a/0x60 [ 619.095078] [<ffffffff81ac343d>] SYSC_recvfrom+0x13d/0x1d0 [ 619.095085] [<ffffffff8100276e>] ? do_audit_syscall_entry+0xae/0xe0 [ 619.095093] [<ffffffff81002a1e>] ? syscall_trace_enter_phase2+0x27e/0x2f0 [ 619.095100] [<ffffffff81ac34de>] SyS_recvfrom+0xe/0x10 [ 619.095105] [<ffffffff81003426>] do_syscall_64+0xa6/0x170 [ 619.095111] [<ffffffff81002479>] ? prepare_exit_to_usermode+0x49/0x80 [ 619.095118] [<ffffffff81dec4fc>] entry_SYSCALL64_slow_path+0x25/0x25 Thanks, Arjun.