Re: [Bugme-new] [Bug 16626] New: Machine hangs with EIP at skb_copy_and_csum_dev
From: Plamen Petrov <hidden>
Date: 2010-08-25 07:05:44
На 24.8.2010 г. 22:19, Eric Dumazet написа:
Le mardi 24 août 2010 à 20:25 +0300, Plamen Petrov a écrit :quoted
Above patch applied, and happy to report the machine now spits data in the logs instead of oopsing. Here is what we have now: [ 707.276981] ---[ end trace 75e4f8534893c910 ]--- [ 707.376998] 100: csum_start 306, offset 16, headroom 390, headlen 70, len 70 [ 707.477015] nr_frags=0 gso_size=0 [ 707.577031] [ 1021.032794] ---[ end trace 75e4f8534893c911 ]--- [ 1021.132812] 100: csum_start 306, offset 16, headroom 390, headlen 153, len 153 [ 1021.232828] nr_frags=0 gso_size=0 [ 1021.332844]Thanks ! csum_offset = 16. so its offsetof(struct tcphdr, check) maybe a bug in net/ipv4/netfilter/nf_nat_helper.c ? We should trace all spots where we set csum_start/csum_offset Or/And trace the skb content. Please add a : print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET, 16, 1, skb->head, skb_end_pointer(skb)-skb->head,true);
Done! See the results below.
call in skb_csum_start_bug(), right after the pr_err("\n") and before
the "return 1;"
int skb_csum_start_bug(const struct sk_buff *skb, int pos)
{
if (skb->ip_summed == CHECKSUM_PARTIAL) {
long csstart;
csstart = skb->csum_start - skb_headroom(skb);
if (WARN_ON(csstart> skb_headlen(skb))) {
int i;
pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
skb_headlen(skb), skb->len);
pr_err("nr_frags=%u gso_size=%u ",
skb_shinfo(skb)->nr_frags,
skb_shinfo(skb)->gso_size);
for (i = 0; i< skb_shinfo(skb)->nr_frags; i++) {
pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
}
pr_err("\n");
print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,
16, 1, skb->head, skb_end_pointer(skb) - skb->head, true);
return 1;
}
}
return 0;
}
I see you liked the previous one, here's some more. ;) This one is based on Linus' latest tree, hence the ID "2.6.36-rc2-FS-00210-geedff42".
[ 10.510191] XFS mounting filesystem md12 [ 10.693540] Ending clean XFS mount for filesystem: md12 [ 11.592737] IPv4 FIB: Using LC-trie version 0.409 [ 11.592827] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1 [ 11.677311] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1 [ 11.687604] tg3 0000:04:00.0: irq 44 for MSI/MSI-X [ 11.719166] ADDRCONF(NETDEV_UP): eth1: link is not ready [ 11.845858] sixxs_t: Disabled Privacy Extensions [ 14.815688] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex [ 14.815693] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX [ 14.815740] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 15.470040] tun0: Disabled Privacy Extensions [ 310.470021] ------------[ cut here ]------------ [ 310.570041] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133() [ 310.670050] Hardware name: PowerEdge SC440 [ 310.770060] Pid: 2960, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00210-geedff42 #1 [ 310.870069] Call Trace: [ 311.070087] [<c102d87c>] ? warn_slowpath_common+0x67/0x8c [ 311.270103] [<c12adca9>] ? skb_csum_start_bug+0x46/0x133 [ 311.470126] [<c12adca9>] ? skb_csum_start_bug+0x46/0x133 [ 311.670144] [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f [ 311.870167] [<c12adca9>] ? skb_csum_start_bug+0x46/0x133 [ 312.070191] [<c102456f>] ? __wake_up_sync_key+0x3c/0x52 [ 312.270207] [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf [ 312.470224] [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a [ 312.670238] [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc [ 312.870253] [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174 [ 313.070268] [<c12c3fb9>] ? nf_iterate+0x69/0x7c [ 313.270283] [<c12e89c6>] ? ip_finish_output+0x0/0x2b6 [ 313.470297] [<c12b013c>] ? dev_queue_xmit+0xc7/0x354 [ 313.670312] [<c12e89c6>] ? ip_finish_output+0x0/0x2b6 [ 313.870326] [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6 [ 314.070341] [<c12e8f61>] ? ip_output+0xa4/0xc3 [ 314.270355] [<c12e89c6>] ? ip_finish_output+0x0/0x2b6 [ 314.470370] [<c12e5049>] ? ip_forward_finish+0x39/0x44 [ 314.670385] [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f [ 314.870399] [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3 [ 315.070413] [<c12ad62b>] ? netif_receive_skb+0x5f/0x64 [ 315.270427] [<c12ad67e>] ? napi_gro_complete+0x4e/0x94 [ 315.470440] [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5 [ 315.670454] [<c12adba4>] ? napi_gro_receive+0x16/0x1f [ 315.870468] [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb [ 316.070483] [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1 [ 316.270498] [<c121ce78>] ? tg3_poll+0x43/0x194 [ 316.470512] [<c12ad7d3>] ? net_rx_action+0xcc/0x15b [ 316.670526] [<c1031cbd>] ? __do_softirq+0x7f/0xfa [ 316.870541] [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6 [ 317.070555] [<c10568ab>] ? move_native_irq+0x9/0x3e [ 317.270569] [<c1031d5f>] ? do_softirq+0x27/0x2a [ 317.470582] [<c1031ead>] ? irq_exit+0x63/0x68 [ 317.670596] [<c1003dda>] ? do_IRQ+0x44/0xa1 [ 317.870610] [<c10035c3>] ? do_device_not_available+0x0/0x49 [ 318.070624] [<c1002d29>] ? common_interrupt+0x29/0x30 [ 318.270639] [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a [ 318.370647] ---[ end trace df8deff2ad2a9760 ]--- [ 318.470656] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151 [ 318.570664] nr_frags=0 gso_size=0 [ 318.670671] [ 318.770680] skb data:00000000: 00 a4 27 cc 17 5e ef ec 00 1a a0 38 8a 1b 08 00 ..'..^.....8.... [ 318.870688] skb data:00000010: 45 00 00 b7 00 00 40 00 40 11 a4 62 c0 a8 0a 01 E.....@.@..b.... [ 318.970697] skb data:00000020: c0 a8 0a 82 00 35 f2 69 00 a3 96 88 d3 4e 81 80 .....5.i.....N.. [ 319.070706] skb data:00000030: 00 01 00 04 00 00 00 00 06 61 6b 61 6d 61 69 0d .........akamai. [ 319.170714] skb data:00000040: 73 6d 61 72 74 61 64 73 65 72 76 65 72 03 63 6f smartadserver.co [ 319.270723] skb data:00000050: 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00 81 0a m............... [ 319.370731] skb data:00000060: 00 28 06 61 6b 61 6d 61 69 0d 73 6d 61 72 74 61 .(.akamai.smarta [ 319.470739] skb data:00000070: 64 73 65 72 76 65 72 03 63 6f 6d 09 65 64 67 65 dserver.com.edge [ 319.570747] skb data:00000080: 73 75 69 74 65 03 6e 65 74 00 c0 36 00 00 00 00 suite.net..6.... [ 319.670756] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 34 ............E..4 [ 319.770764] skb data:000000a0: 0d 39 40 00 40 06 2f 89 7f 00 00 01 7f 00 00 01 .9@.@./......... [ 319.870772] skb data:000000b0: be 75 19 4e 14 43 0f 38 14 5d 49 65 00 00 00 00 .u.N.C.8.]Ie.... [ 319.970780] skb data:000000c0: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 57 ............E..W [ 320.070789] skb data:000000d0: 4b 40 40 00 40 06 f1 5e 7f 00 00 01 7f 00 00 01 K@@.@..^........ [ 320.170797] skb data:000000e0: 19 4e be 75 14 5d 49 65 14 43 0f 38 80 18 04 00 .N.u.]Ie.C.8.... [ 320.270805] skb data:000000f0: fe 4b 00 00 01 01 08 0a 00 00 02 03 00 00 02 03 .K.............. [ 320.370813] skb data:00000100: 00 a8 27 cc 00 00 00 00 00 00 00 00 00 00 00 00 ..'............. [ 320.470821] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 320.570829] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 320.670837] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 320.770845] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 320.870853] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 320.970862] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 321.070870] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 321.170878] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c .........\'....\ [ 321.270887] skb data:00000190: 27 ef 08 00 45 00 00 89 09 27 40 00 7f 06 46 74 '...E....'@...Ft [ 321.370895] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 8f 00 50 98 49 d7 bc ....[g.....P.I.. [ 321.470903] skb data:000001b0: ff 40 e9 4a 50 18 ff ff ac 4f 00 00 33 42 25 32 .@.JP....O..3B%2 [ 321.570911] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25 4sh%3D3%3B%24sw% [ 321.670919] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33 3D3; pid=5021743 [ 321.770927] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f 420310299783..Co [ 321.870936] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d okie2: $Version= [ 321.970944] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 1..Connection: K [ 322.070952] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00 00 eep-Alive....... [ 322.170960] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 322.270968] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 322.370977] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 322.470985] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 322.570993] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 322.671004] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
The rest is in the attached file, in case you need to run it trough some debugging app... Thanks, Plamen
Attachments
- skb-dump.txt [text/plain] 132217 bytes · preview