WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()

We just got an automated report of this WARN being hit in 3.5

 > backtrace:
 > :WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()
 > :Hardware name: P5Q DELUXE
 > :recvmsg bug: copied 99F66400 seq 99F6A4A8 rcvnxt 99F6CDAD fl 0
 > : [<ffffffff811848e6>] do_sync_read+0xe6/0x120
 > : [<ffffffff81275eea>] ? inode_has_perm.isra.31.constprop.61+0x2a/0x30
 > : [<ffffffff81272f32>] ? security_file_permission+0x92/0xb0
 > : [<ffffffff81184d81>] ? rw_verify_area+0x61/0xf0
 > : [<ffffffff811852cd>] vfs_read+0x15d/0x180
 > : [<ffffffff8118533a>] sys_read+0x4a/0x90
 > : [<ffffffff8160fc29>] system_call_fastpath+0x16/0x1b

1594                         /* Now that we have two receive queues this
1595                          * shouldn't happen.
1596                          */
1597                         if (WARN(before(*seq, TCP_SKB_CB(skb)->seq),
1598                                  "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n",
1599                                  *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt,
1600                                  flags))
1601                                 break;


Hopefully this means more to you guys than it does to me.

	Dave

On Mon, Aug 06, 2012 at 11:46:16AM -0400, Dave Jones wrote:
 > We just got an automated report of this WARN being hit in 3.5
 > 
 >  > backtrace:
 >  > :WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()
 >  > :Hardware name: P5Q DELUXE
 >  > :recvmsg bug: copied 99F66400 seq 99F6A4A8 rcvnxt 99F6CDAD fl 0
 >  > : [<ffffffff811848e6>] do_sync_read+0xe6/0x120
 >  > : [<ffffffff81275eea>] ? inode_has_perm.isra.31.constprop.61+0x2a/0x30
 >  > : [<ffffffff81272f32>] ? security_file_permission+0x92/0xb0
 >  > : [<ffffffff81184d81>] ? rw_verify_area+0x61/0xf0
 >  > : [<ffffffff811852cd>] vfs_read+0x15d/0x180
 >  > : [<ffffffff8118533a>] sys_read+0x4a/0x90
 >  > : [<ffffffff8160fc29>] system_call_fastpath+0x16/0x1b
 > 
 > 1594                         /* Now that we have two receive queues this
 > 1595                          * shouldn't happen.
 > 1596                          */
 > 1597                         if (WARN(before(*seq, TCP_SKB_CB(skb)->seq),
 > 1598                                  "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n",
 > 1599                                  *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt,
 > 1600                                  flags))
 > 1601                                 break;
 > 
 > 
 > Hopefully this means more to you guys than it does to me.

We're getting more reports of this happening too.

This guy managed to hit both of the recvmsg BUG's.

https://bugzilla.redhat.com/show_bug.cgi?id=846996
https://bugzilla.redhat.com/show_bug.cgi?id=846991

The first reporter claimed to be doing nothing special, just browsing with google chrome.

Anyone ?

	Dave

On Fri, 2012-08-10 at 16:29 -0400, Dave Jones wrote:

On Mon, Aug 06, 2012 at 11:46:16AM -0400, Dave Jones wrote:
 > We just got an automated report of this WARN being hit in 3.5
 > 
 >  > backtrace:
 >  > :WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()
 >  > :Hardware name: P5Q DELUXE
 >  > :recvmsg bug: copied 99F66400 seq 99F6A4A8 rcvnxt 99F6CDAD fl 0
 >  > : [<ffffffff811848e6>] do_sync_read+0xe6/0x120
 >  > : [<ffffffff81275eea>] ? inode_has_perm.isra.31.constprop.61+0x2a/0x30
 >  > : [<ffffffff81272f32>] ? security_file_permission+0x92/0xb0
 >  > : [<ffffffff81184d81>] ? rw_verify_area+0x61/0xf0
 >  > : [<ffffffff811852cd>] vfs_read+0x15d/0x180
 >  > : [<ffffffff8118533a>] sys_read+0x4a/0x90
 >  > : [<ffffffff8160fc29>] system_call_fastpath+0x16/0x1b
 > 
 > 1594                         /* Now that we have two receive queues this
 > 1595                          * shouldn't happen.
 > 1596                          */
 > 1597                         if (WARN(before(*seq, TCP_SKB_CB(skb)->seq),
 > 1598                                  "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n",
 > 1599                                  *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt,
 > 1600                                  flags))
 > 1601                                 break;
 > 
 > 
 > Hopefully this means more to you guys than it does to me.

We're getting more reports of this happening too.

This guy managed to hit both of the recvmsg BUG's.

https://bugzilla.redhat.com/show_bug.cgi?id=846996
https://bugzilla.redhat.com/show_bug.cgi?id=846991

The first reporter claimed to be doing nothing special, just browsing with google chrome.

Anyone ?

Might be a driver issue with LRO / GRO/ TCP coalescing

So we need a _lot_ of details.

On Fri, Aug 10, 2012 at 10:40:18PM +0200, Eric Dumazet wrote:
 
 > >  >  > backtrace:
 > >  >  > :WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()
 > > 
 > > We're getting more reports of this happening too.
 > 
 > Might be a driver issue with LRO / GRO/ TCP coalescing
 > 
 > So we need a _lot_ of details.

I looked over all the existing reports of this. It's a bit of a mess.
It seems to affect multiple drivers.
A big problem is that the network stack seems to get in a really screwed up state
when this happens, and the automated bug filer files these corrupted traces.

A secondary problem is that because these are automated reports, not all users
are responsive to answer questions afterwards...

Anyway, what I've gathered so far..

3.4-rc	i2400m_usb      https://bugzilla.redhat.com/show_bug.cgi?id=807114

unknown (mangled traces) https://bugzilla.redhat.com/show_bug.cgi?id=841016

possibly e1000e (also mangled) https://bugzilla.redhat.com/show_bug.cgi?id=841769

3.4.4	unknown (mangled)	https://bugzilla.redhat.com/show_bug.cgi?id=849275
 (same guy, probably e1000e)	https://bugzilla.redhat.com/show_bug.cgi?id=849276

3.5.0	r8712u  https://bugzilla.redhat.com/show_bug.cgi?id=845853

3.5.0	unknown https://bugzilla.redhat.com/show_bug.cgi?id=846991


any ideas on any additional debug printk's we could add ?

	Dave

On Mon, 2012-08-20 at 12:20 -0400, Dave Jones wrote:

On Fri, Aug 10, 2012 at 10:40:18PM +0200, Eric Dumazet wrote:
 
 > >  >  > backtrace:
 > >  >  > :WARNING: at net/ipv4/tcp.c:1598 tcp_recvmsg+0x641/0xd30()
 > > 
 > > We're getting more reports of this happening too.
 > 
 > Might be a driver issue with LRO / GRO/ TCP coalescing
 > 
 > So we need a _lot_ of details.

I looked over all the existing reports of this. It's a bit of a mess.
It seems to affect multiple drivers.
A big problem is that the network stack seems to get in a really screwed up state
when this happens, and the automated bug filer files these corrupted traces.

A secondary problem is that because these are automated reports, not all users
are responsive to answer questions afterwards...

Anyway, what I've gathered so far..

3.4-rc	i2400m_usb      https://bugzilla.redhat.com/show_bug.cgi?id=807114

unknown (mangled traces) https://bugzilla.redhat.com/show_bug.cgi?id=841016

possibly e1000e (also mangled) https://bugzilla.redhat.com/show_bug.cgi?id=841769

3.4.4	unknown (mangled)	https://bugzilla.redhat.com/show_bug.cgi?id=849275
 (same guy, probably e1000e)	https://bugzilla.redhat.com/show_bug.cgi?id=849276

3.5.0	r8712u  https://bugzilla.redhat.com/show_bug.cgi?id=845853

3.5.0	unknown https://bugzilla.redhat.com/show_bug.cgi?id=846991


any ideas on any additional debug printk's we could add ?

	Dave

We had a generic bug in old kernels, so you might ignore too old
reports.

This was fixed in commit 1ca7ee30630e1022dbcf1b51be20580815ffab73
(tcp: take care of overlaps in tcp_try_coalesce())