Thread (131 messages) 131 messages, 7 authors, 2010-01-21

Re: audit.c skb - tty race condition - was sky2 panic in 2.6.32.1 under load (new oops)

From: Michael Breuer <hidden>
Date: 2009-12-30 20:44:39
Also in: lkml

A couple more observations:

1) enabling auditd for runlevel 3 mitigates the issue
2) starting a remote x session (XDMCP) while under load and while auditd 
is already running also triggers the sky2 interrupt status messages - so 
maybe not tty1 - but some sort of X & auditd interaction. Even in this 
case, the frequency of the error messages is much less than when auditd 
is started in runlevel 5 for the first time.

On 12/30/2009 2:15 PM, Michael Breuer wrote:
And now looking at audit.c it seems reasonable that there is a race 
condition when auditd is started at roughly the same time as X. I'm 
guessing that the kaudit thread is fired up; the tty connected; and at 
the same time X grabs the tty. Somewhere in there an skb gets hosed 
and is then reused by whatever comes along - in my case sky2 as that's 
where the subsequent demand is. If the demand happens first, the 
contaminated skb (dk in what way yet) is probably waiting to manifest 
as some other bug that's been frustrating people.
On 12/30/2009 12:49 PM, Michael Breuer wrote:
quoted
On 12/30/2009 2:58 AM, Stephen Hemminger wrote:
quoted
On Wed, 30 Dec 2009 02:23:20 -0500
Michael Breuer[off-list ref]  wrote:
quoted
Ok - I called dump_txring from sky2_net_intr:
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2725,8 +2791,10 @@ static void sky2_watchdog(unsigned long arg)
   /* Hardware/software error handling */
   static void sky2_err_intr(struct sky2_hw *hw, u32 status)
   {
-       if (net_ratelimit())
+       if (net_ratelimit()) {
                  dev_warn(&hw->pdev->dev, "error interrupt
status=%#x\n", status);
+               dump_txring(hw, 0);
+       }

          if (status&  Y2_IS_HW_ERR)
                  sky2_hw_intr(hw);

And got this:
Dec 30 02:17:23 mail kernel: sky2 0000:06:00.0: error interrupt
status=0x40000008
Dec 30 02:17:23 mail kernel: sky2 0000:06:00.0: error interrupt
status=0x40000008
Dec 30 02:17:23 mail kernel: sky2 Tx ring pending=28...30 report=29 
done=29
Dec 30 02:17:23 mail kernel: sky2 Tx ring pending=28...30 report=29 
done=29
Dec 30 02:17:23 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x8
Dec 30 02:17:23 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x8
Dec 30 02:17:23 mail kernel: sky2 Tx ring pending=30...32 report=30 
done=31
Dec 30 02:17:23 mail kernel: sky2 Tx ring pending=30...32 report=30 
done=31
I notice that you have NOUVEAU Nvidia drivers loaded? The one 
difference in HW
between your board and mine is that I have ATI video card.
Seems the problem is linked to auditd and X11 (but not nouveau).

Today, I ran a bunch of scenarios. I first determined that the 
problem only manifest in runlevel 5. Next, this occurred with or 
without KMS and with or without nouveau. This happened whether or not 
I was logged in (local or remote), and regardless of window manager 
(xdm, gdm, kdm). I then checked to see what else was different 
between runlevel 3 and 5 - only thing was auditd. I disabled auditd 
and reran - no errors.

Now for the odd stuff:

The errors only manifest if the high throughput data transfer is 
initiated when the system is in runlevel 5 and auditd was started by 
init when transitioning from runlevel 3 to 5. For example, the 
following scenarios do not cause the errors to manifest:

runlevel3; start auditd runlevel 5; start transfer
runlevel3; chkconfig auditd off; runlevel5; start auditd; start transfer
runlevel3; start transfer (note: errors do not occur if I transition 
to runlevel 5 after the high bandwidth transfer has started)
runlevel3; startx; start transfer

The only way I get the problem to manifest is transition to runlevel 
5 with chkconfig auditd on (level 5 only) and then initate the 
windows backup.

I'm guessing that there is some sort of race condition happening 
between X (xdm/gdm/kdm/greeter?) and auditd that is somehow 
corrupting something. I'd hazard a more or less obvious guess that 
whatever's being corrupted differs when there is already a high 
throughput transfer under way.

-- 
To unsubscribe from this list: send the line "unsubscribe 
linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
-- 
To unsubscribe from this list: send the line "unsubscribe 
linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
  
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help