[PATCH] r8169: Reduce looping in the interrupt handler.

As of 2.6.30 I have been observing soft lockups and netdev watchdog
timeouts caused by looping in the r8169 interrupt handler.

- Introduce a hard limit to the maximum number of times we will loop in
  the interrupt handler, and print a message when we hit it.

- Break out of the loop if after looking none of the events in status
  are events we expect to be delivered by an interrupt.

With just the hard limit and message bits of my patch in my test case
I get hit my limit of 10 loops 12 times.  After filtering by intr_mask
and intr_event I don't get any warnings.

Any complaints from those who know the driver better than I?

Signed-off-by: Eric W. Biederman <redacted>
---
 drivers/net/r8169.c |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 3b19e0c..2214945 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c

@@ -61,6 +61,8 @@ static const int multicast_filter_limit = 32;
 /* MAC address length */
 #define MAC_ADDR_LEN	6
 
+#define MAX_INTR_LOOPS 10	/* Limit the msi acking loop from going crazy */
+
 #define MAX_READ_REQUEST_SHIFT	12
 #define RX_FIFO_THRESH	7	/* 7 means NO threshold, Rx buffer level before first PCI xfer. */
 #define RX_DMA_BURST	6	/* Maximum PCI burst, '6' is 1024 */

@@ -3552,6 +3554,7 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
 	void __iomem *ioaddr = tp->mmio_addr;
 	int handled = 0;
 	int status;
+	int count = 0;
 
 	/* loop handling interrupts until we have no new ones or
 	 * we hit a invalid/hotplug case.

@@ -3560,6 +3563,17 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
 	while (status && status != 0xffff) {
 		handled = 1;
 
+		if (count++ >= MAX_INTR_LOOPS) {
+			if (netif_msg_intr(tp) && net_ratelimit()) {
+				printk(KERN_INFO " %s Screaming irq "
+					"status %08x mask %08x event %08x "
+					"napi %08x\n",
+					dev->name, status, tp->intr_mask,
+					tp->intr_event,	tp->napi_event);
+			}
+			break;
+		}
+
 		/* Handle all of the error cases first. These will reset
 		 * the chip, so just exit the loop.
 		 */

@@ -3609,6 +3623,13 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
 		RTL_W16(IntrStatus,
 			(status & RxFIFOOver) ? (status | RxOverflow) : status);
 		status = RTL_R16(IntrStatus);
+
+		/* Ignore the parts of status that reflect more than
+		 * the enabled interrupts.
+		 */
+		smp_rmb();
+		if (!(status & tp->intr_mask & tp->intr_event))
+			break;
 	}
 
 	return IRQ_RETVAL(handled);

-- 
1.6.2.5

On Wed, 2009-08-26 at 00:58 -0700, Eric W. Biederman wrote:

As of 2.6.30 I have been observing soft lockups and netdev watchdog
timeouts caused by looping in the r8169 interrupt handler.

- Introduce a hard limit to the maximum number of times we will loop in
  the interrupt handler, and print a message when we hit it.

- Break out of the loop if after looking none of the events in status
  are events we expect to be delivered by an interrupt.

With just the hard limit and message bits of my patch in my test case
I get hit my limit of 10 loops 12 times.  After filtering by intr_mask
and intr_event I don't get any warnings.

Any complaints from those who know the driver better than I?

Have you tried this under a sustained heavy transmit load? I think you
may have reintroduced the problem the original patch was trying to
prevent -- wedging the MSI interrupt by not ACKing all outstanding
interrupt sources, masked or otherwise. I'll try to test this out on the
machine where I've seen the problem consistently, but it may be this
weekend.

+
+		/* Ignore the parts of status that reflect more than
+		 * the enabled interrupts.
+		 */
+		smp_rmb();
+		if (!(status & tp->intr_mask & tp->intr_event))
+			break;
 	}

This looks like an odd construct, since we're just about to go back the
while condition up top -- why not just mask it here and let the loop
handle it naturally?

On Wed, 2009-08-26 at 09:56 -0400, David Dillow wrote:

On Wed, 2009-08-26 at 00:58 -0700, Eric W. Biederman wrote:

quoted

diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 3b19e0c..2214945 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c

quoted

+
+		/* Ignore the parts of status that reflect more than
+		 * the enabled interrupts.
+		 */
+		smp_rmb();
+		if (!(status & tp->intr_mask & tp->intr_event))
+			break;
 	}

This looks like an odd construct, since we're just about to go back the
while condition up top -- why not just mask it here and let the loop
handle it naturally?

Never mind, I see what you are doing -- avoiding a false loop if we get
status == 0xffff. I still don't like the aesthetics of it, but it makes
sense, and I'll blame it on the card. :)

I should really get some caffeine before posting...