RE: [Fwd: [E1000] NAPI re-insertion w/ changes]

Feldman, Scott writes:

 > Easy enough to revert back.  I don't think we've lost any of the
 > non-perf benefits of NAPI, and if testing shows no meaningful perf
 > difference, let's let Occam's razor rule.

Hello!

Here is a hi-perf routing/DoS to start with...

10 Million pkts injected at high speed into eth2 and forwarded to eth3. Rx and 
Tx buffers are 256 and HW_FLOW is disabled and RxIntDelay=1. Which is same 
parameters as we use for production systems. As seen link now flaps. 
Eventually can hw_flowcontrol and interrupt delays help this... but thats not 
an option at least not for us. 


Twist:        New                        Old
             ====================================
Input rate:    680 (due to link drop)    820 kpps
T-put:         309                       385 kpps 
RX irq's:    78963                       434


Cheers.
						--ro


e1000 2.5.66 NAPI (680 kpps in)
-------------------------------
NETDEV WATCHDOG: eth2: transmit timed out
e1000: eth2 NIC Link is Up 1000 Mbps Full Duplex

Iface   MTU Met  RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR Flags
eth2   1500   0 4175392 7277442 7277442 4696985     18      0      0      0 BRU
eth3   1500   0      1      0      0      0 4555009      0      0      0 BRU

 26:      78963   IO-APIC-level  eth2
 27:      92979   IO-APIC-level  eth3


e1000 2.5.66 NAPI with patch (820 kpps in)
------------------------------------------
Iface   MTU Met  RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR Flags
eth2   1500   0 4708196 8173469 8173469 5291805     18      0      0      0 BRU
eth3   1500   0      1      0      0      0 4708135      0      0      0 BRU

 26:        434   IO-APIC-level  eth2
 27:      74584   IO-APIC-level  eth3

--- e1000_main.c.orig   2003-03-27 14:38:02.000000000 +0100
+++ e1000_main.c        2003-03-27 16:43:38.000000000 +0100

@@ -2000,9 +2000,12 @@
        }
 
 #ifdef CONFIG_E1000_NAPI
-       /* Don't disable interrupts - rely on h/w interrupt
-        * moderation to keep interrupts low.  netif_rx_schedule
-        * is NOP if already polling. */
+       /* Disable interrupts and register for poll. The flush 
+          of the posted write is intentionally left out.
+       */
+
+       atomic_inc(&adapter->irq_sem);
+       E1000_WRITE_REG(&adapter->hw, IMC, ~0);
        netif_rx_schedule(netdev);
 #else
        for(i = 0; i < E1000_MAX_INTR; i++)

@@ -2024,18 +2027,17 @@
        struct e1000_adapter *adapter = netdev->priv;
        int work_to_do = min(*budget, netdev->quota);
        int work_done = 0;
-
-       while(work_done < work_to_do)
-               if(!e1000_clean_rx_irq(adapter, &work_done, work_to_do) &&
-                  !e1000_clean_tx_irq(adapter))
-                       break;
+
+       e1000_clean_tx_irq(adapter);
+       e1000_clean_rx_irq(adapter, &work_done, work_to_do);
 
        *budget -= work_done;
        netdev->quota -= work_done;
 
-       if(work_done < work_to_do)
+       if(work_done < work_to_do) {
                netif_rx_complete(netdev);
-
+               e1000_irq_enable(adapter);
+       }
        return (work_done >= work_to_do);
 }
 #endif

Robert.Olsson@data.slu.se said:

10 Million pkts injected at high speed into eth2 and forwarded to eth3. Rx and 
Tx buffers are 256 and HW_FLOW is disabled and RxIntDelay=1. Which is same 
parameters as we use for production systems. As seen link now flaps. 
Eventually can hw_flowcontrol and interrupt delays help this... but thats not 
an option at least not for us. 


Twist:        New                        Old
             ====================================
Input rate:    680 (due to link drop)    820 kpps
T-put:         309                       385 kpps 
RX irq's:    78963                       434

I've seen pretty much the same thing. I plotted throughput vs. offered
load for e1000 4.4.12-k1, 4.4.19-k3, and 5.0.43-k1 (all backported to
2.4.20). A summary with graphs is at:

http://gtf.org/lunz/linux/net/perf/

5.0.43 seems to be a significant regression, both in terms of throughput
and CPU load.

-- 
Jason Lunz			Reflex Security
lunz@reflexsecurity.com		http://www.reflexsecurity.com/

Jason Lunz writes:

 > I've seen pretty much the same thing. I plotted throughput vs. offered
 > load for e1000 4.4.12-k1, 4.4.19-k3, and 5.0.43-k1 (all backported to
 > 2.4.20). A summary with graphs is at:
 > 
 > http://gtf.org/lunz/linux/net/perf/
 
 Illustrative.

 > 5.0.43 seems to be a significant regression, both in terms of throughput
 > and CPU load.
 
 Can you test the patch I sent for 5.0.43 with your equipment/setup?
 

 Cheers.
						--ro

Robert.Olsson@data.slu.se said:

 Can you test the patch I sent for 5.0.43 with your equipment/setup?

We have a winner! I called the driver with your patch 5.0.43-k1-ro1. I
also removed all the cyclesoak lines; all they really show is that
throughput is really sensitive to how much CPU time ksoftirqd gets.

http://gtf.org/lunz/linux/net/perf/ is still the url.

Jason

Jason Lunz writes:
 > Robert.Olsson@data.slu.se said:
 > >  Can you test the patch I sent for 5.0.43 with your equipment/setup?
 > 
 > We have a winner! I called the driver with your patch 5.0.43-k1-ro1. I
 > also removed all the cyclesoak lines; all they really show is that
 > throughput is really sensitive to how much CPU time ksoftirqd gets.
 > 
 > http://gtf.org/lunz/linux/net/perf/ is still the url.

 Excellent!

 We have a flat performance curve at ~350 kpps at any load and packet size 
 again. Of course this can be improved further.

 I think Intel still did the hard work but left the "fine" tuning and testing 
 for us. :-) 

 I guess we propose the patch to maintainers.

 Cheers
						--ro