Re: Early SPECWeb99 results on 2.5.33 with TSO on e1000

 >

The real question is why NAPI causes so much more work for the client.

 >
[Just a summary from my results from last year. All testing with a 
simple NIC without hw interrupt mitigation, on a Cyrix P150]

My assumption was that NAPI increases the cost of receiving a single 
packet: instead of one hw interrupt with one device access (ack 
interrupt) and the softirq processing, the hw interrupt must ack & 
disable the interrupt, then the processing occurs in softirq context, 
and the interrupts are reenabled at softirq context.

The second point was that interrupt mitigation must remain enabled, even 
with NAPI: the automatic mitigation doesn't work with process space 
limited loads (e.g. TCP: backlog queue is drained quickly, but the 
system is busy processing the prequeue or receive queue)

jamal, it is possible that a driver uses both napi and the normal 
interface, or would that break fairness?
Use netif_rx, until it returns dropping. If that happens, disable the 
interrupt, and call netif_rx_schedule().

Is it possible to determine the average number of packets that are 
processed for each netif_rx_schedule()?

--
	Manfred

   From: Manfred Spraul [off-list ref]
   Date: Fri, 06 Sep 2002 20:35:08 +0200

   The second point was that interrupt mitigation must remain enabled, even 
   with NAPI: the automatic mitigation doesn't work with process space 
   limited loads (e.g. TCP: backlog queue is drained quickly, but the 
   system is busy processing the prequeue or receive queue)

Not true.  NAPI is in fact a %100 replacement for hw interrupt
mitigation strategies.  The cpu usage elimination afforded by
hw interrupt mitigation is also afforded by NAPI and even more
so by NAPI.
   
See Jamal's paper.

Franks a lot,
David S. Miller
davem@redhat.com

David S. Miller wrote:

   From: Manfred Spraul [off-list ref]
   Date: Fri, 06 Sep 2002 20:35:08 +0200

   The second point was that interrupt mitigation must remain enabled, even 
   with NAPI: the automatic mitigation doesn't work with process space 
   limited loads (e.g. TCP: backlog queue is drained quickly, but the 
   system is busy processing the prequeue or receive queue)

Not true.  NAPI is in fact a %100 replacement for hw interrupt
mitigation strategies.  The cpu usage elimination afforded by
hw interrupt mitigation is also afforded by NAPI and even more
so by NAPI.
   
See Jamal's paper.

I've read his paper: it's about MLFFR. There is no alternative to NAPI 
if packets arrive faster than they are processed by the backlog queue.

But what if the backlog queue is empty all the time? Then NAPI thinks 
that the system is idle, and reenables the interrupts after each packet :-(

In my tests, I've used a pentium class system (I have no GigE cards - 
that was the only system where I could saturate the cpu with 100MBit 
ethernet). IIRC 30% cpu time was needed for the copy_to_user(). The 
receive queue was filled, the backlog queue empty. With NAPI, I got 1 
interrupt for each packet, with hw interrupt mitigation the throughput 
was 30% higher for MTU 600.

Dave, do you have interrupt rates from the clients with and without NAPI?

--
	Manfred

   From: Manfred Spraul [off-list ref]
   Date: Fri, 06 Sep 2002 21:40:09 +0200
   
   Dave, do you have interrupt rates from the clients with and without NAPI?

Robert does.

Manfred Spraul:

quoted

But what if the backlog queue is empty all the time? Then NAPI thinks 
that the system is idle, and reenables the interrupts after each packet :-(

Yes and this happens even with without NAPI. Just set RxIntDelay=X and send 
pkts at >= X+1 interval.

quoted

  Dave, do you have interrupt rates from the clients with and without NAPI?

DaveM:

Robert does.

Yes we get into this interesting discussion now... Since with NAPI we can 
safely use RxIntDelay=0 (e1000 terminologi). With the classical IRQ we simply 
had to add latency (RxIntDelay of 64-128 us common for GIGE) this just to 
survive at higher speeds  (GIGE max is 1.48 Mpps) and with the interrupt latency 
also comes higher network latencies... IMO this delay was a "work-around" 
for the old interrupt scheme.

So we now have the option of removing it... But we are trading less latency 
for for more interrupts. So yes Manfred is correct...

So is there a decent setting/compromise? 

Well first approximation is just to do just what DaveM suggested.
RxIntDelay=0. This solved many problems with buggy hardware and complicated 
tuning and RxIntDelay used to be combined with other mitigation parameters to 
compensate for different packets sizes etc. This leading to very "fragile" 
performance where a NIC could perform excellent w. single TCP stream but 
to be seriously broke in many other tests. So tuning to just one "test" 
can cause a lot of mis-tuning as well. 

Anyway. A tulip NAPI variant added mitigation when we reached "some load" to 
avoid the static interrupt delay. (Still keeping things pretty simple):
 
Load   "Mode"
-------------------
Lo  1) RxIntDelay=0
Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring)
Hi  3) Consecutive polling. No RX interrupts.

Is it worth the effort? 

For SMP w/o affinity the delay could eventually reduce the cache bouncing 
since the packets becomes more "batched" at cost the of latency of course. 
We use RxIntDelay=0 for production use. (IP-forwarding on UP)

Cheers.

						--ro

Robert Olsson wrote:
 >
 > Anyway. A tulip NAPI variant added mitigation when we reached "some
 > load" to  avoid the static interrupt delay. (Still keeping things
 > pretty simple):
 >
 > Load   "Mode"
 > -------------------
 > Lo  1) RxIntDelay=0
 > Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring)
 > Hi  3) Consecutive polling. No RX interrupts.
 >
Sounds good.

The difficult part is when to go from Lo to Mid. Unfortunately my tulip 
card is braindead (LC82C168), but I'll try to find something usable for 
benchmarking

In my tests with the winbond card, I've switched at a fixed packet rate:

< 2000 packets/sec: no delay
 > 2000 packets/sec: poll rx at 0.5 ms



--
	Manfred

 >  > Load   "Mode"
 >  > -------------------
 >  > Lo  1) RxIntDelay=0
 >  > Mid 2) RxIntDelay=fix (When we had X pkts on the RX ring)
 >  > Hi  3) Consecutive polling. No RX interrupts.

Manfred Spraul writes:

 > Sounds good.
 > 
 > The difficult part is when to go from Lo to Mid. Unfortunately my tulip 
 > card is braindead (LC82C168), but I'll try to find something usable for 
 > benchmarking

 21143 for tulip's. Well any NIC with "RxIntDelay"  should do.

 > In my tests with the winbond card, I've switched at a fixed packet rate:
 > 
 > < 2000 packets/sec: no delay
 >  > 2000 packets/sec: poll rx at 0.5 ms

 I was experimenting with all sorts of moving averages but never got a good 
 correlation with bursty network traffic as this level of resolution. The 
 only measure I found fast and simple enough for this was the number of 
 packets on the RX ring as I mentioned.


 Cheers.
						--ro