What was your cpu utilization like in the bound vs split scenarios?
Does your e1000 driver have transmit interrupts enabled or disabled?

I'd be really interested to see the results with two flows in opposite
directions.

	Jon

-----Original Message-----
From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com]On
Behalf Of Robert Olsson
Sent: Thursday, October 17, 2002 8:01 AM
To: bert hubert
Cc: Hyochang Nam; niv@us.ibm.com; netdev@oss.sgi.com
Subject: Re: [Question] SMP for Linux



bert hubert writes:
 > On Thu, Oct 17, 2002 at 11:29:28AM +0900, Hyochang Nam wrote:
 > > Many people helped me to solve the interrupt 
distribution problem.
 > > We tested the throughput of Layer 3 forwarding on a SMP machine
 > > which equips two Zero proessor(2Ghz). This is our results:
 > >   -------------------------
 > >        SMP    |  No SMP
 > >   -------------------------
 > >     230 Mbps  | 330 Mbps
 > >   -------------------------
 > 
 > There is something called 'irq affinity' which may be 
interesting for you.
 > See here: 
http://www.dell.com/us/en/esg/topics/power_ps1q02-morse.htm
 > 
 > /proc/irq/?/smp_affinity

 Hello!

 Not always good for routing... Were you still get the 
problem were one
 interface is the output device from devices bound to different CPU's.

 TX-ring can hold skb's from many CPU's so a lot of cache 
bouncing happens 
 when kfree and skb_headerinit is run.

 I've played with some to code to re-route the skb freeing to the CPU
 where it was processed this to minimize cache bouncing and I've seen 
 some good effects of this.

 And to be fair with SMP you should compare multiple flows to 
see if you 
 can get any aggregated performance from SMP.

 An experiment...
 
 Single flow eth0->eth1 w. e1000 NAPI. 2.4.20-pre5. PIII @ 2x933 MHz

 Bound = eth0, eth1 is bound to same CPU.
 Split = eth0, eth1 is bound to differnt CPU's.
 Free  = unbound.

 SMP routing performance
 =======================
 
Bound   Free  Split   "kfree-route"
 ---------------------------------
 421     354    331                 kpps
 491     348    317            437  kpps w. skb recycling


 UP routing performance
 ======================
 494 kpps
 593 kpps w. skb recycling


 With SMP test "kfree-route" the interfaces are not bound to 
any CPU still 
 we now getting closer to "bound" (where both eth0, eth1 is 
bond to the same 
 CPU). 

 But yes UP is gives higher numbers in this single stream 
tests. Aggregated
 throughput tests are to be done.

 Cheers.

						--ro