One-way Gigabit Ethernet TCP performance with Jumbo frames

Hi all,

Lately I've been testing out two Gigabit Ethernet adapters on Pentium 4 
Xeon platforms; onboard Intel 82544GC (e1000 driver) and onboard Broadcom 
BCM5701 (tg3 driver), and I'm experiencing some wierd behaviour on 
one-way tests (ping-ping). The machines I'm testing is connected back to 
back (i.e no switch) and are fairly fast systems (Dual Xeon 2.4 GHz, 1GB 
memory) configured to use Jumbo frames (9000 bytes).

With ping-pong traffic (the attached program run with -bo, or NetPipe 
default) the bandwidth performance is close to wire speed (123 MByte/sec) 
and the ping-pong/2 latency is ~30us with both GbE devices.

But, when running a one-way test (where one machine only sends, and the 
other only receives, i.e ping-ping) there is a serious dip in the 
performance curve at ~768 bytes and the bandwidth levels out at approx 
60 MByte/sec (about half of peak) regadless of application and GbE device. 

However, if the benchmark applications are started at 2048 bytes (and not 
0 which is default), or the MTU is set to standard 1500 bytes, there is no 
such "dip" in the performance curve.

Is there a new "silly window" syndrome going on here ? Both applications 
use TCP_NODELAY.

I'll appreciate any feedback, and I'm happy to assist in the debugging 
process testing out patches etc.

PS

Attached you'll find a yet another "bandwidth" program which measures TCP 
performance i three ways; ping-ping (one-way), ping-pong, and exchange 
(two way). Should compile fine with gcc -O2, and you must start a server 
process (-Ts) on one machine and a client process (-Tc -s <server>) on the 
other machine.

DS


Best regards,
 -- 
  Steffen Persvold   |       Scali AS      
 mailto:sp@scali.com |  http://www.scali.com
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY

Steffen Persvold writes:
 > Hi all,
 > 
 > Lately I've been testing out two Gigabit Ethernet adapters on Pentium 4 
 > Xeon platforms; onboard Intel 82544GC (e1000 driver) and onboard Broadcom 
 > BCM5701 (tg3 driver), and I'm experiencing some wierd behaviour on 
 > one-way tests (ping-ping). The machines I'm testing is connected back to 
 > back (i.e no switch) and are fairly fast systems (Dual Xeon 2.4 GHz, 1GB 
 > memory) configured to use Jumbo frames (9000 bytes).


 > But, when running a one-way test (where one machine only sends, and the 
 > other only receives, i.e ping-ping) there is a serious dip in the 
 > performance curve at ~768 bytes and the bandwidth levels out at approx 
 > 60 MByte/sec (about half of peak) regadless of application and GbE device. 

 I've seen similar problems... and most of the times this seems due to 
 incorrect tuned mitigation. Think of what happens if you don't have TX-
 interrupts enough to clean your TX-ring. Which means your app. can not
 fill it at full speed -- and as long you have RX traffic it contributes 
 with interrupts so the problem is not visile. 

 If you test IP-forwarding with RX soly on one interface and TX soly on the
 other and routing between them. You'll see drops at the qdisc in such case.

 Cheers.
						--ro