Hi Bill,

I started musing if once one side's transmitter got the upper hand, it 
might somehow defer the processing of received packets, causing the 
resultant ACKs to be delayed and thus further slowing down the other 
end's transmitter.  I began to wonder if the txqueuelen could have an 
affect on the TCP performance behavior.  I normally have the txqueuelen 
set to 10000 for 10-GigE testing, so decided to run a test with 
txqueuelen set to 200 (actually settled on this value through some 
experimentation).  Here is a typical result:

[bill@chance4 ~]$ nuttcp -f-beta -Itx -w2m 192.168.6.79 & nuttcp -f-beta -Irx -r -w2m 192.168.6.79
tx:  1120.6345 MB /  10.07 sec =  933.4042 Mbps 12 %TX 9 %RX 0 retrans
rx:  1104.3081 MB /  10.09 sec =  917.7365 Mbps 12 %TX 11 %RX 0 retrans

This is significantly better, but there was more variability in the
results.  The above was with TSO enabled.  I also then ran a test
with TSO disabled, with the following typical result:

[bill@chance4 ~]$ nuttcp -f-beta -Itx -w2m 192.168.6.79 & nuttcp -f-beta -Irx -r -w2m 192.168.6.79
tx:  1119.4749 MB /  10.05 sec =  934.2922 Mbps 13 %TX 9 %RX 0 retrans
rx:  1131.7334 MB /  10.05 sec =  944.8437 Mbps 15 %TX 12 %RX 0 retrans

This was a little better yet and getting closer to expected results.

We'll also try changing txqueuelen.  I have not looked, but I suppose that 
this is set to the default value of 1000.  We'd be delighted to see 
full-duplex performance that was consistent and greater than 900 Mb/s x 2.

I do have some other test systems at work that I might be able to try 
with newer kernels and/or drivers or maybe even with other vendor's GigE 
NICs, but I won't be back to work until early next week sometime.

Bill, we'd be happy to give you root access to a couple of our systems 
here if you want to do additional testing.  We can put the latest drivers 
on them (and reboot if/as needed).  If you want to do this, please just 
send an ssh public key to Carsten.

Cheers,
 	Bruce