First pass at MSG_FASTOPEN support in top-of-trunk netperf

Folks -

I have just checked-in to the top-of-trunk netperf 
(http://www.netperf.org/svn/netperf2/trunk) some changes which enable 
use of sendto(MSG_FASTOPEN) in a TCP_CRR test.  While I was checking the 
system calls I noticed that netperf was calling enable_enobufs() for 
every migrated test, not just a UDP_STREAM test (the one where it is 
needed), so I fixed that at the same time.  Baseline is taken with the 
fix in place.

MSG_FASTOPEN is used when one adds a test-specific -F option to the 
netperf command line:

netperf -t TCP_CRR -H <destination> -- -F

Just testing the client side from a VM on my workstation (running a 
net-next pulled just before 16:30 Pacific time) to my workstation itself 
I notice the following:

Without MSG_FASTOPEN:
raj@tardy-ubuntu-1204:~/netperf2_trunk$ src/netperf -H tardy.usa.hp.com 
-t TCP_CRR -i 3,30
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 
AF_INET to tardy.usa.hp.com () port 0 AF_INET : +/-2.500% @ 99% conf.  : 
demo
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       10.00    2092.33
16384  87380

With MSG_FASTOPEN
raj@tardy-ubuntu-1204:~/netperf2_trunk$ src/netperf -H tardy.usa.hp.com 
-t TCP_CRR -i 3,30 -- -F
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 
AF_INET to tardy.usa.hp.com () port 0 AF_INET : +/-2.500% @ 99% conf.  : 
demo
!!! WARNING
!!! Desired confidence was not achieved within the specified iterations.
!!! This implies that there was variability in the test environment that
!!! must be investigated before going further.
!!! Confidence intervals: Throughput      : 6.565%
!!!                       Local CPU util  : 0.000%
!!!                       Remote CPU util : 0.000%

Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       10.00    2590.18
16384  87380

There was a ~25% increase in TCP_CRR performance, even without the 
server actually accepting the magic TCP option.  Is that actually 
expected?  Admittedly, it eliminates a connect() call, but the sequence 
before was:

socket()
getsockopt(SO_SNDBUF)
getsockopt(SO_RCVBUF)
setsockopt(SO_REUSEADDR)
bind()
connect()
sendto(4, "n", 1, 0, NULL, 0)           = 1
recvfrom(4, "n", 1, 0, NULL, NULL)      = 1
recvfrom(4, "", 1, 0, NULL, NULL)       = 0
close(4)

and with MSG_FASTOPEN only the connect() goes away.  Unless connect() is 
really heavy compared to what sendto(MSG_FASTOPEN) does or something 
else has changed wrt behaviour.

Anyway, feel free to kick the tires on the netperf changes and let me 
know of any problems you encounter.

happy benchmarking,

rick jones