Andi Kleen wrote:

quoted

I believe the problem was that all of my ports were used up with
TIME_WAIT sockets and so it couldn't create more.  My test
case was similar to this:
    

Ah that's simple to solve then :- use more IP addresses and bind 
to them in RR in your user program.

Arguably the Linux TCP code should be able to do this by itself
when enough IP addresses are available, but it's not very hard
to do in user space using bind(2)

BTW it's also an very unusual case -- in most cases there are more
remote IP addresses
  

This could be done, but it does decrease our options for testing certain 
scenarios.

quoted

So, is there a better way to max out the connections per second without 
having to use tcp_tw_recycle?
    

Well did you profile where the bottle necks were?

Perhaps also just increase the memory allowed for TCP sockets.
  

I may be missing something, but I believe the issue is that the sockets 
wait around a while (maybe 30 seconds
or so) in TIME_WAIT state.  So, even if we use all 64k of the local port 
range, that will limit us to about 2000 new sockets
per second, as we have to wait for old ones to transition out of TIME_WAIT.

I guess I could probably decrease TIME_WAIT, but then all of my 
connections would be affected, not just the
ones on the ports creating very large numbers of connections per 
second.  From 'man tcp', it does not seem
I can set the TIME_WAIT on a per-socket basis.

I don't know exactly how the tcp_tw_recycle works, but it seems like it 
could be made to only
take affect when all local ports are used up in TIME_WAIT.  It could 
then recycle the oldest one
as a new socket is requested.  For any normal program, it would be very 
unlikely to ever need to
recycle in this case because there would be enough free IP/port pairs 
available.  But, for weird things
like my own, at least it could be made to work w/out hacking the global 
TIME_WAIT.

Thanks,
Ben

-- 
Ben Greear [off-list ref] 
Candela Technologies Inc  http://www.candelatech.com