Ingo Molnar wrote:

* David Newall [off-list ref] wrote:
  

quoted

You really should start that capture, and on both client and server. 
You don't need to dump everything, only traffic to or from 
server:distcc.
    

It's not feasible. That box did in excess of 200 GB of network traffic 
in the past 7 hours alone.

You only need distcc traffic, and perhaps only after it's hung.  With
250k outstanding per socket, are you certain that no traffic was sent? 
Is it certain that one packet wasn't being sent each three minutes?  I
suppose you're right and the stack really is stuck, but this is such an
easy thing to check and eliminate that you should do so.  I suppose,
too, that you should trace the server-side processes and confirm that
they are waiting for socket input.  You should dump tcp (for the distcc
port) next time the problem recurs and also check that the server
processes are waiting for socket input.