Re: [PATCH] net: reduce number of reference taken on sk_refcnt

David Miller a écrit :

From: David Miller <davem@davemloft.net>
Date: Sat, 09 May 2009 13:34:54 -0700 (PDT)

quoted

Consider the case where we always send some message on CPU A and
then process the ACK on CPU B.  We'll always be cancelling the
timer on a foreign cpu.

I should also mention that TCP has a peculiar optimization of timers
that is likely being thwarted by your workload.  It never deletes
timers under normal operation, it simply lets them still expire
and the handler notices that there is "nothing to do" and returns.

Yes, you refer to INET_CSK_CLEAR_TIMERS condition, never set.

But when the connection does shut down, we have to purge all of
these timers.

That could be another part of why you see timers in your profile.

Well, in my workload they should never expire, since application exchange
enough data on both direction, and they are no losses (Gigabit LAN context)

On machine acting as a server (the one I am focusing to, of course),
each incoming frame :

- Contains ACK for the previous sent frame
- Contains data provided by the client.
- Starts a timer for delayed ACK

Then server applications reacts and sends a new payload, and TCP stack
- Sends a frame including ACK for previous received frame
- Contains data provided by server application
- Starts a timer for retransmiting this frame if no ACK is received later.

So yes, each incoming and each outgoing frame is going to call mod_timer()

Problem is that incoming process is done by CPU 0 (the one that is dedicated
to NAPI processing because of stress situation, cpu 100% in softirq land),
and outgoing processing done by other cpus in the machine.

offsetof(struct inet_connection_sock, icsk_retransmit_timer)=0x208
offsetof(struct inet_connection_sock, icsk_delack_timer)=0x238

So there are cache line ping-pongs, but oprofile seems to point
to a spinlock contention in lock_timer_base(), I dont know why...
shouldnt (in my workload) delack_timer all belongs to cpu 0, and 
retransmit_timers to other cpus ? 

Or is mod_timer never migrates an already established timer ?

That would explain the lock contention on timer_base, we should
take care of it if possible.

Thanks David