On Wed, Jul 29, 2009 at 08:44:28AM +0800, Herbert Xu wrote:

On Tue, Jul 28, 2009 at 12:59:19PM -0700, David Miller wrote:

quoted

The premise is that there'd be only one.  The qdisc lock.

If the traffic is distributed, flow wise, the driver XMIT
lock would spread due to multiqueue.

Suppose that we have a single large flow going through that has
filled up the hardware queue and is now backlogged in the qdisc
with qdisc_run on CPU A.   Now some other flow comes along and
sends a packet on CPU B.

So now CPU A and B will both be processing packets for the first
flow causing loads of lock contention.

But worse yet, we have introduced packet reordering.  So are you
convinced now :)

How about this: instead of the _RUNNING flag we take tx lock while
holding qdisc lock and release qdisc lock just after (before xmit).
This should prevent reordering, and probably could improve cache use:
CPU B which takes qdisc lock only for enqueuing now, would use it for
dequeuing too, plus if accidentally the next xmit goes to a different
tx queue, it could start before CPU A finishes. Otherwise it would
simply wait for CPU A (without tx lock contention). Of course it
needs testing... 

Cheers,
Jarek P.