On Wed, Jul 29, 2009 at 10:38:09AM +0800, Herbert Xu wrote:

I did say it wasn't a good answer :)

Here's another crazy idea:

Forget about using multiple hardware TX queues with rate limiting
qdiscs.  Why? Because these qdiscs are fundamentally serialising
so once the traffic has gone through it there is nothing to be
gained from multiplexing them into the hardware only to be merged
again onto the wire right away.

So does this mean that we give up on CPU scalability on limiting
qdiscs? No, the bottleneck is in fact in the qdisc, not in the
NIC.  So we implement multiqueue in the qdisc instead.

This can be done with a lockless ring buffer.  Each CPU would
deposit its packets in a qdisc ringer buffer assigned to it.
The qdisc would then run on a single CPU harvesting all the
ring buffers in some fair manner.

Essentially we can treat rate limiting qdiscs as NICs.  Another
way of thinking about this is that we're essentially decoupling
a single system into two.  One that is using a default qdisc to
achieve maximum throughput.  This is then fed into a second system
that is dedicated to rate limiting.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt