Re: [PATCH 3/16] Spidernet RX Locking

On Sat, Dec 09, 2006 at 09:47:05AM +1100, Benjamin Herrenschmidt wrote:

A spinlock is expensive in the fast path, which is why Jeff says it's
invasive.

quoted

spider_net_decode_one_descr() is called from
spider_net_poll() (which is the netdev->poll callback)
and also from spider_net_handle_rxram_full(). 

The rxramfull routine is called from a tasklet that
is fired off after a "RX ram full" interrupt is receved.
This interrupt is generated when the hardware runs out
of space to store incoming packets. We are seeing this
interrupt fire when the CPU is heavily loaded, and a
lot of traffic is being fired at the device.

How often does that interrupt happen in that case ?

It is hard to reproduce; it is highly dependent on kernel version
and network config. It seems to occur when the system is somehow
loaded, and the tcp stack is unable to empty out the rx ring in a
timely manner. Jim is able o trigger this trivially for some kernels, 
but not others.

A better approach is to keep the fast path (ie. poll()) lockless, and in
handle_rxram_full(), the slow path, protect against poll using
netif_disable_poll(). Though that means using a work queue, not a
tasklet, since it needs to schedule.

Yes. Actually, I am thinking of treating this interrupt as if it were
just another RX interrupt. What the original drivers seemed to want to
do was to treat this as some sort of "high priority" rx interrupt, but
there doesn't seem to be any real way of doing this, so it seems simpler
just to rip out the tasklet and leave it at that.

or you can schedule rx work from the rxramfull interrupt after setting a
"something bad happened" flag. Then, poll can check this flag and do the
right thing.

Yes, exactly.

--linas