Re: [PATCH net-next 1/2] net: add a napi variant for RT-well-behaved drivers

Hi,

On 15/05/21 22:53, Thomas Gleixner wrote:

On Sat, May 15 2021 at 13:31, Jakub Kicinski wrote:

quoted

On Sat, 15 May 2021 13:07:40 +0200 Sebastian Andrzej Siewior wrote:

quoted

Now assume another interrupt comes in which wakes a force-threaded
handler (while ksoftirqd is preempted). Before the forced-threaded
handler is invoked, BH is disabled via local_bh_disable(). Since
ksoftirqd is preempted with BH disabled, disabling BH forces the
ksoftirqd thread to the priority of the interrupt thread (SCHED_FIFO,
prio 50 by default) due to the priority inheritance protocol. The
threaded handler will run once ksoftirqd is done which has now been
accelerated.

Thanks for the explanation. I'm not married to the patch, if you prefer
we can keep the status quo.

I'd think, however, that always deferring to ksoftirqd is conceptually
easier to comprehend. For power users who need networking there is
prefer-busy-poll (which allows application to ask the kernel to service
queues when it wants to, with some minimal poll frequency guarantees)
and threaded NAPI - which some RT users already started to adapt.

Your call.

quoted

Part of the problem from RT perspective is the heavy use of softirq and
the BH disabled regions which act as a BKL. I *think* having the network
driver running in a thread would be better (in terms of prioritisation).
I know, davem explained the benefits of NAPI/softirq when it comes to
routing/forwarding (incl. NET_RX/TX priority) and part where NAPI kicks
in during a heavy load (say a packet flood) and system is still
responsible.

Right, although with modern multi-core systems where only a subset 
of cores process network Rx things look different.

Bah, I completely forgot about that aspect. Thanks Sebastian for
bringing it up. I was too focussed on the other questions and there is
obviously the onset of alzheimer.

Anyway it's a touch choice to make. There are too many options to chose
from nowadays. 10 years ago running the softirq at the back of the
threaded irq handler which just scheduled NAPI was definitely a win, but
with threaded NAPI, zero copy and other things it's not that important
anymore IMO. But I might be missing something obviously.

I've cc'ed a few RT folks @RHT who might give some insight.

So, I asked around, but got mixed type of answers. My feeling is that an
opt-in approach, if feasible, might be useful to accomodate setups that
might indeed benefit from priority inheritance kicking in. In most cases
I'm aware of net handling is done by housekeeping cpus and isolated cpus
(running RT workload) mostly don't enter the kernel, but there are
exceptions to that.

I'll keep trying to ping more people. :)

Thanks,
Juri