Re: [net-next 0/3] Per epoll context busy poll support

On Fri, Feb 02, 2024 at 05:15:39PM -0800, Jakub Kicinski wrote:

On Fri, 2 Feb 2024 12:23:44 -0800 Joe Damato wrote:

quoted

quoted

Did you see SO_PREFER_BUSY_POLL by any chance? (In combination with
gro_flush_timeout IIRC). We added it a while back with Bjorn, it seems
like a great idea to me at the time but I'm unclear if anyone uses it 
in production..  

I have seen it while reading the code, yes. I think maybe I missed
something about its interaction with gro_flush_timeout. In my use case,
the machine has no traffic until after the app is started.

In this case, I haven't needed to worry about regular NAPI monopolizing the
CPU and preventing busy poll from working.

Maybe I am missing something more nuanced, though? I'll have another look
at the code, just incase.

We reused the gro_flush_timeout as an existing "user doesn't care if
packets get delayed by this much in worst case" value. If you set
SO_PREFER_BUSY_POLL the next time you busy pool the NAPI will be marked 
as "already scheduled" and a timer is set (to gro_flush_timeout).
If NIC IRQ fires before gro_flush_timeout it gets ignored, because NAPI
is already marked as scheduled.
If you busy poll again the timer gets postponed for another
gro_flush_timeout nsec.
If timer fires we go back to normal NAPI processing.

Ah, I see. From my reading of the code in busy_poll_stop (which could be
wrong), defer_hard_irqs_count must also be non-zero to postpone the timer.

Is that right?

If so, I think the tricky thing with this is that these settings are
system-wide, so they'd affect non-busy poll apps, too.

I think in the ideal case being able to set these on a per-NAPI basis would
be very helpful. Maybe something for me to try working on next.

The idea is that you set gro_flush_timeout to some high value, like 
10 msec, and expect your app to poll more often than every 10 msec. 

Yea, that makes sense.

Then the normal NAPI processing will never kick in, and there will 
be only 1 NIC IRQ after which the HW IRQ remains masked.
With high coalescing timer you technically still get an IRQ every
so often and interrupt the app. Worst case (UDP flood) you may even
get into an overload where the app gets starved out completely..

Yup, this is true. I had been using a modified version of a patch from a
research paper to avoid enabling NIC IRQs [1][2], but I think making
defer_hard_irqs_count and gro_flush_timeout per NAPI parameters would make
more sense.

[1]: https://gitlab.uwaterloo.ca/p5cai/netstack-exp/-/raw/master/kernel-polling-5.15.79-base.patch?ref_type=heads
[2]: https://dl.acm.org/doi/pdf/10.1145/3626780