On 2022-09-28 18:15:46 [+0200], Jason A. Donenfeld wrote:

Hi Sebastian,

Hi Jason,

On Wed, Sep 28, 2022 at 02:06:45PM +0200, Sebastian Andrzej Siewior wrote:

quoted

On 2022-09-27 12:42:33 [+0200], Jason A. Donenfeld wrote:
…

quoted

This is an ordinary pattern done all over the kernel. However, Sherry
noticed a 10% performance regression in qperf TCP over a 40gbps
InfiniBand card. Quoting her message:

quoted

MT27500 Family [ConnectX-3] cards:
Infiniband device 'mlx4_0' port 1 status:

…

While looking at the mlx4 driver, it looks like they don't use any NAPI
handling in their interrupt handler which _might_ be the case that they
handle more than 1k interrupts a second. I'm still curious to get that
ACKed from Sherry's side.

Are you sure about that? So far as I can tell drivers/net/ethernet/
mellanox/mlx4 has plenty of napi_schedule/napi_enable and such. Or are
you looking at the infiniband driver instead? I don't really know how
these interact.

I've been looking at mlx4_msi_x_interrupt() and it appears that it
iterates over a ring buffer. I guess that mlx4_cq_completion() will
invoke mlx4_en_rx_irq() which schedules NAPI.

But yea, if we've got a driver not using NAPI at 40gbps that's obviously
going to be a problem.

So I'm wondering if we get 1 worker a second which kills the performance
or if we get more than 1k interrupts in less than second resulting in
more wakeups within a second..

quoted

Jason, from random's point of view: deferring until 1k interrupts + 1sec
delay is not desired due to low entropy, right?

Definitely || is preferable to &&.

quoted

quoted

Rather than incur the scheduling latency from queue_work_on, we can
instead switch to running on the next timer tick, on the same core. This
also batches things a bit more -- once per jiffy -- which is okay now
that mix_interrupt_randomness() can credit multiple bits at once.

Hmmm. Do you see higher contention on input_pool.lock? Just asking
because if more than once CPUs invokes this timer callback aligned, then
they block on the same lock.

I've been doing various experiments, sending mini patches to Oracle and
having them test this in their rig. So far, it looks like the cost of
the body of the worker itself doesn't matter much, but rather the cost
of the enqueueing function is key. Still investigating though.

It's a bit frustrating, as all I have to work with are results from the
tests, and no perf analysis. It'd be great if an engineer at Oracle was
capable of tackling this interactively, but at the moment it's just me
sending them patches. So we'll see. Getting closer though, albeit very
slowly.

Oh boy. Okay.

Jason

Sebastian