On 2020-12-07 16:52:57 [-0700], Jens Axboe wrote:

On 12/4/20 12:13 PM, Sebastian Andrzej Siewior wrote:

quoted

Controllers with multiple queues have their IRQ-handelers pinned to a
CPU. The core shouldn't need to complete the request on a remote CPU.

Remove this case and always raise the softirq to complete the request.

I don't like this one at all, it'll add a softirq jump for the fast path
for eg nvme devices. Did you run any performance testing with this? I
can give it a spin, will do so anyway, but was curious if anything but
"this still works" testing was done.

I understood that the nvme devices have a queue per CPU and don't need
to complete on a remote-CPU in general.
Also, they don't jump to softirq but invoke softirq on the return from
IRQ. So if softirq is already busy because of network then softirq of
the block layer is delayed. Otherwise there should be no significant
delay if the CPU is idle (so the IRQ is handled and softirq right after
it).

Sagi mentioned nvme-tcp as a user of this remote completion and Daniel
has been kind to run some nvme-tcp tests.

-- 
Jens Axboe

Sebastian