On Mon, Aug 21, 2023 at 5:17 PM Jakub Kicinski [off-list ref] wrote:

On Sat, 19 Aug 2023 10:18:57 -0400 Willem de Bruijn wrote:

quoted

Right. Many devices only allow bringing all queues down at the same time.

Once a descriptor is posted and the ring head is written, there is no
way to retract that. Since waiting for the device to catch up is not
acceptable, the only option is to bring down the queue, right? Which
will imply bringing down the entire device on many devices. Not ideal,
but acceptable short term, imho.

That may be an incentive for vendors to support per-queue
start/stop/alloc/free. Maybe the ones that support RDMA already do?

Are you talking about HW devices, or virt? I thought most HW made
in the last 10 years should be able to take down individual queues :o

That's great. This is currently mostly encapsulated device-wide behind
ndo_close, with code looping over all rx rings, say.

Taking a look at one driver, bnxt, it indeed has a per-ring
communication exchange with the device, in hwrm_ring_free_send_msg
("/* Flush rings and disable interrupts */"), which is called before
the other normal steps: napi disable, dma unmap, posted mem free,
irq_release, napi delete and ring mem free.

This is what you meant? The issue I was unsure of was quiescing the
device immediately, i.e., that hwrm_ring_free_send_msg.

I guess this means that this could all be structured on a per-queue
basis rather than from ndo_close. Would be a significant change to
many drivers, I'd imagine.