On Fri, 18 Aug 2023 19:34:32 -0600 David Ahern wrote:

On 8/18/23 3:52 PM, Mina Almasry wrote:

quoted

The sticking points are:
1. From David: this proposal doesn't give an application the ability
to flush an rx queue, which means that we have to rely on a driver
reset that affects all queues to refill the rx queue buffers.  

Generically, the design needs to be able to flush (or invalidate) all
references to the dma-buf once the process no longer "owns" it.

Are we talking about the ability for the app to flush the queue
when it wants to (do no idea what)? Or auto-flush when app crashes?

quoted

2. From Jakub: the uAPI and implementation here needs to be in line
with his general direction & extensible to apply to existing use cases
`ethtool -L/-G`, etc.  

I think this is a bit more open ended given the openness of the netdev
netlink API. i.e., managing a H/W queue (create, delete, stop / flush,
associate a page_pool) could be done through this API.

quoted

AFAIU this is what I need to do in the next version:

1. The uAPI will be changed such that it will either re-configure an
existing queue to bind it to the dma-buf, or allocate a new queue
bound to the dma-buf (not sure which is better at the moment). Either  

1. API to manage a page-pool (create, delete, update).

I wasn't anticipating a "create page pool" API.

I was thinking of a scheme where user space sets page pool parameters,
but the driver still creates the pool.

But I guess it is doable. More work, tho. Are there ibverbs which
can do it? lol.

2. API to add and remove a dma-buf (or host memory buffer) with a
page-pool. Remove may take time to flush references pushed to hardware
so this would be asynchronous.

3. Create a queue or use an existing queue id and associate a page-pool
with it.

quoted

way, the configuration will take place immediately, and not rely on an
entire driver reset to actuate the change.  

yes

quoted

2. The uAPI will be changed such that if the netlink socket is closed,
or the process dies, the rx queue will be unbound from the dma-buf or
the rx queue will be freed entirely (again, not sure which is better  

I think those are separate actions. But, if the queue was created by and
referenced by a process, then closing an fd means it should be freed.

quoted

at the moment). The configuration will take place immediately without
relying on a driver reset.  

yes on the reset.

quoted

3. I will add 4 new net_device_ops that Jakub specified:
queue_mem_alloc/free(), and queue_start/stop().

4. The uAPI mentioned in #1 will use the new net_device_ops to
allocate or reconfigure a queue attached to the provided dma-buf.

I'd leave 2, 3, 4 alone for now. Focus on binding a page pool to 
an existing queue.

quoted

Does this sound roughly reasonable here?

AFAICT the only technical difficulty is that I'm not sure it's
feasible for a driver to start or stop 1 rx-queue without triggering a
full driver reset. The (2) drivers I looked at both do a full reset to
change any queue configuration. I'll investigate.