On 7/28/25 21:21, Stanislav Fomichev wrote:

On 07/28, Pavel Begunkov wrote:

quoted

On 7/28/25 18:13, Stanislav Fomichev wrote:

...>>> Supporting big buffers is the right direction, but I have the same

quoted

quoted

feedback:

Let me actually check the feedback for the queue config RFC...

it would be nice to fit a cohesive story for the devmem as well.

Only the last patch is zcrx specific, the rest is agnostic,
devmem can absolutely reuse that. I don't think there are any
issues wiring up devmem?

Right, but the patch number 2 exposes per-queue rx-buf-len which
I'm not sure is the right fit for devmem, see below. If all you

I guess you're talking about uapi setting it, because as an
internal per queue parameter IMHO it does make sense for devmem.

care is exposing it via io_uring, maybe don't expose it from netlink for

Sure, I can remove the set operation.

now? Although I'm not sure I understand why you're also passing
this per-queue value via io_uring. Can you not inherit it from the
queue config?

It's not a great option. It complicates user space with netlink.
And there are convenience configuration features in the future
that requires io_uring to parse memory first. E.g. instead of
user specifying a particular size, it can say "choose the largest
length under 32K that the backing memory allows".

quoted

quoted

We should also aim for another use-case where we allocate page pool
chunks from the huge page(s),

Separate huge page pool is a bit beyond the scope of this series.

this should push the perf even more.

And not sure about "even more" is from, you can already
register a huge page with zcrx, and this will allow to chunk
them to 32K or so for hardware. Is it in terms of applicability
or you have some perf optimisation ideas?

What I'm looking for is a generic system-wide solution where we can
set up the host to use huge pages to back all (even non-zc) networking queues.
Not necessary needed, but might be an option to try.

Probably like what Jakub was once suggesting with the initial memory
provider patch, got it.

quoted

quoted

We need some way to express these things from the UAPI point of view.

Can you elaborate?

quoted

Flipping the rx-buf-len value seems too fragile - there needs to be
something to request 32K chunks only for devmem case, not for the (default)
CPU memory. And the queues should go back to default 4K pages when the dmabuf
is detached from the queue.

That's what the per-queue config is solving. It's not default, zcrx
configures it only for the specific queue it allocated, and the value
is cleared on restart in netdev_rx_queue_restart(), if not even too
aggressively. Maybe I should just stash it into mp_params to make
sure it's not cleared if a provider is still attached on a spurious
restart.

If we assume that at some point niov can be backed up by chunks larger
than PAGE_SIZE, the assumed workflow for devemem is:
1. change rx-buf-len to 32K
   - this is needed only for devmem, but not for CPU RAM, but we'll have
     to refill the queues from the main memory anyway

Urgh, that's another reason why I prefer to just pass it through
zcrx and not netlink. So maybe you can just pass the len to devmem
on creation, and internally it sets up its queues with it.

   - there is also a question on whether we need to do anything about
     MAX_PAGE_ORDER/PAGE_ALLOC_COSTLY_ORDER - do we just let the driver
     allocations fail?
2. attach dmabuf to the queue to refill from dmabuf sgt, essentially wasting
    all the effort on (1)
3. on detach, something needs to also not forget to reset the rx-buf-len
   back to PAGE_SIZE

Sure

I was hoping that maybe we can bind rx-buf-len to dmabuf for devmem,
that should avoid all that useless refill from the main memory with
large chunks. But I'm not sure it's the right way to go either.

-- 
Pavel Begunkov