Re: [RFC v1 00/22] Large rx buffer support for zcrx

On 7/31/25 20:34, Mina Almasry wrote:

On Wed, Jul 30, 2025 at 8:50 AM Stanislav Fomichev [off-list ref] wrote:

quoted

On 07/30, Pavel Begunkov wrote:

quoted

On 7/29/25 17:33, Stanislav Fomichev wrote:

quoted

On 07/28, Pavel Begunkov wrote:

quoted

On 7/28/25 23:06, Stanislav Fomichev wrote:

quoted

On 07/28, Pavel Begunkov wrote:

quoted

On 7/28/25 21:21, Stanislav Fomichev wrote:

quoted

On 07/28, Pavel Begunkov wrote:

quoted

On 7/28/25 18:13, Stanislav Fomichev wrote:

...>>> Supporting big buffers is the right direction, but I have the same

quoted

quoted

quoted

feedback:

Let me actually check the feedback for the queue config RFC...

it would be nice to fit a cohesive story for the devmem as well.

Only the last patch is zcrx specific, the rest is agnostic,
devmem can absolutely reuse that. I don't think there are any
issues wiring up devmem?

Right, but the patch number 2 exposes per-queue rx-buf-len which
I'm not sure is the right fit for devmem, see below. If all you

I guess you're talking about uapi setting it, because as an
internal per queue parameter IMHO it does make sense for devmem.

quoted

care is exposing it via io_uring, maybe don't expose it from netlink for

Sure, I can remove the set operation.

quoted

now? Although I'm not sure I understand why you're also passing
this per-queue value via io_uring. Can you not inherit it from the
queue config?

It's not a great option. It complicates user space with netlink.
And there are convenience configuration features in the future
that requires io_uring to parse memory first. E.g. instead of
user specifying a particular size, it can say "choose the largest
length under 32K that the backing memory allows".

Don't you already need a bunch of netlink to setup rss and flow

Could be needed, but there are cases where configuration and
virtual queue selection is done outside the program. I'll need
to ask which option we currently use.

If the setup is done outside, you can also setup rx-buf-len outside, no?

You can't do it without assuming the memory layout, and that's
the application's role to allocate buffers. Not to mention that
often the app won't know about all specifics either and it'd be
resolved on zcrx registration.

I think, fundamentally, we need to distinguish:

1. chunk size of the memory pool (page pool order, niov size)
2. chunk size of the rx queue entries (this is what this series calls
    rx-buf-len), mostly influenced by MTU?

For devmem (and same for iou?), we want an option to derive (2) from (1):
page pools with larger chunks need to generate larger rx entries.

To be honest I'm not following. #1 and #2 seem the same to me.
rx-buf-len is just the size of each rx buffer posted to the NIC.

With pp_params.order = 0 (most common configuration today), rx-buf-len
== 4K. Regardless of MTU. With pp_params.order=1, I'm guessing 8K
then, again regardless of MTU.

There are drivers that fragment the buffer they get from a page
pool and give smaller chunks to the hw. It's surely a good idea to
be more explicit on what's what, but from the whole setup and uapi
perspective I'm not too concerned.

The parameter the user passes to zcrx must controls 1. As for 2.
I'd expect the driver to use the passed size directly or fail
validation, but even if that's not the case, zcrx / devmem would
just continue to work without any change in uapi, so we have
the freedom to patch up the nuances later on if anything sticks
out.

  > I think if the user has not configured rx-buf-len, the driver is

probably free to pick whatever it wants and that can be a derivative
of the MTU.

When the rx-buf-len is configured by the user, I assume the driver
puts aside all MTU-related heuristics (if it has them) and uses
whatever the userspace specified.

Note that the memory provider may reject the request. For example
iouring and pages providers can only do page-order allocations. Devmem
can in theory do any byte-aligned allocation, since gen_pool doesn't
have a restriction AFAIR.

-- 
Pavel Begunkov