Re: [PATCH 2/2] net/9p: increase default msize to 128k

Eric Van Hensbergen wrote on Sun, Sep 05, 2021 at 06:44:13PM -0500:

there will likely be a tradeoff with tcp in terms of latency to first
message so while
absolute bw may be higher processing time may suffer.  8k was default msize
to more closely match it to jumbo frames on an ethernet.  of course all
that intuition is close to 30 years out of date….

It's not because the max size is 128k (or 1MB) that this much is sent
over the wire everytime -- if a message used to fit in 8KB, then its
on-the-wire size won't change and speed/latency won't be affected for
these.

For messages that do require more than 8KB (read/write/readdir) then you
can fit more data per message, so for a given userspace request (feed me
xyz amount of data) you'll have less client-server round-trips, and the
final user-reflected latency will be better as well -- that's why
e.g. NFS has been setting a max size of 1MB by default for a while now,
and they allow even more (32MB iirc? not sure)

I've only had done these tests years ago and no longer have access to
the note that was written back then, but TCP also definitely benefits
from > 64k msize as long as there's enough memory available.


The downside (because it's not free) is there though, you need more
memory for 9p with big buffers even if we didn't need so much in the
first place.
The code using a slab now means that the memory is not locked per mount
as it used to, but that also means allocations can fail if there is a
big pressure after not having been released. OTOH as long as it's
consistently used the buffers will be recycled so it's not necessarily
too bad performance-wise in hot periods.

Ideally we'd need to rework transports to allow scatter-gather (iovec or
similar API), and work with smaller allocations batched together on
send, but I don't have time for something like that... If we do that we
can probably get the best of both worlds -- and could consider >1MB, but
that's unrealistic as of now, regardless of the transport.

-- 
Dominique