On Fri, 06 Dec 2024, Jeff Layton wrote:

On Fri, 2024-12-06 at 11:43 +1100, NeilBrown wrote:

quoted

If a client ever uses the highest available slot for a given session,
attempt to allocate more slots so there is room for the client to use
them if wanted.  GFP_NOWAIT is used so if there is not plenty of
free memory, failure is expected - which is what we want.  It also
allows the allocation while holding a spinlock.

Each time we increase the number of slots by 20% (rounded up).  This
allows fairly quick growth while avoiding excessive over-shoot.

We would expect to stablise with around 10% more slots available than
the client actually uses.

Signed-off-by: NeilBrown <redacted>
---
 fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 67dfc699e411..ec4468ebbd40 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c

@@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	slot = xa_load(&session->se_slots, seq->slotid);
 	dprintk("%s: slotid %d\n", __func__, seq->slotid);
 
-	/* We do not negotiate the number of slots yet, so set the
-	 * maxslots to the session maxreqs which is used to encode
-	 * sr_highest_slotid and the sr_target_slot id to maxslots */
-	seq->maxslots = session->se_fchannel.maxreqs;
-
 	trace_nfsd_slot_seqid_sequence(clp, seq, slot);
 	status = check_slot_seqid(seq->seqid, slot->sl_seqid,
 					slot->sl_flags & NFSD4_SLOT_INUSE);

@@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	cstate->session = session;
 	cstate->clp = clp;
 
+	/*
+	 * If the client ever uses the highest available slot,
+	 * gently try to allocate another 20%.  This allows
+	 * fairly quick growth without grossly over-shooting what
+	 * the client might use.
+	 */

20% seems like a reasonable place to start, but I do wonder if this
might need to be tunable under some workloads. Oh well, we can cross
that bridge if/when someone complains.

I think that if we need a tunable, then it is a failure of design.
If?when someone complains we may well need to redesign.  I hope we could
avoid a tunable in that design!

quoted

+	if (seq->slotid == session->se_fchannel.maxreqs - 1 &&
+	    session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) {
+		int s = session->se_fchannel.maxreqs;
+		int cnt = DIV_ROUND_UP(s, 5);
+
+		do {
+			/*
+			 * GFP_NOWAIT is a low-priority non-blocking
+			 * allocation which can be used under
+			 * client_lock and only succeeds if there is
+			 * plenty of memory.
+			 * Use GFP_ATOMIC which is higher priority for
+			 * xa_store() so we are less likely to waste the
+			 * effort of the first allocation.
+			 */

I don't know here. Why not just use GFP_NOWAIT for the xa_store too? If
we're so memory constrained that that fails, we're probably better off
releasing the slot.

Maybe.  I'm open simple using GFP_NOWAIT both places.
Most often xa_store won't need to allocate anything - it adds slots to
the array in batches (at least I assume it does - anything else would be
inefficient).  So it mostly won't matter.
So if seems at all inelegant - let's drop it.

Thanks,
NeilBrown

quoted

+			slot = kzalloc(slot_bytes(&session->se_fchannel),
+				       GFP_NOWAIT);
+			if (slot &&
+			    !xa_is_err(xa_store(&session->se_slots, s, slot,
+						GFP_ATOMIC | __GFP_NOWARN))) {
+				s += 1;
+				session->se_fchannel.maxreqs = s;
+			} else {
+				kfree(slot);
+			}
+		} while (slot && --cnt > 0);
+	}
+	seq->maxslots = session->se_fchannel.maxreqs;
+
 out:
 	switch (clp->cl_cb_state) {
 	case NFSD4_CB_DOWN:

-- 
Jeff Layton [off-list ref]