Re: [PATCH net-next] Revert "RDS: IB: split the mr registration and invalidation path"

On 11/6/18 4:52 PM, Gerd Rausch wrote:

This reverts commit 56012459310a1dbcc55c2dbf5500a9f7571402cb.

Subject: rds_ib_post_reg_frmr spins and spins and spins

RDS kept spinning inside function "rds_ib_post_reg_frmr", waiting for
"i_fastreg_wrs" to become incremented:
         while (atomic_dec_return(&ibmr->ic->i_fastreg_wrs) <= 0) {
                 atomic_inc(&ibmr->ic->i_fastreg_wrs);
                 cpu_relax();
         }

Looking at the original commit:

commit 56012459310a ("RDS: IB: split the mr registration and invalidation path")

In there, the "rds_ib_mr_cqe_handler" was changed in the following way:

quoted

@@ -289,9 +289,10 @@ void rds_ib_mr_cqe_handler(struct rds_ib_connection *ic,

struct ib_wc *wc)
         if (frmr->fr_inv) {
                 frmr->fr_state = FRMR_IS_FREE;
                 frmr->fr_inv = false;
+               atomic_inc(&ic->i_fastreg_wrs);
+       } else {
+               atomic_inc(&ic->i_fastunreg_wrs);
         }
-
-       atomic_inc(&ic->i_fastreg_wrs);
  }

It looks like it's got it exactly backwards:

Function "rds_ib_post_reg_frmr" keeps track of the outstanding requests
via "i_fastreg_wrs".

Function "rds_ib_post_inv" keeps track of the outstanding requests
via "i_fastunreg_wrs" (post original commit). It also sets:
         frmr->fr_inv = true;

However the completion handler "rds_ib_mr_cqe_handler" adjusts "i_fastreg_wrs"
when "fr_inv" had been true, and adjusts "i_fastunreg_wrs" otherwise.

The original commit was done in the name of performance:

quoted

to remove the performance bottleneck

No performance benefit could be observed with a fixed-up version
of the original commit measured between two Oracle X7 servers,
both equipped with Mellanox Connect-X5 HCAs.

The prudent course of action is to revert this commit.

Signed-off-by: Gerd Rausch <redacted>
---

As mentioned in offline discussion, when we added this change, the test
server was equipped with Connect-X3 NIC running with FRWR mode and it
did show step improvements. Unfortunately I don't have that data
stored, so it is just from memory.

But looking at Connext-X5 numbers(thanks), am fine to back this out.

Acked-by: Santosh Shilimkar <redacted>