Re: [PATCH] qed: avoid spin loops in _qed_mcp_cmd_and_union()

Here you might sleep/schedule, while CAN_SLEEP was not set ?

I also do not know this driver, just trying to fix an observed latency issue.
As far as I can tell, the CAN_SLEEP flag is set/unset depending on
which function called qed_mcp_cmd_and_union();
it does not indicate whether the function is running in atomic context.
For example, qed_mcp_cmd() calls it without CAN_SLEEP,
yet qed_mcp_drain() calls msleep() immediately after qed_mcp_cmd().

We were concerned that this function might be called in atomic context,
so we added a WARN_ON_ONCE(in_atomic()). We never saw the warning fire
during two weeks of testing, so we believe sleeping is possible here.

I would suggest using usleep_range() instead, because cond_resched()
can be a NOP under some circumstances.
Then perhaps not count against max_retries, but based on total elapsed time ?

I agree these would both be improvements to the current code.
I was trying to provide a minimal change that would allow these loops
to yield the CPU,
but will happily do this refactoring if the driver authors think it
would be beneficial.

On Wed, Oct 27, 2021 at 3:25 PM Eric Dumazet [off-list ref] wrote:

On 10/27/21 2:45 PM, Caleb Sander wrote:

quoted

By default, qed_mcp_cmd_and_union() sets max_retries to 500K and
usecs to 10, so these loops can together delay up to 5s.
We observed thread scheduling delays of over 700ms in production,
with stacktraces pointing to this code as the culprit.

Add calls to cond_resched() in both loops to yield the CPU if necessary.

Signed-off-by: Caleb Sander <redacted>
Reviewed-by: Joern Engel <redacted>
---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index 24cd41567..d6944f020 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c

@@ -485,10 +485,12 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,

              spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);

-             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP))
+             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP)) {

I do not know this driver, but apparently, there is this CAN_SLEEP test
hinting about being able to sleep.

quoted

                      msleep(msecs);
-             else
+             } else {
+                     cond_resched();

Here you might sleep/schedule, while CAN_SLEEP was not set ?

quoted

                      udelay(usecs);

I would suggest using usleep_range() instead, because cond_resched()
can be a NOP under some circumstances.

quoted

+             }
      } while (++cnt < max_retries);

Then perhaps not count against max_retries, but based on total elapsed time ?

quoted

      if (cnt >= max_retries) {

@@ -517,10 +519,12 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
               * The spinlock stays locked until the list element is removed.
               */

-             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP))
+             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP)) {
                      msleep(msecs);
-             else
+             } else {
+                     cond_resched();
                      udelay(usecs);
+             }

              spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);