On 2021/1/21 16:58, Hannes Reinecke wrote:

On 1/21/21 8:03 AM, Chao Leng wrote:

quoted

When reconnect, the request may be completed with NVME_SC_HOST_PATH_ERROR
in nvmf_fail_nonready_command. The state of request will be changed to
MQ_RQ_IN_FLIGHT before call nvme_complete_rq. If free the request
asynchronously such as in nvme_submit_user_cmd, in extreme scenario
the request may be completed again in tear down process.
nvmf_fail_nonready_command do not need calling blk_mq_start_request
before complete the request. nvmf_fail_nonready_command should set
the state of request to MQ_RQ_COMPLETE before complete the request.

So what you are saying is that there is a race condition between
blk_mq_start_request()
and
nvme_complete_request()

Yes. The race is:
process1:error recovery->tear down->quiesce queue(wait dispatch done)
process2:dispatch->queue_rq->nvmf_fail_nonready_command->
     nvme_complete_rq(if the request is freed asynchronously, wake
	nvme_submit_user_cmd( for example) but have no chance to run).
process1:continue ->cancle suspend request, check the state is not
     MQ_RQ_IDLE and MQ_RQ_COMPLETE, complete(free) the request.
process3: nvme_submit_user_cmd now has chance to run, and the free the
     request again.
Test Injection Method: inject a msleep before call blk_mq_free_request
in nvme_submit_user_cmd.

quoted

Signed-off-by: Chao Leng <redacted>
---
  drivers/nvme/host/fabrics.c | 4 +---
  1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 72ac00173500..874e4320e214 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c

@@ -553,9 +553,7 @@ blk_status_t nvmf_fail_nonready_command(struct nvme_ctrl *ctrl,

          !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH))
          return BLK_STS_RESOURCE;
-    nvme_req(rq)->status = NVME_SC_HOST_PATH_ERROR;
-    blk_mq_start_request(rq);
-    nvme_complete_rq(rq);
+    nvme_complete_failed_req(rq);
      return BLK_STS_OK;
  }
  EXPORT_SYMBOL_GPL(nvmf_fail_nonready_command);
I'd rather have 'nvme_complete_failed_req()' accept the status as 

argument, like

nvme_complete_failed_request(rq, NVME_SC_HOST_PATH_ERROR)

that way it's obvious what is happening, and the status isn't hidden in the function.

Ok, good idea. Thank you for your suggestion.

Cheers,

Hannes