Work with nvme native multipath, if a path related error occurs when
queue_rq call HBA drive to send request, queue_rq will return
BLK_STS_IOERR to blk-mq. The request is completed with BLK_STS_IOERR
instead of fail over to retry.
queue_rq need call nvme_complete_rq to complete the request with
NVME_SC_HOST_PATH_ERROR, the request will fail over to retry if needed.
Signed-off-by: Chao Leng <redacted>
---
drivers/nvme/host/rdma.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
Work with nvme native multipath, if a path related error occurs when
queue_rq call HBA drive to send request, queue_rq will return
BLK_STS_IOERR to blk-mq. The request is completed with BLK_STS_IOERR
instead of fail over to retry.
queue_rq need call nvme_complete_rq to complete the request with
NVME_SC_HOST_PATH_ERROR, the request will fail over to retry if needed.
Signed-off-by: Chao Leng <redacted>
---
drivers/nvme/host/fc.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
nvme drivers need to set the state of request to MQ_RQ_COMPLETE when
directly complete request in queue_rq.
So add blk_mq_set_request_complete.
Signed-off-by: Chao Leng <redacted>
---
include/linux/blk-mq.h | 5 +++++
1 file changed, 5 insertions(+)
Work with nvme native multipath, if a path related error occurs when
queue_rq call HBA drive to send request, queue_rq need complete the
request with NVME_SC_HOST_PATH_ERROR, the request will fail over to
retry if needed.
So introduce nvme_complete_failed_req for queue_rq and
nvmf_fail_nonready_command.
Signed-off-by: Chao Leng <redacted>
---
drivers/nvme/host/core.c | 8 ++++++++
drivers/nvme/host/nvme.h | 1 +
2 files changed, 9 insertions(+)
When reconnect, the request may be completed with NVME_SC_HOST_PATH_ERROR
in nvmf_fail_nonready_command. The state of request will be changed to
MQ_RQ_IN_FLIGHT before call nvme_complete_rq. If free the request
asynchronously such as in nvme_submit_user_cmd, in extreme scenario
the request may be completed again in tear down process.
nvmf_fail_nonready_command do not need calling blk_mq_start_request
before complete the request. nvmf_fail_nonready_command should set
the state of request to MQ_RQ_COMPLETE before complete the request.
Signed-off-by: Chao Leng <redacted>
---
drivers/nvme/host/fabrics.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Meant to do that only for -EIO. We should absolutely not do any of this
for stuff like EINVAL, EOPNOTSUPP, EPERM or any strange error that may
return due to a bug or anything like that.
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Meant to do that only for -EIO. We should absolutely not do any of this
for stuff like EINVAL, EOPNOTSUPP, EPERM or any strange error that may
return due to a bug or anything like that.
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Meant to do that only for -EIO. We should absolutely not do any of this
for stuff like EINVAL, EOPNOTSUPP, EPERM or any strange error that may
return due to a bug or anything like that.
blk_mq_hw_ctx *hctx,
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
- if (unlikely(err))
+ if (unlikely(err)) {
+ if (err == -EIO) {
+ nvme_complete_failed_rq(rq,
NVME_SC_HOST_PATH_ERROR);
I was thinking about:
--
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
if (unlikely(err)) {
if (err == -EIO) {
/*
* Fail the reuqest so upper layer can failover I/O
* if another path is available
*/
req->status = NVME_SC_HOST_PATH_ERROR;
nvme_rdma_complete_rq(rq);
return BLK_STS_OK;
}
goto err_unmap;
}
--
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Meant to do that only for -EIO. We should absolutely not do any of this
for stuff like EINVAL, EOPNOTSUPP, EPERM or any strange error that may
return due to a bug or anything like that.
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
- if (unlikely(err))
+ if (unlikely(err)) {
+ if (err == -EIO) {
+ nvme_complete_failed_rq(rq, NVME_SC_HOST_PATH_ERROR);
I was thinking about:
--
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
if (unlikely(err)) {
if (err == -EIO) {
/*
* Fail the reuqest so upper layer can failover I/O
* if another path is available
*/
req->status = NVME_SC_HOST_PATH_ERROR;
nvme_rdma_complete_rq(rq);
return BLK_STS_OK;Need to do clean. so can not directly return.
}
goto err_unmap;
}
--
.
Why not just call set the status and call nvme_rdma_complete_rq and
return here?
If the err is ENOMEM or EAGAIN, I am not sure the err must be a
path-related error for all HBA drivers. So reused the error check code.
I think it would be more reasonable to assume any errors returned by HBA
driver as path-related errors.
If you think so, I will modify it in next patch version.
Meant to do that only for -EIO. We should absolutely not do any of this
for stuff like EINVAL, EOPNOTSUPP, EPERM or any strange error that may
return due to a bug or anything like that.
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
- if (unlikely(err))
+ if (unlikely(err)) {
+ if (err == -EIO) {
+ nvme_complete_failed_rq(rq, NVME_SC_HOST_PATH_ERROR);
I was thinking about:
--
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
if (unlikely(err)) {
if (err == -EIO) {
/*
* Fail the reuqest so upper layer can failover I/O
* if another path is available
*/
req->status = NVME_SC_HOST_PATH_ERROR;
nvme_rdma_complete_rq(rq);
return BLK_STS_OK;
blk_mq_hw_ctx *hctx,
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
- if (unlikely(err))
+ if (unlikely(err)) {
+ if (err == -EIO) {
+ nvme_complete_failed_rq(rq,
NVME_SC_HOST_PATH_ERROR);
I was thinking about:
--
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
if (unlikely(err)) {
if (err == -EIO) {
/*
* Fail the reuqest so upper layer can
failover I/O
* if another path is available
*/
req->status = NVME_SC_HOST_PATH_ERROR;
nvme_rdma_complete_rq(rq);
return BLK_STS_OK;Need to do clean. so can
not directly return.
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
- if (unlikely(err))
+ if (unlikely(err)) {
+ if (err == -EIO) {
+ nvme_complete_failed_rq(rq, NVME_SC_HOST_PATH_ERROR);
I was thinking about:
--
err = nvme_rdma_post_send(queue, sqe, req->sge, req->num_sge,
req->mr ? &req->reg_wr.wr : NULL);
if (unlikely(err)) {
if (err == -EIO) {
/*
* Fail the reuqest so upper layer can failover I/O
* if another path is available
*/
req->status = NVME_SC_HOST_PATH_ERROR;
nvme_rdma_complete_rq(rq);
return BLK_STS_OK;Need to do clean. so can not directly return.
The completion path cleans up though
ok, i see.
But we need to use the new helper nvme_complete_failed_rq to avoid double request completion.
Or we can do like this:
req->status = NVME_SC_HOST_PATH_ERROR;
blk_mq_set_request_complete(req);
nvme_rdma_complete_rq(rq);