Thread (19 messages) 19 messages, 5 authors, 2025-04-01

Re: nvme-tcp: fix a possible UAF when failing to send request

From: Max Gurtovoy <mgurtovoy@nvidia.com>
Date: 2025-02-10 11:34:47
Also in: linux-nvme, lkml

hi

On 10/02/2025 13:16, zhang.guanghui@cestc.cn wrote:
Hi,



      Thank you for your reply.


      In nvme-rdma we use nvme_host_path_error(rq) ,   the prerequisites are -EIO,  ignore this condition?
  in nvme-tcp this judgment condition is different,  use the function nvme_host_path_error ,
it should be possible to go to nvme_complete_rq ->  nvme_retry_req -- request->mq_hctx have been freed, is NULL.
did you try the proposal or are you assuming it will be NULL ?




zhang.guanghui@cestc.cn



  



From: Max Gurtovoy



Date: 2025-02-10 18:24



To: Maurizio Lombardi; zhang.guanghui@cestc.cn; sagi; kbusch; sashal; chunguang.xu



CC: linux-kernel; linux-nvme; linux-block



Subject: Re: nvme-tcp: fix a possible UAF when failing to send request



  



On 10/02/2025 12:01, Maurizio Lombardi wrote:


quoted
On Mon Feb 10, 2025 at 8:41 AM CET, zhang.guanghui@cestc.cn wrote:
quoted
quoted
Hello




quoted
I guess you have to fix your mail client.


quoted
quoted
       When using the nvme-tcp driver in a storage cluster, the driver may trigger a null pointer causing the host to crash several times.
quoted
quoted
By analyzing the vmcore, we know the direct cause is that  the request->mq_hctx was used after free.




quoted
quoted
CPU1                                                                   CPU2


quoted
quoted
nvme_tcp_poll                                                          nvme_tcp_try_send  --failed to send reqrest 13
quoted
This simply looks like a race condition between nvme_tcp_poll() and nvme_tcp_try_send()
quoted
Personally, I would try to fix it inside the nvme-tcp driver without
quoted
touching the core functions.


quoted
Maybe nvme_tcp_poll should just ensure that io_work completes before
quoted
calling nvme_tcp_try_recv(), the POLLING flag should then prevent io_work
quoted
from getting rescheduled by the nvme_tcp_data_ready() callback.




quoted
Maurizio

  



It seems to me that the HOST_PATH_ERROR handling can be improved in



nvme-tcp.



  



In nvme-rdma we use nvme_host_path_error(rq) and nvme_cleanup_cmd(rq) in



case we fail to submit a command..



  



can you try to replacing nvme_tcp_end_request(blk_mq_rq_from_pdu(req),



NVME_SC_HOST_PATH_ERROR); call with the similar logic we use in



nvme-rdma for host path error handling ?



  



  



  



  
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help