Re: use-after-free access in bt_iter()

On 2021-02-05 21:51, pragalla@codeaurora.org wrote:

On 2021-02-05 21:37, John Garry wrote:

quoted

- bouncing jianchao.w.wang@oracle.com

quoted

quoted

quoted

Some time ago you replied the following to an email from me with a
suggestion for a fix: "Please let me consider it a bit more." Are you
still working on a fix?

Unfortunately I have not had a chance, sorry. But I can look again.

So I have only seen KASAN use-after-free's myself, but never an actual
oops. IIRC, someone did report an oops.

Hi John,

quoted

@Pradeep, do you have a reliable re-creator? I noticed the timeout
handler stackframe in your mail, so I guess not. However, as an
experiment, could you test:
https://lore.kernel.org/linux-block/1608203273-170555-2-git-send-email-john.garry@huawei.com/

Yes, i don't have a reliable re-creator. The oops was noticed as a part of stability testing and
was not an intentional try. This was noticed couple of times.
Please share the steps (if any) to easy hit or to exercise this path more frequently.
Meanwhile, i will go with the usual stability procedure. i will update the results here later.

Hi John,

we ran the stability with the above patch
(https://lore.kernel.org/linux-block/1608203273-170555-2-git-send-email-john.garry@huawei.com/)
with switching the io-schedulers in b/w for ~88hrs on 2 devices, we didn't notice any crash/issue.

quoted

Do you have a full kernel log for your crash?

Yes. Attaching the full kernel dmesg log.

quoted

So there are different flavors of this issue, and you reported a crash
from blk_mq_queue_tag_busy_iter().

If you check:
https://lore.kernel.org/linux-block/76190c94-c5c1-9553-5509-9969fc323544@huawei.com/

You can see how I artificially trigger an issue in blk_mq_queue_tag_busy_iter().

Sure, i will go through the steps on the recreation part. Thanks.

quoted

quoted

quoted

This should fix the common issue. But no final solution to issues
discussed from patch 2/2, which is more exotic.

BTW, is this the same Pradeep who reported:
https://lore.kernel.org/linux-block/1606402925-24420-1-git-send-email-ppvk@codeaurora.org/

Thanks,
John

Thanks and Regards,
Pradeep

On 19/02/2021 06:22, pragalla@codeaurora.org wrote:

quoted

quoted

quoted

Hi John,

we ran the stability with the above patch
(https://lore.kernel.org/linux-block/1608203273-170555-2-git-send-email-john.garry@huawei.com/)

with switching the io-schedulers in b/w for ~88hrs on 2 devices, we didn't notice any crash/issue.

Oh, good. I assume that this is same test which you were reporting crashes on previously.

So we still have the issue of changing IO scheduler and the tagset iters holding old references to tags. I tried Bart's idea to deal with the request queue tagset iter, and it seems to work, but not sure if acceptable. And also I might have a solution for blk_mq_tagset_busy_iter().

However I am not sure if these are seen in real-life, and whether the patch you tested is good enough for now.

Let me look at this again and I will report back.

Thanks very much,
John

quoted

quoted

Do you have a full kernel log for your crash?