On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Sounds like it is. 4.19 has the new ANA code, so the multipath code
has some churn.
BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x207/0x290
Can you resolve the address using gdb on vmlinux to a specific
line of code?
From: Keith Busch <hidden> Date: 2018-09-20 17:01:45
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Not sure if the following is what you're hitting since it wouldn't be a
regression. It looks like removing a namespace path and clearing it as
the current path is in the wrong order, such that the very next IO
may reference the namespace being deleted.
---
From: bvanassche@acm.org (Bart Van Assche) Date: 2018-09-20 17:24:01
On Thu, 2018-09-20@00:10 -0700, Christoph Hellwig wrote:
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Sounds like it is. 4.19 has the new ANA code, so the multipath code
has some churn.
quoted
BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x207/0x290
Can you resolve the address using gdb on vmlinux to a specific
line of code?
Sure. The gdb output (which is probably not very useful) is as follows:
(gdb) list *(srcu_invoke_callbacks+0x207)
0xffffffff811872e7 is in srcu_invoke_callbacks (./include/linux/compiler.h:188).
183 })
184
185 static __always_inline
186 void __read_once_size(const volatile void *p, void *res, int size)
187 {
188 __READ_ONCE_SIZE;
189 }
190
191 #ifdef CONFIG_KASAN
192 /*
This may be more useful:
(gdb) list *(srcu_invoke_callbacks+0x1fa)
0xffffffff811872da is in srcu_invoke_callbacks (kernel/rcu/srcutree.c:1206).
1201 /*
1202 * Update counts, accelerate new callbacks, and if needed,
1203 * schedule another round of callback invocation.
1204 */
1205 spin_lock_irq_rcu_node(sdp);
1206 rcu_segcblist_insert_count(&sdp->srcu_cblist, &ready_cbs);
1207 (void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
1208 rcu_seq_snap(&sp->srcu_gp_seq));
1209 sdp->srcu_cblist_invoking = false;
1210 more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist);
Bart.
From: bvanassche@acm.org (Bart Van Assche) Date: 2018-09-20 17:31:29
On Thu, 2018-09-20@11:01 -0600, Keith Busch wrote:
quoted hunk
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Not sure if the following is what you're hitting since it wouldn't be a
regression. It looks like removing a namespace path and clearing it as
the current path is in the wrong order, such that the very next IO
may reference the namespace being deleted.
---
From: Keith Busch <hidden> Date: 2018-09-20 17:36:25
On Thu, Sep 20, 2018@10:31:29AM -0700, Bart Van Assche wrote:
On Thu, 2018-09-20@11:01 -0600, Keith Busch wrote:
quoted
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Not sure if the following is what you're hitting since it wouldn't be a
regression. It looks like removing a namespace path and clearing it as
the current path is in the wrong order, such that the very next IO
may reference the namespace being deleted.
---
From: bvanassche@acm.org (Bart Van Assche) Date: 2018-09-20 17:36:30
On Thu, 2018-09-20@10:31 -0700, Bart Van Assche wrote:
On Thu, 2018-09-20@11:01 -0600, Keith Busch wrote:
quoted
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Not sure if the following is what you're hitting since it wouldn't be a
regression. It looks like removing a namespace path and clearing it as
the current path is in the wrong order, such that the very next IO
may reference the namespace being deleted.
---
From: Keith Busch <hidden> Date: 2018-09-20 17:45:10
On Thu, Sep 20, 2018@10:36:30AM -0700, Bart Van Assche wrote:
On Thu, 2018-09-20@10:31 -0700, Bart Van Assche wrote:
quoted
On Thu, 2018-09-20@11:01 -0600, Keith Busch wrote:
quoted
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Not sure if the following is what you're hitting since it wouldn't be a
regression. It looks like removing a namespace path and clearing it as
the current path is in the wrong order, such that the very next IO
may reference the namespace being deleted.
---
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Sounds like it is. 4.19 has the new ANA code, so the multipath code
has some churn.
Note that this is testing nvmf with dm-multipath.
quoted
BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x207/0x290
Can you resolve the address using gdb on vmlinux to a specific
line of code?
Did this get resolved?
Bart, did this reproduce with nvme.multipath=1? or was it disabled?
[Adding Paul]
Hi Paul,
Bart reported a use after free in the SRCU code when testing the
nvme multipath code here:
http://lists.infradead.org/pipermail/linux-nvme/2018-September/020009.html
Based on his analsys it appears to me the use after free is on the
srcu_data structure, which is internal to the SRCU implementation.
While I don't want to exclude an actual cause in the nvme code I wonder
if you have any additional insights from the RCU perspective.
On Thu, Sep 20, 2018@10:24:01AM -0700, Bart Van Assche wrote:
On Thu, 2018-09-20@00:10 -0700, Christoph Hellwig wrote:
quoted
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
Sounds like it is. 4.19 has the new ANA code, so the multipath code
has some churn.
quoted
BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x207/0x290
Can you resolve the address using gdb on vmlinux to a specific
line of code?
Sure. The gdb output (which is probably not very useful) is as follows:
(gdb) list *(srcu_invoke_callbacks+0x207)
0xffffffff811872e7 is in srcu_invoke_callbacks (./include/linux/compiler.h:188).
183 })
184
185 static __always_inline
186 void __read_once_size(const volatile void *p, void *res, int size)
187 {
188 __READ_ONCE_SIZE;
189 }
190
191 #ifdef CONFIG_KASAN
192 /*
This may be more useful:
(gdb) list *(srcu_invoke_callbacks+0x1fa)
0xffffffff811872da is in srcu_invoke_callbacks (kernel/rcu/srcutree.c:1206).
1201 /*
1202 * Update counts, accelerate new callbacks, and if needed,
1203 * schedule another round of callback invocation.
1204 */
1205 spin_lock_irq_rcu_node(sdp);
1206 rcu_segcblist_insert_count(&sdp->srcu_cblist, &ready_cbs);
1207 (void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
1208 rcu_seq_snap(&sp->srcu_gp_seq));
1209 sdp->srcu_cblist_invoking = false;
1210 more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist);
Bart.
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
From: Paul E. McKenney <hidden> Date: 2018-09-26 03:14:17
On Tue, Sep 25, 2018@04:32:11PM -0700, Christoph Hellwig wrote:
[Adding Paul]
Hi Paul,
Bart reported a use after free in the SRCU code when testing the
nvme multipath code here:
http://lists.infradead.org/pipermail/linux-nvme/2018-September/020009.html
Based on his analsys it appears to me the use after free is on the
srcu_data structure, which is internal to the SRCU implementation.
While I don't want to exclude an actual cause in the nvme code I wonder
if you have any additional insights from the RCU perspective.
On Thu, Sep 20, 2018@10:24:01AM -0700, Bart Van Assche wrote:
quoted
On Thu, 2018-09-20@00:10 -0700, Christoph Hellwig wrote:
quoted
On Tue, Sep 18, 2018@02:16:48PM -0700, Bart Van Assche wrote:
quoted
Hello,
If I run the nvmeof-mp tests from https://github.com/bvanassche/blktests
against kernel v4.19-rc4 then a KASAN complaint appears. This complaint does
not appear when I run these tests against kernel v4.18. Could this be a
regression?
I would be quite surprised if any of the SRCU commits since v4.18 caused
this sort of a problem, but there are not that many of them. so easy to
check (at least assuming that this is reproducible):
gitk v4.18.. -- kernel/rcu/srcu* include/linux/*srcu*
But checking below...
quoted
quoted
Sounds like it is. 4.19 has the new ANA code, so the multipath code
has some churn.
quoted
BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x207/0x290
Can you resolve the address using gdb on vmlinux to a specific
line of code?
Sure. The gdb output (which is probably not very useful) is as follows:
(gdb) list *(srcu_invoke_callbacks+0x207)
0xffffffff811872e7 is in srcu_invoke_callbacks (./include/linux/compiler.h:188).
183 })
184
185 static __always_inline
186 void __read_once_size(const volatile void *p, void *res, int size)
187 {
188 __READ_ONCE_SIZE;
189 }
190
191 #ifdef CONFIG_KASAN
192 /*
This may be more useful:
(gdb) list *(srcu_invoke_callbacks+0x1fa)
0xffffffff811872da is in srcu_invoke_callbacks (kernel/rcu/srcutree.c:1206).
1201 /*
1202 * Update counts, accelerate new callbacks, and if needed,
1203 * schedule another round of callback invocation.
1204 */
1205 spin_lock_irq_rcu_node(sdp);
1206 rcu_segcblist_insert_count(&sdp->srcu_cblist, &ready_cbs);
1207 (void)rcu_segcblist_accelerate(&sdp->srcu_cblist,
1208 rcu_seq_snap(&sp->srcu_gp_seq));
1209 sdp->srcu_cblist_invoking = false;
1210 more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist);
I would expect something like this if someone did a double call_srcu()
or passed something to call_srcu() but then kept using it (for an example
of the latter, failed to make it inaccessible to readers before invoking
call_srcu() on it). Yet another way to get here is to have unioned the
rcu_head structure with something used by the SRCU readers.
The double call_srcu() can be located by building your kernel with
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y and rerunning your tests. The other
two usually require inspection or bisection.
So, the eternal question: Is bisection feasible?
Thanx, Paul
On Tue, Sep 25, 2018@08:14:17PM -0700, Paul E. McKenney wrote:
I would expect something like this if someone did a double call_srcu()
or passed something to call_srcu() but then kept using it (for an example
of the latter, failed to make it inaccessible to readers before invoking
call_srcu() on it). Yet another way to get here is to have unioned the
rcu_head structure with something used by the SRCU readers.
The double call_srcu() can be located by building your kernel with
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y and rerunning your tests. The other
two usually require inspection or bisection.
So, the eternal question: Is bisection feasible?
Looks like I misread the lines pointed to by gdb, and it indeed seems
like a premature free of the nvme_ns_head structure, but I fail to see
where.
I've tried to bring up Barts testcase but failed so far.
Bart, can you help out a bit?
The last commit of the original merge 4.19 nvme merge is:
b369b30cf510fe94d8884837039362e2ec223cec
Can you check if that shows the problem (I suspect it does) and if so
bisect between that and 9b89bc3857a6c0dfda18ddae2a42c114ecc32753, which
as the first commit for the merge?
On Fri, Oct 05, 2018@12:38:33AM -0700, Christoph Hellwig wrote:
Looks like I misread the lines pointed to by gdb, and it indeed seems
like a premature free of the nvme_ns_head structure, but I fail to see
where.
I've tried to bring up Barts testcase but failed so far.
Bart, can you help out a bit?
The last commit of the original merge 4.19 nvme merge is:
b369b30cf510fe94d8884837039362e2ec223cec
Can you check if that shows the problem (I suspect it does) and if so
bisect between that and 9b89bc3857a6c0dfda18ddae2a42c114ecc32753, which
as the first commit for the merge?
From: bvanassche@acm.org (Bart Van Assche) Date: 2018-10-17 14:38:35
On 10/16/18 11:39 PM, Christoph Hellwig wrote:
On Fri, Oct 05, 2018@12:38:33AM -0700, Christoph Hellwig wrote:
quoted
Looks like I misread the lines pointed to by gdb, and it indeed seems
like a premature free of the nvme_ns_head structure, but I fail to see
where.
I've tried to bring up Barts testcase but failed so far.
Bart, can you help out a bit?
The last commit of the original merge 4.19 nvme merge is:
b369b30cf510fe94d8884837039362e2ec223cec
Can you check if that shows the problem (I suspect it does) and if so
bisect between that and 9b89bc3857a6c0dfda18ddae2a42c114ecc32753, which
as the first commit for the merge?
Bart, do you cycles to help out bisecting this?
Hi Christoph,
I will try to free up some time to search for the root cause of this
complaint.
Bart.
From: bvanassche@acm.org (Bart Van Assche) Date: 2018-10-17 17:32:27
On 10/16/18 11:39 PM, Christoph Hellwig wrote:
On Fri, Oct 05, 2018@12:38:33AM -0700, Christoph Hellwig wrote:
quoted
Looks like I misread the lines pointed to by gdb, and it indeed seems
like a premature free of the nvme_ns_head structure, but I fail to see
where.
I've tried to bring up Barts testcase but failed so far.
Bart, can you help out a bit?
The last commit of the original merge 4.19 nvme merge is:
b369b30cf510fe94d8884837039362e2ec223cec
Can you check if that shows the problem (I suspect it does) and if so
bisect between that and 9b89bc3857a6c0dfda18ddae2a42c114ecc32753, which
as the first commit for the merge?
Bart, do you cycles to help out bisecting this?
Hi Christoph,
With your current nvme-4.20 branch I have not been able to reproduce
this. The nvmeof-mp test scripts have not been changed. So I don't think
we have to spend more time on this.
Bart.