ODEBUG: object ffffc90000fd8bc8 is NOT on stack ffffc900022a0000, but annotated.
This is saying that the object was supposed to be on the stack because
debug objects was told that, but it isn't on the stack per the
definition of object_is_on_stack().
This line looks like
INIT_WORK_ONSTACK(&rew.rew_work, wait_rcu_exp_gp);
inside synchronize_rcu_expedited(). The rew structure is declared on the
stack
struct rcu_exp_work rew;
Yes, but object_is_on_stack() checks for task stacks only. And the splat
here is entirely correct:
softirq()
...
synchronize_rcu_expedited()
INIT_WORK_ONSTACK()
queue_work()
wait_event()
is obviously broken. You cannot wait in soft irq context.
synchronize_rcu_expedited() should really have a might_sleep() at the
beginning to make that more obvious.
The splat is clobbered btw:
[ 416.415111][ C1] ODEBUG: object ffffc90000fd8bc8 is NOT on stack ffffc900022a0000, but annotated.
[ 416.423424][T14850] truncated
[ 416.431623][ C1] ------------[ cut here ]------------
[ 416.438913][T14850] ------------[ cut here ]------------
[ 416.440189][ C1] WARNING: CPU: 1 PID: 2971 at lib/debugobjects.c:548 __debug_object_init.cold+0x252/0x2e5
[ 416.455797][T14850] refcount_t: addition on 0; use-after-free.
So there is a refcount_t violation as well.
Nevertheless a hint for finding the culprit is obviously here in that
call chain:
Seems that we have a debugobject in the incorrect state, but it doesn't
necessarily mean there's something wrong in the bdi code. It's just
that the bdi code happened to be the place which called
synchronize_rcu_expedited().
Again, it cannot do that from a softirq because
synchronize_rcu_expedited() might sleep.
Is it possible that object_is_on_stack() doesn't work in IRQ context?
I'm not really following along on x86 but I could see where
task_stack_page() gets the wrong "stack" pointer because the task has one
stack and the irq stack is some per-cpu dedicated allocation?
Even if debug objects would support objects on irq stacks, the above is
still bogus. But it does not and will not because the operations here
have to be fully synchronous:
init() -> queue() or arm() -> wait() -> destroy()
because you obviously cannot queue work or arm a timer which are on stack
and then leave the function without waiting for the operation to complete.
So these operations have to be synchronous which is a NONO when running
in hard or soft interrupt context because waiting for the operation to
complete is not possible there.
Thanks,
tglx
ODEBUG: object ffffc90000fd8bc8 is NOT on stack ffffc900022a0000, but annotated.
This is saying that the object was supposed to be on the stack because
debug objects was told that, but it isn't on the stack per the
definition of object_is_on_stack().
This line looks like
INIT_WORK_ONSTACK(&rew.rew_work, wait_rcu_exp_gp);
inside synchronize_rcu_expedited(). The rew structure is declared on the
stack
struct rcu_exp_work rew;
Yes, but object_is_on_stack() checks for task stacks only. And the splat
here is entirely correct:
softirq()
...
synchronize_rcu_expedited()
INIT_WORK_ONSTACK()
queue_work()
wait_event()
is obviously broken. You cannot wait in soft irq context.
synchronize_rcu_expedited() should really have a might_sleep() at the
beginning to make that more obvious.
The splat is clobbered btw:
[ 416.415111][ C1] ODEBUG: object ffffc90000fd8bc8 is NOT on stack ffffc900022a0000, but annotated.
[ 416.423424][T14850] truncated
[ 416.431623][ C1] ------------[ cut here ]------------
[ 416.438913][T14850] ------------[ cut here ]------------
[ 416.440189][ C1] WARNING: CPU: 1 PID: 2971 at lib/debugobjects.c:548 __debug_object_init.cold+0x252/0x2e5
[ 416.455797][T14850] refcount_t: addition on 0; use-after-free.
So there is a refcount_t violation as well.
Nevertheless a hint for finding the culprit is obviously here in that
call chain:
The inode code uses RCU for freeing an inode object which then ends up
calling bdi_put() and subsequently in synchronize_rcu_expedited().
Commit 889c05cc5834 ("block: ensure the bdi is freed after
inode_detach_wb") might be a good place to start looking here. It
moved the release of the bdi from ->evict context to the RCU freeing
of the blockdev inode...
Christoph?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
The inode code uses RCU for freeing an inode object which then ends up
calling bdi_put() and subsequently in synchronize_rcu_expedited().
Commit 889c05cc5834 ("block: ensure the bdi is freed after
inode_detach_wb") might be a good place to start looking here. It
moved the release of the bdi from ->evict context to the RCU freeing
of the blockdev inode...
Well, the block code already does a bdi_unregister in del_gendisk.
So if we end up freeing the whole device bdev with a registered bdi
something is badly going wrong. Unfortunately the log in this report
isn't much help on how we got there. IIRC syzbot will eventually spew
out a reproducer, so it might be worth to wait for that.
The inode code uses RCU for freeing an inode object which then ends up
calling bdi_put() and subsequently in synchronize_rcu_expedited().
Commit 889c05cc5834 ("block: ensure the bdi is freed after
inode_detach_wb") might be a good place to start looking here. It
moved the release of the bdi from ->evict context to the RCU freeing
of the blockdev inode...
Well, the block code already does a bdi_unregister in del_gendisk.
So if we end up freeing the whole device bdev with a registered bdi
something is badly going wrong. Unfortunately the log in this report
isn't much help on how we got there. IIRC syzbot will eventually spew
out a reproducer, so it might be worth to wait for that.
If it does turn out that you need to block in an RCU callback,
queue_rcu_work() can be helpful. This schedules a workqueue from the RCU
callback, allowing the function passed to the preceding INIT_RCU_WORK()
to block.
Thanx, Paul
From: Christoph Hellwig <hch@lst.de> Date: 2021-09-20 12:46:02
On Mon, Sep 20, 2021 at 05:38:59AM -0700, Paul E. McKenney wrote:
quoted
Well, the block code already does a bdi_unregister in del_gendisk.
So if we end up freeing the whole device bdev with a registered bdi
something is badly going wrong. Unfortunately the log in this report
isn't much help on how we got there. IIRC syzbot will eventually spew
out a reproducer, so it might be worth to wait for that.
If it does turn out that you need to block in an RCU callback,
queue_rcu_work() can be helpful. This schedules a workqueue from the RCU
callback, allowing the function passed to the preceding INIT_RCU_WORK()
to block.
In this case we really should not block here. The problem is that
we are hitting the strange bdi auto-unregister misfeature due to a bug
elsewhere. Which reminds that I have a patch series to remove this
auto unregistration which I need to bring bag once this is fixed.
That being said queue_rcu_work would have been really useful in a few
places I touched in that past.
From: "Paul E. McKenney" <paulmck@kernel.org> Date: 2021-09-20 12:54:34
On Mon, Sep 20, 2021 at 02:45:57PM +0200, Christoph Hellwig wrote:
On Mon, Sep 20, 2021 at 05:38:59AM -0700, Paul E. McKenney wrote:
quoted
quoted
Well, the block code already does a bdi_unregister in del_gendisk.
So if we end up freeing the whole device bdev with a registered bdi
something is badly going wrong. Unfortunately the log in this report
isn't much help on how we got there. IIRC syzbot will eventually spew
out a reproducer, so it might be worth to wait for that.
If it does turn out that you need to block in an RCU callback,
queue_rcu_work() can be helpful. This schedules a workqueue from the RCU
callback, allowing the function passed to the preceding INIT_RCU_WORK()
to block.
In this case we really should not block here. The problem is that
we are hitting the strange bdi auto-unregister misfeature due to a bug
elsewhere. Which reminds that I have a patch series to remove this
auto unregistration which I need to bring bag once this is fixed.
That being said queue_rcu_work would have been really useful in a few
places I touched in that past.
Glad it helped elsewhere and apologies for the noise here!
Thanx, Paul