Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to the stable releases from v5.4 and older. It fixes a spin_unlock_bh() in peernet2id() called with IRQs off. I think this neat side-effect of commit 2dce224f469f was quite un-intentional, hence no Fixes: tag or CC: stable.
The details:
From bugzilla.redhat.com/show_bug.cgi?id=1384179 (an ancient 4.9.0-0.rc0 kernel):
dump_stack+0x86/0xc3
__warn+0xcb/0xf0
warn_slowpath_null+0x1d/0x20
__local_bh_enable_ip+0x9d/0xc0
_raw_spin_unlock_bh+0x35/0x40
peernet2id+0x54/0x80
netlink_broadcast_filtered+0x220/0x3c0
netlink_broadcast+0x1d/0x20
audit_log+0x6a/0x90
security_set_bools+0xee/0x200
[]
Note, security_set_bools() calls write_lock_irq(). peernet2id() calls spin_unlock_bh().
From an internal (UEK) stack trace based on the v4.14.35 kernel (LTS 4.14.231):
queued_spin_lock_slowpath+0xb/0xf
_raw_spin_lock_irqsave+0x46/0x48
send_mad+0x3d2/0x590 [ib_core]
ib_sa_path_rec_get+0x223/0x4d0 [ib_core]
path_rec_start+0xa3/0x140 [ib_ipoib]
ipoib_start_xmit+0x2b0/0x6a0 [ib_ipoib]
dev_hard_start_xmit+0xb2/0x237
sch_direct_xmit+0x114/0x1bf
__dev_queue_xmit+0x592/0x818
dev_queue_xmit+0x10/0x12
arp_xmit+0x38/0xa6
arp_send_dst.part.16+0x61/0x84
arp_process+0x825/0x889
arp_rcv+0x140/0x1c9
__netif_receive_skb_core+0x401/0xb39
__netif_receive_skb+0x18/0x59
netif_receive_skb_internal+0x45/0x119
napi_gro_receive+0xd8/0xf6
ipoib_ib_handle_rx_wc+0x1ca/0x520 [ib_ipoib]
ipoib_poll+0xcd/0x150 [ib_ipoib]
net_rx_action+0x289/0x3f4
__do_softirq+0xe1/0x2b5
do_softirq_own_stack+0x2a/0x35
</IRQ>
do_softirq+0x4d/0x6a
__local_bh_enable_ip+0x57/0x59
_raw_spin_unlock_bh+0x23/0x25
peernet2id+0x51/0x73
netlink_broadcast_filtered+0x223/0x41b
netlink_broadcast+0x1d/0x1f
rdma_nl_multicast+0x22/0x30 [ib_core]
send_mad+0x3e5/0x590 [ib_core]
ib_sa_path_rec_get+0x223/0x4d0 [ib_core]
rdma_resolve_route+0x287/0x810 [rdma_cm]
rds_rdma_cm_event_handler_cmn+0x311/0x7d0 [rds_rdma]
rds_rdma_cm_event_handler_worker+0x22/0x30 [rds_rdma]
process_one_work+0x169/0x3a6
worker_thread+0x4d/0x3e5
kthread+0x105/0x138
ret_from_fork+0x24/0x49
Here, pay attention to ib_nl_make_request() which calls spin_lock_irqsave() on a global lock just before calling rdma_nl_multicast(). Thereafter, peernet2id() enables SoftIRQs, and ipoib starts and calls the same path and end up trying to acquire the same global lock again.
I have tried to repro this with no luck. But, stack traces seldom lies ;-)
Thxs, Håkon
On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
RCU") to the stable releases from v5.4 and older. It fixes a
spin_unlock_bh() in peernet2id() called with IRQs off. I think this
neat side-effect of commit 2dce224f469f was quite un-intentional,
hence no Fixes: tag or CC: stable.
Please provide a working backport for all of the relevant kernel
verisons, as it does not apply cleanly on it's own.
thanks,
greg k-h
On 9 Sep 2021, at 15:30, Greg KH [off-list ref] wrote:
On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
quoted
Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
RCU") to the stable releases from v5.4 and older. It fixes a
spin_unlock_bh() in peernet2id() called with IRQs off. I think this
neat side-effect of commit 2dce224f469f was quite un-intentional,
hence no Fixes: tag or CC: stable.
Please provide a working backport for all of the relevant kernel
verisons, as it does not apply cleanly on it's own.
I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
fba143c66abb ("netns: avoid disabling irq for netns id")
Will get'm out during the weekend.
Thxs, Håkon
On Fri, Sep 10, 2021 at 02:22:32PM +0000, Haakon Bugge wrote:
quoted
On 9 Sep 2021, at 15:30, Greg KH [off-list ref] wrote:
On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
quoted
Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
RCU") to the stable releases from v5.4 and older. It fixes a
spin_unlock_bh() in peernet2id() called with IRQs off. I think this
neat side-effect of commit 2dce224f469f was quite un-intentional,
hence no Fixes: tag or CC: stable.
Please provide a working backport for all of the relevant kernel
verisons, as it does not apply cleanly on it's own.
I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
fba143c66abb ("netns: avoid disabling irq for netns id")
Will get'm out during the weekend.
On 13 Sep 2021, at 10:42, Greg KH [off-list ref] wrote:
On Fri, Sep 10, 2021 at 02:22:32PM +0000, Haakon Bugge wrote:
quoted
quoted
On 9 Sep 2021, at 15:30, Greg KH [off-list ref] wrote:
On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
quoted
Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
RCU") to the stable releases from v5.4 and older. It fixes a
spin_unlock_bh() in peernet2id() called with IRQs off. I think this
neat side-effect of commit 2dce224f469f was quite un-intentional,
hence no Fixes: tag or CC: stable.
Please provide a working backport for all of the relevant kernel
verisons, as it does not apply cleanly on it's own.
I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
fba143c66abb ("netns: avoid disabling irq for netns id")
Will get'm out during the weekend.
All now queued up, thanks.
Thanks for helping me out here, Greg!
Appreciated, Håkon