Re: [PATCH net v2] napi: fix race inside napi_enable

Dany Madden [drt@linux.ibm.com] wrote:

We hit two napi related crashes while attempting mtu size change.

1st crash:
[430425.020051] ------------[ cut here ]------------
[430425.020053] kernel BUG at ../net/core/dev.c:6938!
[430425.020057] Oops: Exception in kernel mode, sig: 5 [#1]
[430425.020068] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[430425.020075] Modules linked in: binfmt_misc rpadlpar_io rpaphp xt_tcpudp
iptable_filter ip_tables x_tables pseries_rng ibmvnic rng_core ibmveth
vmx_crypto gf128mul fuse btrfs blake2b_generic xor zstd_compress
lzo_compress raid6_pq dm_service_time crc32c_vpmsum dm_mirror dm_region_hash
dm_log dm_multipath scsi_dh_rdac scsi_dh_alua autofs4
[430425.020123] CPU: 0 PID: 34337 Comm: kworker/0:3 Kdump: loaded Tainted: G
W     5.15.0-rc2-suka-00486-gce916130f5f6 #3
[430425.020133] Workqueue: events_long __ibmvnic_reset [ibmvnic]
[430425.020145] NIP: c000000000cb03f4 LR: c0080000014a4ce8 CTR:
c000000000cb03b0
[430425.020151] REGS: c00000002e5d37e0 TRAP: 0700  Tainted: G    W
(5.15.0-rc2-suka-00486-gce916130f5f6)
[430425.020159] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR:
28248428 XER: 20000001
[430425.020176] CFAR: c0080000014ad9cc IRQMASK: 0
        GPR00: c0080000014a4ce8 c00000002e5d3a80 c000000001b12100
c0000001274f3190
        GPR04: 00000000ffff36dc fffffffffffffff6 0000000000000019
0000000000000010
        GPR08: c00000002ec48900 0000000000000001 c0000001274f31a0
c0080000014ad9b8
        GPR12: c000000000cb03b0 c000000001d00000 0000000080000000
00000000000003fe
        GPR16: 00000000000006e3 0000000000000000 0000000000000008
c00000002ec48af8
        GPR20: c00000002ec48db0 0000000000000000 0000000000000004
0000000000000000
        GPR24: c00000002ec48000 0000000000000004 c00000002ec49070
0000000000000006
        GPR28: c00000002ec48900 c00000002ec48900 0000000000000002
c00000002ec48000
[430425.020248] NIP [c000000000cb03f4] napi_enable+0x44/0xc0
[430425.020257] LR [c0080000014a4ce8] __ibmvnic_open+0xf0/0x440 [ibmvnic]
[430425.020265] Call Trace:
[430425.020269] [c00000002e5d3a80] [c00000002ec48900] 0xc00000002ec48900
(unreliable)
[430425.020277] [c00000002e5d3ab0] [c0080000014a4f40]
__ibmvnic_open+0x348/0x440 [ibmvnic]
[430425.020286] [c00000002e5d3b40] [c0080000014ace58]
__ibmvnic_reset+0xb10/0xe40 [ibmvnic]
[430425.020296] [c00000002e5d3c60] [c0000000001673a4]
process_one_work+0x2d4/0x5d0
[430425.020305] [c00000002e5d3d00] [c000000000167718]
worker_thread+0x78/0x6c0
[430425.020314] [c00000002e5d3da0] [c000000000173388] kthread+0x188/0x190
[430425.020322] [c00000002e5d3e10] [c00000000000cee4]
ret_from_kernel_thread+0x5c/0x64
[430425.020331] Instruction dump:
[430425.020335] 38a0fff6 39430010 e92d0c80 f9210028 39200000 60000000
60000000 e9030010
[430425.020348] f9010020 e9210020 7d2948f8 792907e0 <0b090000> e9230038
7d072838 89290889
[430425.020364] ---[ end trace 3abb5ec5589518ca ]---
[430425.068100]
[430425.068108] Sending IPI to other CPUs
[430425.068206] IPI complete
[430425.090333] kexec: Starting switchover sequence.

Jakub,

We hit this napi_enable() BUG_ON() crash three times this week. In two
of those times it appears that

	napi->state = netdev_priv(netdev)

i.e it contains ibmvnic_adapter* in our case.

	# Crash was on eth3

	crash> net |grep eth3
	c00000002e948000  eth3   10.1.194.173

	crash> net_device |grep SIZE
	SIZE: 2304

	crash> px 2304
	$1 = 0x900

	crash> ibmvnic_adapter c00000002e948900 |grep napi
	  napi = 0xc00000003b7dc000,
	  num_active_rx_napi = 8,
	  napi_enabled = false,

	crash> napi_struct 0xc00000003b7dc000 |grep state
	  state = 13835058056063650048,
	    state = 0 '\000',

	crash> px 13835058056063650048
	$2 = 0xc00000002e948900		#eth3 ibmvnic_adapter above

In the third case napi->state was 16 (i.e NAPI_STATE_SCHED was clear and
hence the bug in napi_enable()).

Let us know if any other fields are of interest. Do we have any clues on
when this started?

Sukadev