Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
where it seems that num_tc_x_num_ch can become zero in a modulo operation:
if (unlikely(txq_ix >= num_tc_x_num_ch))
txq_ix %= num_tc_x_num_ch;
I think error were introduced in:
- 214baf22870c ("net/mlx5e: Support HTB offload")
The modulo operation was introduced in:
- 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
The crash looks like this:
[ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
[ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
[ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
[ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
[ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
[ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
[ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
[ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
This is an RFC, because I don't think this is a proper fix, but
at least it allows me to boot with mlx5 driver enabled.
Signed-off-by: Jesper Dangaard Brouer <redacted>
---
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index bdbffe484fce..4dc26c1f2b53 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -131,6 +131,7 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
int ch_ix;
/* Sync with mlx5e_update_num_tc_x_num_ch - avoid refetching. */
+ // XXX this reading of num_tc_x_num_ch looks strange
num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);
if (unlikely(dev->real_num_tx_queues > num_tc_x_num_ch)) {
/* Order maj_id before defcls - pairs with mlx5e_htb_root_add. */@@ -142,19 +143,21 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
return txq_ix;
}
- if (unlikely(priv->channels.port_ptp))
+ txq_ix = netdev_pick_tx(dev, skb, NULL);
+
+ if (unlikely(priv->channels.port_ptp)) {
if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) &&
mlx5e_use_ptpsq(skb))
return mlx5e_select_ptpsq(dev, skb);
- txq_ix = netdev_pick_tx(dev, skb, NULL);
/* Fix netdev_pick_tx() not to choose ptp_channel and HTB txqs.
* If they are selected, switch to regular queues.
* Driver to select these queues only at mlx5e_select_ptpsq()
* and mlx5e_select_htb_queue().
*/
- if (unlikely(txq_ix >= num_tc_x_num_ch))
- txq_ix %= num_tc_x_num_ch;
+ if (unlikely(txq_ix >= num_tc_x_num_ch))
+ txq_ix %= num_tc_x_num_ch;
+ }
} else {
txq_ix = netdev_pick_tx(dev, skb, NULL);
}
On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
where it seems that num_tc_x_num_ch can become zero in a modulo operation:
if (unlikely(txq_ix >= num_tc_x_num_ch))
txq_ix %= num_tc_x_num_ch;
I think error were introduced in:
- 214baf22870c ("net/mlx5e: Support HTB offload")
The modulo operation was introduced in:
- 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
The crash looks like this:
[ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
[ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
[ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
[ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
[ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
[ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
[ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
[ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
This is an RFC, because I don't think this is a proper fix, but
at least it allows me to boot with mlx5 driver enabled.
Signed-off-by: Jesper Dangaard Brouer <redacted>
---
Hi Jesper,
Thanks for your report and patch.
We'll check it and post the proper fix.
I wonder what's special in your configuration / use case, as I'm not
aware of such failure.
Can you please share more info about the reproduction?
Regards,
Tariq
On Tue, 23 Feb 2021 10:28:35 +0200
Tariq Toukan [off-list ref] wrote:
On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
quoted
Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
where it seems that num_tc_x_num_ch can become zero in a modulo operation:
if (unlikely(txq_ix >= num_tc_x_num_ch))
txq_ix %= num_tc_x_num_ch;
I think error were introduced in:
- 214baf22870c ("net/mlx5e: Support HTB offload")
The modulo operation was introduced in:
- 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
The crash looks like this:
[ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
[ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
[ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
[ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
[ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
[ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
[ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
[ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
This is an RFC, because I don't think this is a proper fix, but
at least it allows me to boot with mlx5 driver enabled.
Signed-off-by: Jesper Dangaard Brouer <redacted>
---
Hi Jesper,
Thanks for your report and patch.
We'll check it and post the proper fix.
I wonder what's special in your configuration / use case, as I'm not
aware of such failure.
Can you please share more info about the reproduction?
The machine simply crash under boot, that is how I reproduced.
You should look at why: num_tc_x_num_ch can be zero in:
u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
struct net_device *sb_dev)
{
[...]
num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);
Here are some info on the mlx5 NIC I'm using:
$ ethtool -i mlx5p1
driver: mlx5_core
version: 5.11.0-net-next-alloc_pages_bul
firmware-version: 16.23.1020 (MT_0000000009)
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies Device 0002
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 26
Memory at f0000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fba00000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1c0] Secondary PCI Express
Capabilities: [230] Access Control Services
Capabilities: [320] Lane Margining at the Receiver <?>
Capabilities: [370] Physical Layer 16.0 GT/s <?>
Capabilities: [420] Data Link Feature <?>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
03:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies Device 0002
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 86
Memory at ee000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fb900000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [230] Access Control Services
Capabilities: [420] Data Link Feature <?>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer