Re: [PATCH net-next 0/9] mlx4: order-0 allocations and page recycling

On 07/02/2017 5:50 PM, Tariq Toukan wrote:

Hi Eric,

Thanks for your series.

On 07/02/2017 5:02 AM, Eric Dumazet wrote:

quoted

As mentioned half a year ago, we better switch mlx4 driver to order-0
allocations and page recycling.

This reduces vulnerability surface thanks to better skb->truesize 
tracking
and provides better performance in most cases.

The series makes significant change in the RX data-path, that requires 
deeper checks, in addition to code review.
We applied your series and started running both our functional and 
performance regression.
We will have results by tomorrow morning, and will analyze them during 
the day. I'll update about that.

We hit a kernel panic when running traffic after configuring a large MTU 
(9000).
I will take deeper look into this soon and will keep you updated.

[56136.982183] BUG: unable to handle kernel paging request at 
000000022f9e7020
[56136.990426] IP: mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en]
[56136.995303] PGD 220b7c067
[56136.995304] PUD 0
[56136.997941]
[56137.001807] Oops: 0000 [#1] SMP
[56137.004540] Modules linked in: netconsole mlx4_ib mlx4_en(E) 
mlx4_core(E) nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser 
libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp 
scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm 
dm_mirror dm_region_hash dm_log ib_cm dm_mod iw_cm ppdev parport_pc 
i2c_piix4 sg virtio_balloon parport pcspkr acpi_cpufreq nfsd auth_rpcgss 
nfs_acl lockd grace sunrpc ip_tables mlx5_ib sd_mod ata_generic 
pata_acpi ib_core mlx5_core cirrus drm_kms_helper syscopyarea 
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix floppy libata ptp 
e1000 crc32c_intel virtio_pci pps_core serio_raw virtio_ring i2c_core 
virtio [last unloaded: netconsole]
[56137.046028] CPU: 1 PID: 16 Comm: ksoftirqd/1 Tainted: G            
E   4.10.0-rc6+ #26
[56137.051501] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[56137.055817] task: ffff880236245200 task.stack: ffffc90000d04000
[56137.060154] RIP: 0010:mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en]
[56137.064712] RSP: 0018:ffffc90000d07c90 EFLAGS: 00010282
[56137.068646] RAX: 0000000000000003 RBX: 000000022f9e7000 RCX: 
ffff880234988880
[56137.073588] RDX: ffff8802349888e0 RSI: 0000000000000000 RDI: 
ffff880235dad0a0
[56137.078563] RBP: ffffc90000d07ce0 R08: 0000000000000000 R09: 
ffff8802225a08c0
[56137.083370] R10: ffff8802335c7800 R11: 0000000000000000 R12: 
ffffc90001da1048
[56137.088123] R13: 0000000000000b36 R14: ffff8802225af040 R15: 
0000000000000b36
[56137.092837] FS:  0000000000000000(0000) GS:ffff88023fc40000(0000) 
knlGS:0000000000000000
[56137.098495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[56137.102626] CR2: 000000022f9e7020 CR3: 000000023441f000 CR4: 
00000000000006e0
[56137.107581] Call Trace:
[56137.109955]  mlx4_en_process_rx_cq+0x35c/0xda0 [mlx4_en]
[56137.113894]  ? mlx4_en_free_tx_desc+0x14e/0x350 [mlx4_en]
[56137.117992]  ? load_balance+0x1ac/0x900
[56137.121285]  mlx4_en_poll_rx_cq+0x30/0xa0 [mlx4_en]
[56137.125023]  net_rx_action+0x23d/0x3a0
[56137.128146]  __do_softirq+0xd1/0x2a2
[56137.131178]  run_ksoftirqd+0x29/0x50
[56137.134180]  smpboot_thread_fn+0x110/0x160
[56137.137530]  kthread+0x101/0x140
[56137.140330]  ? sort_range+0x30/0x30
[56137.143255]  ? kthread_park+0x90/0x90
[56137.146304]  ? __kthread_parkme+0x50/0x70
[56137.149466]  ret_from_fork+0x2c/0x40
[56137.152426] Code: c0 8b 45 cc 41 8b 8a cc 00 00 00 48 63 d0 49 03 8a 
d0 00 00 00 48 83 c2 03 48 c1 e2 04 48 01 ca 48 89 1a 44 89 5a 08 44 89 
7a 0c <48> 8b 53 20 f6 c2 01 0f 85 76 01 00 00 48 89 da 48 83 7a 10 ff
[56137.164855] RIP: mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en] RSP: 
ffffc90000d07c90
[56137.170211] CR2: 000000022f9e7020
[56137.175430] ---[ end trace 6a259f16967a0cff ]---

quoted

Worth noting this patch series deletes more than 100 lines of code ;)

Eric Dumazet (9):
   mlx4: use __skb_fill_page_desc()
   mlx4: dma_dir is a mlx4_en_priv attribute
   mlx4: remove order field from mlx4_en_frag_info
   mlx4: get rid of frag_prefix_size
   mlx4: rx_headroom is a per port attribute
   mlx4: reduce rx ring page_cache size
   mlx4: removal of frag_sizes[]
   mlx4: use order-0 pages for RX
   mlx4: add page recycling in receive path

  drivers/net/ethernet/mellanox/mlx4/en_rx.c   | 350 
+++++++++------------------
  drivers/net/ethernet/mellanox/mlx4/en_tx.c   |   4 +-
  drivers/net/ethernet/mellanox/mlx4/mlx4_en.h |  28 +--
  3 files changed, 129 insertions(+), 253 deletions(-)

Thanks,
Tariq