Re: [PATCH net-next 0/9] mlx4: order-0 allocations and page recycling

On 08/02/2017 11:02 AM, Tariq Toukan wrote:

On 07/02/2017 5:50 PM, Tariq Toukan wrote:

quoted

Hi Eric,

Thanks for your series.

On 07/02/2017 5:02 AM, Eric Dumazet wrote:

quoted

As mentioned half a year ago, we better switch mlx4 driver to order-0
allocations and page recycling.

This reduces vulnerability surface thanks to better skb->truesize 
tracking
and provides better performance in most cases.

The series makes significant change in the RX data-path, that 
requires deeper checks, in addition to code review.
We applied your series and started running both our functional and 
performance regression.
We will have results by tomorrow morning, and will analyze them 
during the day. I'll update about that.

We hit a kernel panic when running traffic after configuring a large 
MTU (9000).
I will take deeper look into this soon and will keep you updated.

Doesn't happen before applying patch 9/9:
mlx4: add page recycling in receive path

[56136.982183] BUG: unable to handle kernel paging request at 
000000022f9e7020
[56136.990426] IP: mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en]
[56136.995303] PGD 220b7c067
[56136.995304] PUD 0
[56136.997941]
[56137.001807] Oops: 0000 [#1] SMP
[56137.004540] Modules linked in: netconsole mlx4_ib mlx4_en(E) 
mlx4_core(E) nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod 
ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp 
scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm 
dm_mirror dm_region_hash dm_log ib_cm dm_mod iw_cm ppdev parport_pc 
i2c_piix4 sg virtio_balloon parport pcspkr acpi_cpufreq nfsd 
auth_rpcgss nfs_acl lockd grace sunrpc ip_tables mlx5_ib sd_mod 
ata_generic pata_acpi ib_core mlx5_core cirrus drm_kms_helper 
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix floppy 
libata ptp e1000 crc32c_intel virtio_pci pps_core serio_raw 
virtio_ring i2c_core virtio [last unloaded: netconsole]
[56137.046028] CPU: 1 PID: 16 Comm: ksoftirqd/1 Tainted: G            
E   4.10.0-rc6+ #26
[56137.051501] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[56137.055817] task: ffff880236245200 task.stack: ffffc90000d04000
[56137.060154] RIP: 0010:mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en]
[56137.064712] RSP: 0018:ffffc90000d07c90 EFLAGS: 00010282
[56137.068646] RAX: 0000000000000003 RBX: 000000022f9e7000 RCX: 
ffff880234988880
[56137.073588] RDX: ffff8802349888e0 RSI: 0000000000000000 RDI: 
ffff880235dad0a0
[56137.078563] RBP: ffffc90000d07ce0 R08: 0000000000000000 R09: 
ffff8802225a08c0
[56137.083370] R10: ffff8802335c7800 R11: 0000000000000000 R12: 
ffffc90001da1048
[56137.088123] R13: 0000000000000b36 R14: ffff8802225af040 R15: 
0000000000000b36
[56137.092837] FS:  0000000000000000(0000) GS:ffff88023fc40000(0000) 
knlGS:0000000000000000
[56137.098495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[56137.102626] CR2: 000000022f9e7020 CR3: 000000023441f000 CR4: 
00000000000006e0
[56137.107581] Call Trace:
[56137.109955]  mlx4_en_process_rx_cq+0x35c/0xda0 [mlx4_en]
[56137.113894]  ? mlx4_en_free_tx_desc+0x14e/0x350 [mlx4_en]
[56137.117992]  ? load_balance+0x1ac/0x900
[56137.121285]  mlx4_en_poll_rx_cq+0x30/0xa0 [mlx4_en]
[56137.125023]  net_rx_action+0x23d/0x3a0
[56137.128146]  __do_softirq+0xd1/0x2a2
[56137.131178]  run_ksoftirqd+0x29/0x50
[56137.134180]  smpboot_thread_fn+0x110/0x160
[56137.137530]  kthread+0x101/0x140
[56137.140330]  ? sort_range+0x30/0x30
[56137.143255]  ? kthread_park+0x90/0x90
[56137.146304]  ? __kthread_parkme+0x50/0x70
[56137.149466]  ret_from_fork+0x2c/0x40
[56137.152426] Code: c0 8b 45 cc 41 8b 8a cc 00 00 00 48 63 d0 49 03 
8a d0 00 00 00 48 83 c2 03 48 c1 e2 04 48 01 ca 48 89 1a 44 89 5a 08 
44 89 7a 0c <48> 8b 53 20 f6 c2 01 0f 85 76 01 00 00 48 89 da 48 83 7a 
10 ff
[56137.164855] RIP: mlx4_en_complete_rx_desc+0x130/0x2e0 [mlx4_en] 
RSP: ffffc90000d07c90
[56137.170211] CR2: 000000022f9e7020
[56137.175430] ---[ end trace 6a259f16967a0cff ]---

quoted

quoted

Worth noting this patch series deletes more than 100 lines of code ;)

Eric Dumazet (9):
   mlx4: use __skb_fill_page_desc()
   mlx4: dma_dir is a mlx4_en_priv attribute
   mlx4: remove order field from mlx4_en_frag_info
   mlx4: get rid of frag_prefix_size
   mlx4: rx_headroom is a per port attribute
   mlx4: reduce rx ring page_cache size
   mlx4: removal of frag_sizes[]
   mlx4: use order-0 pages for RX
   mlx4: add page recycling in receive path

  drivers/net/ethernet/mellanox/mlx4/en_rx.c   | 350 
+++++++++------------------
  drivers/net/ethernet/mellanox/mlx4/en_tx.c   |   4 +-
  drivers/net/ethernet/mellanox/mlx4/mlx4_en.h |  28 +--
  3 files changed, 129 insertions(+), 253 deletions(-)

Thanks,
Tariq