Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

On 18.09.26 08:06, jiabin deng wrote:

Hi Dragos,

Thank you for pointing us to this fix.

Our original test used Ubuntu kernel 7.0.0-30-generic, based on
v7.0.12, and did not include that change. We have now repeated the
test using Ubuntu's mainline build of Linux 7.2.6
(7.2.6-070206-generic), with its in-tree mlx5_core driver.

I checked the v7.2.6 source: it contains the small-frame guard from
commit e2466392a0b8496000e12181cb1ee1535eb0da25  In
mlx5e_handle_rx_cqe_mpwrq_shampo(), frames with
cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN set match = false and
flush = true [2].

The zero-filled corruption ending at the next 64-byte boundary in
GPU memory still reproduces with this kernel.

We kept the same sender and receiver data-checking logic, with
tcp-data-split on and rx-gro-hw on. One additional environment
change is that the NVIDIA GPU driver was updated from 595.71.05 to
595.91.07. The NIC firmware remains 32.43.2400 (MT_0000001117).

We repeated the same eight message sizes, from 32 KiB to 4 MiB,
with 50 iterations per size:

- All 400 iterations completed.
- 105 iterations passed; 295 contained corrupted data.
- 21,599 corrupted ranges were recorded, all zero-filled and
  ending at the next 64-byte boundary in GPU memory.
- For each size from 256 KiB through 4 MiB, all 50 iterations
  contained corruption.
- The full-message snapshots taken before and after
  cuFlushGPUDirectRDMAWrites() were identical in all iterations.

For example, in the first 32 KiB iteration, an 8-byte corrupted
range starts at offset 2147495416 bytes from the beginning of the
GPU buffer. This offset is 56 bytes into a 64-byte-aligned block,
so the eight zero bytes end exactly at the next 64-byte boundary.

These results come from the reproducer's runtime byte comparisons.

Could you suggest the next targeted checks or additional debug
information that would help narrow this down?

Did you also try CPU buffers? From the NIC perspective it shouldn't
be any different.

Could you share a program + reproduction steps?

Thanks,
Dragos