Re: ethtool fails to read some QSFP+ modules.

Sorry, I'm not subscribed to netdev and didn't get a CC so I'm trying to follow via the web archives
and I can't see who was CC'd.

On Mon, Jul 01, 2024 at 10:28:39AM +0300, Ido Schimmel wrote:

On Sun, Jun 30, 2024 at 01:27:07PM -0400, Dan Merillat wrote:

quoted

I was testing an older Kaiam XQX2502 40G-LR4 and ethtool -m failed with netlink error.  It's treating a failure to read
the optional page3 data as a hard failure.

This patch allows ethtool to read qsfp modules that don't implement the voltage/temperature alarm data.

Thanks for the report and the patch. Krzysztof Olędzki reported the same
issue earlier this year:
https://lore.kernel.org/netdev/9e757616-0396-4573-9ea9-3cb5ef5c901a@ans.pl/ (local)

Krzysztof, are you going to submit the ethtool and mlx4 patches?

quoted

From 3144fbfc08fbfb90ecda4848fc9356bde8933d4a Mon Sep 17 00:00:00 2001
From: Dan Merillat <redacted>
Date: Sun, 30 Jun 2024 13:11:51 -0400
Subject: [PATCH] Some qsfp modules do not support page 3

Tested on an older Kaiam XQX2502 40G-LR4 module.
ethtool -m aborts with netlink error due to page 3
not existing on the module. Ignore the error and
leave map->page_03h NULL.

User space only tries to read this page because the module advertised it
as supported. It is more likely that the NIC driver does not return all
the pages. Which driver is it?

Same as Kyrzysztof, a connectx3 board.  I'm using mlx4_core/en in the stock 6.9.3 kernel.

Raw dump:
# ethtool -m enp65s0d1 raw on | hexdump -C 
00000000  0d 05 00 0f 00 00 00 00  00 44 44 00 00 44 44 00  |.........DD..DD.|
00000010  00 00 00 00 00 00 30 a4  00 00 82 3b 00 00 00 00  |......0....;....|
00000020  00 00 00 10 00 11 00 31  00 00 36 ff 36 ff 36 ff  |.......1..6.6.6.|
00000030  36 83 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |6...............|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 03 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  0d 80 07 02 00 00 00 00  00 00 00 00 67 00 02 00  |............g...|
00000090  00 00 00 40 4b 41 49 41  4d 20 43 4f 52 50 20 20  |...@KAIAM CORP  |
000000a0  20 20 20 20 00 14 ed e4  58 51 58 32 35 30 32 20  |    ....XQX2502 |
000000b0  20 20 20 20 20 20 20 20  31 41 66 58 05 14 46 14  |        1AfX..F.|
000000c0  00 00 00 92 4b 44 37 30  32 30 31 31 35 39 20 20  |....KD70201159  |
000000d0  20 20 20 20 31 37 30 32  30 31 30 30 08 00 00 0d  |    17020100....|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100


The ioctl path handles it properly when ethtool is built with --disable-netlink, since it checks the
total buffer size returned by the kernel and does not set map->page_03h if the size does not match.

Status byte 2, bit 2 is 'Upper memory flat or paged.   0x1b: Flat memory.  0x0b: Paging (at least upper
page 0x03h supported)'

SFF8636_STATUS_FLAT_MEMORY would be clearer, as page_00h[STATUS_2] & PAGE_3_PRESENT is really a
negative test.

I confirmed that status 2 is 0x00, which would indicate the module supports paged memory.

So either this cheap module is non-compliant or there's a problem in the driver/firmware for
connectx3/3pro cards.  Either way, it would be better to warn about the non-compliance and still
display  the optical information gathered rather than a fatal error that shows nothing.  If it can
also be fixed in the mlx4_core driver to properly read page 3, that would be better, but that
will require someone with more knowledge of mellanox/nVidia internals than I have.

If anyone wants to test with a different adapter, these xqx2502 modules are dirt cheap on ebay, only $4 when 
I bought mine a few weeks ago.