Re: [RFC PATCH net-next 2/8] ethtool: Add ability to reset transceiver modules

On Mon, Aug 09, 2021 at 09:13:47PM +0200, Andrew Lunn wrote:

On Mon, Aug 09, 2021 at 01:21:46PM +0300, Ido Schimmel wrote:

quoted

From: Ido Schimmel <idosch@nvidia.com>

Add a new ethtool message, 'ETHTOOL_MSG_MODULE_RESET_ACT', which allows
user space to request a reset of transceiver modules. A successful reset
results in a notification being emitted to user space in the form of a
'ETHTOOL_MSG_MODULE_RESET_NTF' message.

Reset can be performed by either asserting the relevant hardware signal
("Reset" in CMIS / "ResetL" in SFF-8636) or by writing to the relevant
reset bit in the module's EEPROM (page 00h, byte 26, bit 3 in CMIS /
page 00h, byte 93, bit 7 in SFF-8636).

Reset is useful in order to allow a module to transition out of a fault
state. From section 6.3.2.12 in CMIS 5.0: "Except for a power cycle, the
only exit path from the ModuleFault state is to perform a module reset
by taking an action that causes the ResetS transition signal to become
TRUE (see Table 6-11)".

To avoid changes to the operational state of the device, reset can only
be performed when the device is administratively down.

Example usage:

 # ethtool --reset-module swp11
 netlink error: Cannot reset module when port is administratively up
 netlink error: Invalid argument

 # ip link set dev swp11 down

 # ethtool --reset-module swp11

Monitor notifications:

 $ ethtool --monitor
 listening...

 Module reset done for swp11

Again, i'm wondering, why is user space doing the reset? Can you think
of any other piece of hardware where Linux relies on user space
performing a reset before the kernel can properly use it?

How long does a reset take? Table 10-1 says the reset pulse must be
10uS and table 10-2 says the reset should not take longer than
2000ms.

Takes about 1.5ms to get an ACK on the reset request and another few
seconds to ensure module is in a valid operational state (will remove
RTNL in next version).

So maybe reset it on ifup if it is in a bad state?

We can have multiple ports (split) using the same module and in CMIS
each data path is controlled by a different state machine. Given the
complexity of these modules and possible faults, it is possible to
imagine a situation in which a few ports are fine and the rest are
unable to obtain a carrier.

Resetting the module on ifup of swp1s0 is not intuitive and it shouldn't
affect other split ports (e.g., swp1s1). With the dedicated reset
command we have the ability to enforce all the required restrictions
from the start instead of changing the behavior of existing commands.

I assume the driver/firmware is monitoring the SFP and if it goes into
a state which requires a reset it indicates carrier down? Wasn't there
some patches which added link down reasons? It would make sense to add
enum ethtool_link_ext_substate_sfp_fault? You can then use ethtool to
see what state the module is in, and a down/ip should reset it?

I will look into extending the interface with more reasons and parse the
CMIS ModuleFaultCause (see table 8-15) in ethtool(8).