On 11/21/22 19:17, Vladimir Oltean wrote:

On Mon, Nov 21, 2022 at 05:42:44PM -0500, Sean Anderson wrote:

quoted

Are you certain this is the cause of the issue? It's also possible that
there is some errata for the PCS which is causing the issue. I have
gotten no review/feedback from NXP regarding the phylink conversion
(aside from acks for the cleanups).

Erratum which does what out of the ordinary? Your description of the
hardware failure seems consistent with the most plausible explanation
that doesn't involve any bugs.

Well, I don't have a setup which doesn't require in-band AN, so I can't
say one way or the other where the problem lies. To me, the Lynx PCS is
just as opaque as the phy.

If you enable C37/SGMII AN in the PCS (of the PHY or of the MAC) and AN
does not complete (because it's not enabled on the other end), that
system side of the link remains down. Which you don't see when you
operate in MLO_AN_PHY mode, because phylink only considers the PCS link
state in MLO_AN_INBAND mode. So this is why you see the link as up but
it doesn't work.

Actually, I checked the PCS manually in phy mode, and the link was up.
I expected it to be down, so this was a bit surprising to me.

To confirm whether I'm right or wrong, there's a separate SERDES
Interrupt Status Register at page 0xde1 offset 0x12, whose bit 4 is
"SERDES link status change" and bit 0 is "SERDES auto-negotiation error".
These bits should both be set when you double-read them (regardless of
IRQ enable I think) when your link is down with MLO_AN_PHY, but should
be cleared with MLO_AN_INBAND.

This register is always 0s for me...

quoted

This is used for SGMII to RGMII bridge mode (figure 4). It doesn't seem
to contain useful information for UTP mode (figure 1).

So it would seem. It was a hasty read last time, sorry. Re-reading, the
field says that when it's set, the SGMII code word being transmitted is
"selected by the register" SGMII ANAR. And in the SGMII ANLPAR, you can
see what the MAC said.

... possibly because of this.

That said, ANLPAR is 0x4001 (all reserved bits) when we use in-band:

[    8.191146] RTL8211F Gigabit Ethernet 0x0000000001afc000:04: INER=0000 INSR=0000 ANARSEL=0000 ANAR=0050 ANLPAR=4001

but all zeros without:

[   11.263245] RTL8211F Gigabit Ethernet 0x0000000001afc000:04: INER=0000 INSR=0000 ANARSEL=0000 ANAR=0050 ANLPAR=0000

It's all 1s when using RGMII. These bits are reserved, so it's not that
interesting, but maybe these registers are not as useless as they seem.

Of course, it doesn't say what happens when the bit for software-driven
SGMII autoneg is *not* set, if the process can be at all bypassed.
I suppose now that it can't, otherwise the ANLPAR register could also be
writable over MDIO, they would have likely reused at least partly the
same mechanisms.

quoted

quoted

+	ret = phy_read_paged(phydev, 0xd08, RTL8211FS_SGMII_ANARSEL);

That said, you have to use the "Indirect access method" to access this
register (per section 8.5). This is something like

#define RTL8211F_IAAR				0x1b
#define RTL8211F_IADR				0x1c

#define RTL8211F_IAAR_PAGE			GENMASK(15, 4)
#define RTL8211F_IAAR_REG			GENMASK(3, 1)
#define INDIRECT_ADDRESS(page, reg) \
	(FIELD_PREP(RTL8211F_IAAR_PAGE, page) | \
	 FIELD_PREP(RTL8211F_IAAR_REG, reg - 16))

	ret = phy_write_paged(phydev, 0xa43, RTL8211F_IAAR,
			      INDIRECT_ADDRESS(0xd08, RTL8211FS_SGMII_ANARSEL));
	if (ret < 0)
		return ret;

	ret = phy_read_paged(phydev, 0xa43, RTL8211F_IADR);
	if (ret < 0)
		return ret;

I dumped the rest of the serdes registers using this method, but I
didn't see anything interesting (all defaults).

I'm _really_ not sure where you got the "Indirect access method" via
registers 0x1b/0x1c from.

Huh. Looks like this is a second case of differing datasheets. Mine is
revision 1.8 dated 2021-04-21. The documentation for indirect access was
added in revision 1.7 dated 2020-07-08. Although it seems like the
SERDES registers were also added in this revision, so maybe you just
missed this section?

My datasheet for RTL8211FS doesn't show
offsets 0x1b and 0x1c in page 0xa43.

Neither does mine. These registers are only documented by reference from
section 8.5. They also aren't named, so the above defines are my own
coinage.

Additionally, I cross-checked with
other registers that are accessed by the driver (like the Interrupt
Enable Register), and the driver access procedure -
phy_write_paged(phydev, 0xa42, RTL821x_INER, val) - seems to be pretty
much in line with what my datasheet shows.

| The SERDES related registers should be read and written through indirect
| access method. The registers include Page 0xdc0 to Page 0xdcf and Page
| 0xde0 to Page 0xdf0.

Each register accessed this way also has

| Note: This register requires indirect access.

below the register table.

quoted

I think it would be better to just return PHY_AN_INBAND_ON when using
SGMII.

Well, of course hardcoding PHY_AN_INBAND_ON in the driver is on the
table, if it isn't possible to alter this setting to the best of our
knowledge (or if it's implausible that someone modified it). And this
seems more and more like the case.

I meant something like

	if (interface == PHY_INTERFACE_MODE_SGMII)
		return PHY_AN_INBAND_ON;

	return PHY_AN_INBAND_UNKNOWN;

Although for RGMII, in-band status is (per MIICR2):

- Enabled by default
- Disablable
- Optional

So maybe we should do (PHY_AN_INBAND_ON | PHY_AN_INBAND_OFF) in that
case. That said, RGMII in-band is not supported by phylink (yet).

--Sean