RE: [PATCH net-next 1/2] rtnetlink: Add new RTM_GETEECSTATE message to get SyncE status

-----Original Message-----
From: Jakub Kicinski <kuba@kernel.org>
Sent: Wednesday, September 8, 2021 6:21 PM
Subject: Re: [PATCH net-next 1/2] rtnetlink: Add new RTM_GETEECSTATE
message to get SyncE status

On Wed, 8 Sep 2021 08:03:35 +0000 Machnikowski, Maciej wrote:

quoted

quoted

quoted

Yep! Yet let's go one step at a time. I believe once we have the basics

(EEC

quoted

quoted

quoted

monitoring and recovered clock configuration) we'll be able to

implement

quoted

quoted

quoted

a basic functionality - and add bells and whistles later on, as there are

more

quoted

quoted

quoted

capabilities that we could support in SW.

The set API may shape how the get API looks. We need a minimal viable
API where the whole control part of it is not "firmware or proprietary
tools take care of that".

Do you have public docs on how the whole solution works?

The best reference would be my netdev 0x15 tutorial:
https://netdevconf.info/0x15/session.html?Introduction-to-time-

synchronization-over-Ethernet

quoted

The SyncE API considerations starts ~54:00, but basically we need API for:
- Controlling the lane to pin mapping for clock recovery
- Check the EEC/DPLL state and see what's the source of reference

frequency

quoted

(in more advanced deployments)
- control additional input and output pins (GNSS input, external inputs,

recovered

quoted

  frequency reference)

Thanks, good presentation! I haven't seen much in terms of system
design, at the level similar to the Broadcom whitepaper you shared.

See the "drawing" below.
 

quoted

quoted

quoted

I believe this is the state-of-art: here's the Broadcom public one
https://docs.broadcom.com/doc/1211168567832, I believe Marvel
has similar solution. But would also be happy to hear others.

Interesting. That reveals the need for also marking the backup
(/secondary) clock.

That's optional, but useful. And here's where we need a feedback
on which port/lane is currently used, as the switch may be automatic

What do you mean by optional? How does the user know if there is
fallback or not? Is it that Intel is intending to support only
devices with up to 2 ports and the second port is always the
backup (apologies for speculating).

That will be a part of pin config API. It needs to give info about the number
of supported pins that the PHY/MAC can use as recovered clock outputs.
Once the pin is assigned to the lane the recovered clock (divided or not)
will appear on the configured PHY/MAC pin and EEC will be able to
use it. If there is more than 1 available - they will have some priority
assigned to them that will be known to the EEC/board designer.

And we plan to support devices that only comes with 1 recovered clock
output as well.
 

quoted

quoted

Have you seen any docs on how systems with discreet PHY ASICs mux
the clocks?

Yes - unfortunately they are not public :(

Mumble, mumble. Ido, Florian - any devices within your purview which
would support SyncE?

OK Found one that's public: 
https://www.marvell.com/content/dam/marvell/en/public-collateral/transceivers/marvell-phys-transceivers-alaska-c-88x5113-datasheet.pdf
see Fig. 23 and chapter 3.11  for details, but conceptually it's similar.
 

quoted

quoted

quoted

Ethernet IP/PHY usually outputs a divided clock signal (since it's
easier to route) derived from the RX clock.
The DPLL connectivity is vendor-specific, as you can use it to connect
some external signals, but you can as well just care about relying
the SyncE clock and only allow recovering it and passing along
the QL info when your EEC is locked. That's why I backed up from
a full DPLL implementation in favor of a more generic EEC clock.

What is an ECC clock? To me the PLL state in the Ethernet port is the
state of the recovered clock. enum if_eec_state has values like
holdover which seem to be more applicable to the "system wide" PLL.

EEC is Ethernet Equipment Clock. In most cases this will be a DPLL, but that's
not mandatory and I believe it may be different is switches where
you only need to drive all ports TX from a single frequency source. In this
case the DPLL can be embedded in the multiport PHY,

quoted

Let me ask this - if one port is training the link and the other one has
the lock and is the source - what state will be reported for each port?

In this case the port that has the lock source will report the lock and
the EEC_SRC_PORT flag. The port that trains the link will show the
lock without the flag and once it completes the training sequence it will
use the EEC's frequency to transmit the data so that the next hop is able
to synchronize its EEC to the incoming RX frequency

Alright, I don't like that. It feels like you're attaching one object's
information (ECC) to other objects (ports), and repeating it. Prof
Goczyła and dr Landowska would not be proud.

Hahaha - did not expect them to pop up here :)
It's true, but the problem is that they depend on each other. The path is:

Lane0
------------- |\  Pin0     RefN   ____
------------- | |-----------------|         |      synced clk
...               | |-----------------| EEC  |------------------
------------- |/ PinN     RefM|____ |
Lane N      MUX

To get the full info a port needs to know the EEC state and which lane is used
as a source (or rather - my lane or any other).
The lane -> Pin mapping is buried in the PHY/MAC, but the source of frequency
is in the EEC.
What's even more - the Pin->Ref mapping is board specific.

The viable solutions are:
- Limit to the proposed "I drive the clock" vs "Someone drives it" and assume the
   Driver returns all info
- return the EEC Ref index, figure out which pin is connected to it and then check
  which MAC/PHY lane that drives it.

I assume option one is easy to implement and keep in the future even if we
finally move to option 2 once we define EEC/DPLL subsystem.

In future #1 can take the lock information from the DPLL subsystem, but
will also enable simple deployments that won't expose the whole DPLL, 
like a filter PLL embedded in a multiport PHY that will only work for
SyncE in which case this API will only touch a single component.
 

quoted

quoted

quoted

The Time IP is again relative and vendor-specific. If SyncE is deployed
alongside PTP it will most likely be tightly coupled, but if you only
care about having a frequency source - it's not mandatory and it can be
as well in the PHY IP.

I would not think having just the freq is very useful.

This depends on the deployment. There are couple popular frequencies
Most popular are 2,048 kHz, 10 MHz and 64 kHz. There are many
deployments that only require frequency sync without the phase
and/or time. I.e. if you deploy frequency division duplex you only need the
frequency reference, and the higher frequency you have - the faster you

can

quoted

lock to it.