SyncE - Synchronous Ethernet is defined in ITU-T Rec. G.8264
(https://www.itu.int/rec/T-REC-G.8264)
SyncE allows synchronizing the frequency of ethernet PHY clock signal
(the frequency used to send the data onto wire), to some reference
clock signal.
Multiple reference clock sources can be available. PHY ports recover
the frequency at which the transmitter sent the data on the RX side.
Alternatively, we can use external sources like 1PPS GPS, etc.
This patch series introduces basic interfaces for communication
with a SyncE capable device.
The first part of the interface allows acquiring the synchronization
state of DPLL (Digital Phase Locked Loop). DPLL LOCKED state means
that the frequency generated by it is locked to the input frequency.
As a result, PHYs connected to it are synchronized to the chosen input
frequency signal.
The second part can be used to select the port from which the clock
gets recovered. Each PHY chip can have multiple pins on which the
recovered clock can be propagated. For example, a SyncE-capable PHY
can recover the carrier frequency of the first port, divide it
internally, and output it as a reference clock on PIN 0.
When such a signal is enabled, the DPLL can LOCK to the frequency
recovered on PIN 0.
Next steps:
- Add CONFIG_SYNCE definition into Kconfig
- Add more configuration interfaces. Aiming at devlink, since this
would be device-wide configuration
Arkadiusz Kubalewski (7):
ptp: Add interface for acquiring DPLL state
selftests/ptp: Add usage of PTP_DPLL_GETSTATE ioctl in testptp
ice: add get_dpll_state ptp interface usage
net: add ioctl interface for recover reference clock on netdev
selftests/net: Add test app for SIOC{S|G}SYNCE
ice: add SIOC{S|G}SYNCE interface usage to recover reference signal
ice: add sysfs interface to configure PHY recovered reference signal
.../net/ethernet/intel/ice/ice_adminq_cmd.h | 62 +++++
drivers/net/ethernet/intel/ice/ice_common.c | 101 ++++++++
drivers/net/ethernet/intel/ice/ice_common.h | 9 +
drivers/net/ethernet/intel/ice/ice_main.c | 4 +
drivers/net/ethernet/intel/ice/ice_ptp.c | 234 +++++++++++++++++-
drivers/net/ethernet/intel/ice/ice_ptp.h | 9 +
drivers/net/ethernet/intel/ice/ice_ptp_hw.h | 6 +
drivers/ptp/ptp_chardev.c | 15 ++
drivers/ptp/ptp_clockmatrix.h | 12 -
drivers/ptp/ptp_private.h | 2 +
drivers/ptp/ptp_sysfs.c | 48 ++++
include/linux/ptp_clock_kernel.h | 9 +
include/uapi/linux/net_synce.h | 21 ++
include/uapi/linux/ptp_clock.h | 27 ++
include/uapi/linux/sockios.h | 4 +
net/core/dev_ioctl.c | 6 +-
tools/testing/selftests/net/Makefile | 1 +
tools/testing/selftests/net/phy_ref_clk.c | 138 +++++++++++
tools/testing/selftests/ptp/testptp.c | 27 +-
19 files changed, 720 insertions(+), 15 deletions(-)
create mode 100644 include/uapi/linux/net_synce.h
create mode 100644 tools/testing/selftests/net/phy_ref_clk.c
base-commit: aba1e4adb54e020d3ca85a4df3ef0f8febe87548
--
2.24.0
Previously there was no common interface for monitoring
synchronization state of Digital Phase Locked Loop.
Add interface through PTP ioctl subsystem for tools,
as well as sysfs human-friendly part of the interface.
enum dpll_state moved to uapi definition, it is required to
have common definition of DPLL states in uapi.
Add new callback function, must be implemented by ptp
enabled driver in order to get the state of dpll.
Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
---
drivers/ptp/ptp_chardev.c | 15 ++++++++++
drivers/ptp/ptp_clockmatrix.h | 12 --------
drivers/ptp/ptp_private.h | 2 ++
drivers/ptp/ptp_sysfs.c | 48 ++++++++++++++++++++++++++++++++
include/linux/ptp_clock_kernel.h | 9 ++++++
include/uapi/linux/ptp_clock.h | 27 ++++++++++++++++++
6 files changed, 101 insertions(+), 12 deletions(-)
@@ -106,6 +106,14 @@ int ptp_open(struct posix_clock *pc, fmode_t fmode)return0;}+intptp_get_dpll_state(structptp_clock*ptp,structptp_dpll_state*ds)+{+if(!ptp->info->get_dpll_state)+return-EOPNOTSUPP;++returnptp->info->get_dpll_state(ptp->info,ds);+}+longptp_ioctl(structposix_clock*pc,unsignedintcmd,unsignedlongarg){structptp_clock*ptp=container_of(pc,structptp_clock,clock);
@@ -119,6 +127,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)structptp_clock_capscaps;structptp_clock_time*pct;unsignedinti,pin_index;+structptp_dpll_stateds;structptp_pin_descpd;structtimespec64ts;intenable,err=0;
@@ -141,6 +141,7 @@ static void usage(char *progname)" -S set the system time from the ptp clock time\n"" -t val shift the ptp clock time by 'val' seconds\n"" -T val set the ptp clock time to 'val' seconds\n"+" -u get list of available DPLLs and their state values"" -z test combinations of rising/falling external time stamp flags\n",progname);}
@@ -156,6 +157,7 @@ int main(int argc, char *argv[])structtimextx;structptp_clock_time*pct;structptp_sys_offset*sysoff;+structptp_dpll_state*ds;char*progname;unsignedinti;
@@ -177,6 +179,7 @@ int main(int argc, char *argv[])intpps=-1;intseconds=0;intsettime=0;+intdpll_state=0;int64_tt1,t2,tp;int64_tinterval,offset;
@@ -186,7 +189,7 @@ int main(int argc, char *argv[])progname=strrchr(argv[0],'/');progname=progname?1+progname:argv[0];-while(EOF!=(c=getopt(argc,argv,"cd:e:f:ghH:i:k:lL:p:P:sSt:T:w:z"))){+while(EOF!=(c=getopt(argc,argv,"cd:e:f:ghH:i:k:lL:p:P:sSt:T:uw:z"))){switch(c){case'c':capabilities=1;
@@ -242,6 +245,9 @@ int main(int argc, char *argv[])settime=3;seconds=atoi(optarg);break;+case'u':+dpll_state=1;+break;case'w':pulsewidth=atoi(optarg);break;
@@ -506,6 +512,25 @@ int main(int argc, char *argv[])free(sysoff);}+if(dpll_state){+ds=calloc(1,sizeof(*ds));+if(!ds){+perror("calloc");+return-1;+}++if(ioctl(fd,PTP_DPLL_GETSTATE,ds))+perror("PTP_DPLL_GETSTATE");+else+puts("get dpll state request okay");++printf("dpll state:\n");+for(i=0;i<ds->dpll_num;i++)+printf("dpll id:%i state:%u\n",i,ds->state[i]);++free(ds);+}+close(fd);return0;}
Add new Admin Queue Command definitions for
getting status of Digital Phase Locked Loop.
Implement new part of ptp interface for getting
state of Digital Phase Locked Loop.
Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
---
.../net/ethernet/intel/ice/ice_adminq_cmd.h | 33 +++++++++++++++
drivers/net/ethernet/intel/ice/ice_common.c | 37 +++++++++++++++++
drivers/net/ethernet/intel/ice/ice_common.h | 3 ++
drivers/net/ethernet/intel/ice/ice_ptp.c | 40 +++++++++++++++++++
drivers/net/ethernet/intel/ice/ice_ptp_hw.h | 6 +++
5 files changed, 119 insertions(+)
@@ -112,6 +112,12 @@ struct ice_cgu_pll_params_e822 {externconststructice_cgu_pll_params_e822e822_cgu_params[NUM_ICE_TIME_REF_FREQ];+enumice_e810t_cgu_dpll{+ICE_CGU_DPLL_SYNCE,+ICE_CGU_DPLL_PTP,+ICE_CGU_DPLL_MAX+};+/* Table of constants related to possible TIME_REF sources */externconststructice_time_ref_info_e822e822_time_ref[NUM_ICE_TIME_REF_FREQ];
Previously there was no similar interface. It is required for
configuration of Synchronous Ethernet (SyncE).
User must be able to enable or disable propagation of its
PHY recovered clock signal onto available output pin.
This allows the signal to be used as frequency reference signal
for different hardware chips.
Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
---
include/uapi/linux/net_synce.h | 21 +++++++++++++++++++++
include/uapi/linux/sockios.h | 4 ++++
net/core/dev_ioctl.c | 6 +++++-
3 files changed, 30 insertions(+), 1 deletion(-)
create mode 100644 include/uapi/linux/net_synce.h
@@ -153,6 +153,10 @@#define SIOCSHWTSTAMP 0x89b0 /* set and get config */#define SIOCGHWTSTAMP 0x89b1 /* get config */+/* synchronous ethernet config per physical function */+#define SIOCSSYNCE 0x89c0 /* set and get config */+#define SIOCGSYNCE 0x89c1 /* get config */+/* Device private ioctl calls *//*
@@ -406,7 +406,9 @@ static int dev_ifsioc(struct net *net, struct ifreq *ifr, void __user *data,cmd==SIOCGMIIREG||cmd==SIOCSMIIREG||cmd==SIOCSHWTSTAMP||-cmd==SIOCGHWTSTAMP){+cmd==SIOCGHWTSTAMP||+cmd==SIOCSSYNCE||+cmd==SIOCGSYNCE){err=dev_eth_ioctl(dev,ifr,cmd);}elseif(cmd==SIOCBONDENSLAVE||cmd==SIOCBONDRELEASE||
@@ -577,6 +579,7 @@ int dev_ioctl(struct net *net, unsigned int cmd, struct ifreq *ifr,caseSIOCBRADDIF:caseSIOCBRDELIF:caseSIOCSHWTSTAMP:+caseSIOCSSYNCE:if(!ns_capable(net->user_ns,CAP_NET_ADMIN))return-EPERM;fallthrough;
@@ -605,6 +608,7 @@ int dev_ioctl(struct net *net, unsigned int cmd, struct ifreq *ifr,default:if(cmd==SIOCWANDEV||cmd==SIOCGHWTSTAMP||+cmd==SIOCGSYNCE||(cmd>=SIOCDEVPRIVATE&&cmd<=SIOCDEVPRIVATE+15)){dev_load(net,ifr->ifr_name);
Test usage of new netdev IOCTLs: SIOCSSYNCE and SIOCGSYNCE.
Allow set or get the netdev driver configuration
of PHY reference clock on output pins.
Signed-off-by: Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>
---
tools/testing/selftests/net/Makefile | 1 +
tools/testing/selftests/net/phy_ref_clk.c | 138 ++++++++++++++++++++++
2 files changed, 139 insertions(+)
create mode 100644 tools/testing/selftests/net/phy_ref_clk.c
@@ -0,0 +1,138 @@+// SPDX-License-Identifier: GPL-2.0-only+/*+*Thisprogramallowstotestbehaviorofnetdevaftersending+*SyncErelatedioctl:SIOCGSYNCEandSIOCSSYNCE.+*SIOCGSYNCE-wasdesignedtocheckhowoutputpinonPHYport+*wasconfigured.+*SIOCSSYNCE-wasdesignedtoconfigure(enableordisable)+*oneofthepins,thatPHYcanpropagateitsrecoveredclock+*signalonto.+*+*Copyright(C)2021IntelCorporation.+*Author:ArkadiuszKubalewski<arkadiusz.kubalewski@intel.com>+*/++#include<stdio.h>+#include<stdlib.h>+#include<errno.h>+#include<string.h>+#include<unistd.h>+#include<sys/socket.h>+#include<sys/ioctl.h>+#include<arpa/inet.h>+#include<net/if.h>++#include<asm/types.h>+#include<linux/sockios.h>+#include<linux/net_synce.h>++staticvoidusage(constchar*error)+{+if(error)+printf("invalid: %s\n\n",error);+printf("phy_ref_clk <interface> <pin_id> [enable]\n\n"+"Enable or disable phy-recovered reference clock signal on given output pin.\n"+"Depending on HW configuration, phy recovered clock may be enabled\n"+"or disabled on one of output pins which are at hardware's disposal\n\n"+"Params:\n"+" <interface> - name of netdev implementing SIOCGSYNCE and SIOCSSYNCE\n"+" <pin_id> - pin on which clock recovered from PHY shall be propagated\n"+" (0-X), X - number of output pins at HW disposal\n"+" In case no other arguments are given, ask the driver\n"+" for the current config of recovered clock on the interface.\n\n"+" [enable] - if pin shal be enabled or disabled (0/1)\n\n");+exit(1);+}++staticintget_ref_clk(constchar*ifname,__u8pin)+{+structsynce_ref_clk_cfgref_clk;+structifreqifdata;+intsd,rc;++if(!ifname||*ifname=='\0')+return-1;++memset(&ifdata,0,sizeof(ifdata));++strncpy(ifdata.ifr_name,ifname,IFNAMSIZ);++sd=socket(PF_INET,SOCK_DGRAM,IPPROTO_IP);+if(sd<0){+printf("socket failed\n");+return-1;+}++ref_clk.pin_id=pin;+ifdata.ifr_data=(void*)&ref_clk;++rc=ioctl(sd,SIOCGSYNCE,(char*)&ifdata);+close(sd);+if(rc!=0){+printf("ioctl(SIOCGSYNCE) failed\n");+returnrc;+}+printf("GET: pin %u is %s\n",+ref_clk.pin_id,ref_clk.enable?"enabled":"disabled");++return0;+}++staticintset_ref_clk(constchar*ifname,__u8pin,_Boolenable)+{+structsynce_ref_clk_cfgref_clk;+structifreqifdata;+intsd,rc;++if(!ifname||*ifname=='\0')+return-1;++memset(&ifdata,0,sizeof(ifdata));++strcpy(ifdata.ifr_name,ifname);++sd=socket(PF_INET,SOCK_DGRAM,IPPROTO_IP);+if(sd<0){+printf("socket failed\n");+return-1;+}++ref_clk.pin_id=pin;+ref_clk.enable=enable;+ifdata.ifr_data=(void*)&ref_clk;++rc=ioctl(sd,SIOCSSYNCE,(char*)&ifdata);+close(sd);+if(rc!=0){+printf("ioctl(SIOCSSYNCE) failed\n");+returnrc;+}+printf("SET: pin %u is %s",+ref_clk.pin_id,ref_clk.enable?"enabled":"disabled");++return0;+}++intmain(intargc,char**argv)+{+_Boolenable;+__u8pin;+intret;++if(argc>4||argc<3)+usage("argument count");++ret=sscanf(argv[2],"%u",&pin);+if(ret!=1)+usage(argv[2]);++if(argc==3){+ret=get_ref_clk(argv[1],pin);+}elseif(argc==4){+ret=sscanf(argv[3],"%u",&enable);+if(ret!=1)+usage(argv[3]);+ret=set_ref_clk(argv[1],pin,enable);+}+returnret;+}
@@ -24,6 +24,12 @@ struct ice_perout_channel {u64start_time;};+enumice_phy_rclk_pins{+ICE_C827_RCLKA_PIN,/* SCL pin */+ICE_C827_RCLKB_PIN,/* SDA pin */+ICE_C827_RCLK_PINS_NUM/* number of pins */+};+/* The ice hardware captures Tx hardware timestamps in the PHY. The timestamp*isstoredinabufferofregisters.Dependingonthespecifichardware,*thisbuffermightbesharedacrossmultiplePHYports.
@@ -153,6 +153,10 @@#define SIOCSHWTSTAMP 0x89b0 /* set and get config */#define SIOCGHWTSTAMP 0x89b1 /* get config */+/* synchronous ethernet config per physical function */+#define SIOCSSYNCE 0x89c0 /* set and get config */+#define SIOCGSYNCE 0x89c1 /* get config */
I understand that these are traditionally using the old-style 16-bit
numbers, but is there any reason to keep doing that rather than
making them modern like this?
#define SIOCSSYNCE _IOWR(0x89, 0xc0, struct synce_ref_clk_cfg)
/* set and get config */
#define SIOCGSYNCE _IOR(0x89, 0xc1, struct synce_ref_clk_cfg)
/* get config */
Arnd
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-16 23:54:08
On Mon, Aug 16, 2021 at 06:07:11PM +0200, Arkadiusz Kubalewski wrote:
Previously there was no common interface for monitoring
synchronization state of Digital Phase Locked Loop.
Add interface through PTP ioctl subsystem for tools,
as well as sysfs human-friendly part of the interface.
enum dpll_state moved to uapi definition, it is required to
have common definition of DPLL states in uapi.
This has nothing to do with PTP, right?
So why isn't it a RTNL feature instead?
Thanks,
Richard
From: Machnikowski, Maciej <hidden> Date: 2021-08-17 09:41:54
quoted
Previously there was no common interface for monitoring
synchronization state of Digital Phase Locked Loop.
Add interface through PTP ioctl subsystem for tools, as well as sysfs
human-friendly part of the interface.
enum dpll_state moved to uapi definition, it is required to have
common definition of DPLL states in uapi.
This has nothing to do with PTP, right?
So why isn't it a RTNL feature instead?
Thanks,
Richard
The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL on Network adapters, in most cases, drive the PTP timer.
Having access to it in the PTP subsystem is beneficial, as Telco standards, like G.8275.1/2, require a different behavior depending on the SyncE availability and state.
Multiport adapters use a single PLL to drive all ports. If we add the PLL interface to the PTP device - a tool that would implement support for Telco standards would have a single source of information about the presence and state of physical sync.
Moreover, it'll open the path for implementing PLL state-aware ext_ts that would generate events only when the PLL device is locked to the incoming signals improving the quality of external TS events. I.e. an external PLL can be used to monitor incoming 1PPS signal and disable event generation when its frequency drifts off 1Hz by preset margins.
If we bind it to the Network port, a tool would need to find the port that owns the PLL and read the state from a different place, which sounds suboptimal. Additionally we'll lose ability to rely on external HW to monitor external TS events.
Regards
Maciek
@@ -153,6 +153,10 @@#define SIOCSHWTSTAMP 0x89b0 /* set and get config */#define SIOCGHWTSTAMP 0x89b1 /* get config */+/* synchronous ethernet config per physical function */+#define SIOCSSYNCE 0x89c0 /* set and get config */+#define SIOCGSYNCE 0x89c1 /* get config */
I understand that these are traditionally using the old-style 16-bit
numbers, but is there any reason to keep doing that rather than
making them modern like this?
Personally I would try to keep it one way, just for consistency,
but you might be right - making it modern way is better option.
If no other objections to this comment I am going to change it according to
Arnd's suggestion in next patch.
#define SIOCSSYNCE _IOWR(0x89, 0xc0, struct synce_ref_clk_cfg)
/* set and get config */
#define SIOCGSYNCE _IOR(0x89, 0xc1, struct synce_ref_clk_cfg)
/* get config */
Arnd
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-18 17:03:12
On Tue, Aug 17, 2021 at 09:41:49AM +0000, Machnikowski, Maciej wrote:
The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL on Network adapters, in most cases, drive the PTP timer.
So what? The logic in the HW has nothing to do with the proper user
space interfaces. For example, we have SO_TIMESTAMPING and PHC as
separate APIs, even though HW devices often implement both.
Having access to it in the PTP subsystem is beneficial, as Telco
standards, like G.8275.1/2, require a different behavior depending
on the SyncE availability and state.
Right, but this does say anything about the API.
Multiport adapters use a single PLL to drive all ports. If we add
the PLL interface to the PTP device - a tool that would implement
support for Telco standards would have a single source of
information about the presence and state of physical sync.
Not really. Nothing guarantees a sane mapping from MAC to PHC. In
many systems, there a many of each.
Moreover, it'll open the path for implementing PLL state-aware
ext_ts that would generate events only when the PLL device is locked
to the incoming signals improving the quality of external TS
events. I.e. an external PLL can be used to monitor incoming 1PPS
signal and disable event generation when its frequency drifts off
1Hz by preset margins.
I don't see how this prevents using RTNL (or something else) as the
API for the SyncE configuration.
If we bind it to the Network port, a tool would need to find the
port that owns the PLL and read the state from a different place,
which sounds suboptimal.
This is exactly how it works in ptpl4 today. Some information comes
from the PHC, some from RTNL (link state), some from ethtool
(phc-interface mapping and time stamping abilities).
Additionally we'll lose ability to rely on external HW to monitor
external TS events.
Sorry, I can't see that at all.
Please do NOT tack this stuff onto the PHC subsystem just because you
can.
Thanks,
Richard
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-18 17:05:43
On Mon, Aug 16, 2021 at 06:07:10PM +0200, Arkadiusz Kubalewski wrote:
The second part can be used to select the port from which the clock
gets recovered. Each PHY chip can have multiple pins on which the
recovered clock can be propagated. For example, a SyncE-capable PHY
can recover the carrier frequency of the first port, divide it
internally, and output it as a reference clock on PIN 0.
This really sounds like its own thing, and not a PHC at all.
Next steps:
- Add CONFIG_SYNCE definition into Kconfig
- Add more configuration interfaces. Aiming at devlink, since this
would be device-wide configuration
As a first step, finding an appropriate kernel/user space API would be
needed.
Thanks,
Richard
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-18 17:08:55
On Mon, Aug 16, 2021 at 06:07:10PM +0200, Arkadiusz Kubalewski wrote:
Multiple reference clock sources can be available. PHY ports recover
the frequency at which the transmitter sent the data on the RX side.
Alternatively, we can use external sources like 1PPS GPS, etc.
There is a generic PHY subsystem (drivers/phy) used by USB, PCIe, CAN,
and so on. Why not use that?
Thanks,
Richard
From: "Keller, Jacob E" <jacob.e.keller@intel.com> Date: 2021-08-18 18:14:20
-----Original Message-----
From: Intel-wired-lan <redacted> On Behalf Of
Richard Cochran
Sent: Wednesday, August 18, 2021 10:03 AM
To: Machnikowski, Maciej <redacted>
Cc: cong.wang@bytedance.com; arnd@arndb.de; gustavoars@kernel.org;
netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
colin.king@canonical.com; intel-wired-lan@lists.osuosl.org; nikolay@nvidia.com;
linux-kselftest@vger.kernel.org; kuba@kernel.org; shuah@kernel.org;
davem@davemloft.net
Subject: Re: [Intel-wired-lan] [RFC net-next 1/7] ptp: Add interface for acquiring
DPLL state
On Tue, Aug 17, 2021 at 09:41:49AM +0000, Machnikowski, Maciej wrote:
quoted
The logic behind adding the DPLL state to the PTP subsystem is that SyncE DPLL
on Network adapters, in most cases, drive the PTP timer.
So what? The logic in the HW has nothing to do with the proper user
space interfaces. For example, we have SO_TIMESTAMPING and PHC as
separate APIs, even though HW devices often implement both.
quoted
Having access to it in the PTP subsystem is beneficial, as Telco
standards, like G.8275.1/2, require a different behavior depending
on the SyncE availability and state.
Right, but this does say anything about the API.
quoted
Multiport adapters use a single PLL to drive all ports. If we add
the PLL interface to the PTP device - a tool that would implement
support for Telco standards would have a single source of
information about the presence and state of physical sync.
Not really. Nothing guarantees a sane mapping from MAC to PHC. In
many systems, there a many of each.
Well, I think the point of placing it in the PTP subsystem is that there is a sane mapping between PHC <-> DPLL. There's only one DPLL for the PHC.
Thanks,
Jake
From: Machnikowski, Maciej <hidden> Date: 2021-08-18 22:36:15
-----Original Message-----
From: Richard Cochran <richardcochran@gmail.com>
Sent: Wednesday, August 18, 2021 7:03 PM
To: Machnikowski, Maciej <redacted>
Cc: Kubalewski, Arkadiusz <arkadiusz.kubalewski@intel.com>; linux-
kernel@vger.kernel.org; intel-wired-lan@lists.osuosl.org;
netdev@vger.kernel.org; linux-kselftest@vger.kernel.org; Brandeburg,
Jesse [off-list ref]; Nguyen, Anthony L
[off-list ref]; davem@davemloft.net; kuba@kernel.org;
shuah@kernel.org; arnd@arndb.de; nikolay@nvidia.com;
cong.wang@bytedance.com; colin.king@canonical.com;
gustavoars@kernel.org
Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state
quoted
Additionally we'll lose ability to rely on external HW to monitor
external TS events.
Sorry, I can't see that at all.
Please do NOT tack this stuff onto the PHC subsystem just because you can.
Thanks,
Richard
OK, Let's take a step back and forget about SyncE.
A PTP clock is a device that has a phase and a frequency, and its frequency can be adjusted using API calls.
On the other hand, there's the physical side of the PTP clock. The PTP clock can run on cheap quartz, on the CSAC, or the PLL.
The first two of them will get a clock signal from a passive device, but in the PLL case, it'll get it from an active one.
If it runs on an active PLL device, you get another place that adjusts the frequency of your PTP clock.
No matter what source you use as a reference for it - CSAC, SyncE, or GNSS receiver.
Adding the PLL state to the PTP subsystem is just another indicator of the state of the PTP clock.
The upper layer can use it, or ignored it, but it fits into the timer subsystem, as the time generated by the PTP on top will be derived from the frequency generated by the PLL.
And it is applicable to both a PHC and a completely separate implementation of timer, like the one that's present in the Time Card .
Regards
Maciek
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-19 15:34:24
On Wed, Aug 18, 2021 at 10:36:03PM +0000, Machnikowski, Maciej wrote:
OK, Let's take a step back and forget about SyncE.
Ahem, the title of this series is:
[RFC net-next 0/7] Add basic SyncE interfaces
I'd be happy to see support for configuring SyncE.
But I guess this series is about something totally different.
Thanks,
Richard
From: Machnikowski, Maciej <hidden> Date: 2021-08-19 15:40:35
-----Original Message-----
From: Richard Cochran <richardcochran@gmail.com>
Sent: Thursday, August 19, 2021 5:34 PM
To: Machnikowski, Maciej <redacted>
Cc: Kubalewski, Arkadiusz <arkadiusz.kubalewski@intel.com>; linux-
kernel@vger.kernel.org; intel-wired-lan@lists.osuosl.org;
netdev@vger.kernel.org; linux-kselftest@vger.kernel.org; Brandeburg,
Jesse [off-list ref]; Nguyen, Anthony L
[off-list ref]; davem@davemloft.net; kuba@kernel.org;
shuah@kernel.org; arnd@arndb.de; nikolay@nvidia.com;
cong.wang@bytedance.com; colin.king@canonical.com;
gustavoars@kernel.org; Bross, Kevin [off-list ref]; Stanton,
Kevin B [off-list ref]; Ahmad Byagowi [off-list ref]
Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state
On Wed, Aug 18, 2021 at 10:36:03PM +0000, Machnikowski, Maciej wrote:
quoted
OK, Let's take a step back and forget about SyncE.
Ahem, the title of this series is:
[RFC net-next 0/7] Add basic SyncE interfaces
I'd be happy to see support for configuring SyncE.
But I guess this series is about something totally different.
Thanks,
Richard
If it helps we'd be happy to separate that in 2 separate RFCs.
This was squashed together under SyncE support umbrella to show one of the use cases, but PTP changes are more generic and cover all PTP clocks that use DPLL for the physical clock generation.
Regards
Maciek
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-20 15:55:45
On Thu, Aug 19, 2021 at 03:40:22PM +0000, Machnikowski, Maciej wrote:
If it helps we'd be happy to separate that in 2 separate RFCs.
It would help me if you could explain the connection. I have a
totally different understanding of SyncE which I explained here:
https://lore.kernel.org/netdev/20150317161128.GA8793@localhost.localdomain/
So you need to implement two things, one in kernel and one in user
space.
1. Control bits according to IEEE 802.3 Section 40.5.2 as Ethtool or RTNL.
2. User space Ethernet Synchronization Messaging Channel (ESMC)
service according to IEEE 802.3ay
The PHY should be automatically controlled by #1.
As I said before, none of this belongs in the PHC subsystem.
Thanks,
Richard
From: Machnikowski, Maciej <hidden> Date: 2021-08-20 18:30:18
-----Original Message-----
From: Richard Cochran <richardcochran@gmail.com>
Sent: Friday, August 20, 2021 5:56 PM
To: Machnikowski, Maciej <redacted>
Cc: Kubalewski, Arkadiusz <arkadiusz.kubalewski@intel.com>; linux-
kernel@vger.kernel.org; intel-wired-lan@lists.osuosl.org;
netdev@vger.kernel.org; linux-kselftest@vger.kernel.org; Brandeburg,
Jesse [off-list ref]; Nguyen, Anthony L
[off-list ref]; davem@davemloft.net; kuba@kernel.org;
shuah@kernel.org; arnd@arndb.de; nikolay@nvidia.com;
cong.wang@bytedance.com; colin.king@canonical.com;
gustavoars@kernel.org; Bross, Kevin [off-list ref]; Stanton,
Kevin B [off-list ref]; Ahmad Byagowi [off-list ref]
Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state
1. Control bits according to IEEE 802.3 Section 40.5.2 as Ethtool or RTNL.
2. User space Ethernet Synchronization Messaging Channel (ESMC)
service according to IEEE 802.3ay
The PHY should be automatically controlled by #1.
As I said before, none of this belongs in the PHC subsystem.
Thanks,
Richard
Sure!
I did a talk at netDev 0x15 covering SyncE - you can refer to the slides for more detailed info, and hopefully the recording will be available soon as well:
https://netdevconf.info/0x15/session.html?Introduction-to-time-synchronization-over-Ethernet
At its core - SyncE requires 2 parts (see slide 22/23)
- SyncE capable PHY
- the external DPLL
The SyncE capable PHY is a PHY that can recover the physical clock, at which the data symbols are transferred, (usually) divide it and output it to the external PLL. It can also redirect the recovered and divided clock to more than one pin.
Since the 40.5.2 is not applicable to higher-speed ethernet which don't use auto-negotiation, but rather the link training sequence where the RX side always syncs its clock to the TX side.
The external DPLL tunes the frequency generated by a crystal to the frequency recovered by the PHY, and drives the outputs.
On the other end - the SyncE PHY uses the clock generated by the DPLL to transmit the data to the next element.
So to be able to control SyncE we need 2 interfaces:
- Interface to enable the recovered clock output at the given pin
- interface to monitor the DPLL to see if the clock that we got is valid, or not.
If it comes to ESMC (G.8264) messages, SyncE itself can run in 2 modes (slides 29/30 will give you more details):
- QL-Disabled - with no ESMC messages - it base on the local information from the PLL to make all decisions
- QL-Enabled - that adds ESMC and quality message transfer between the nodes.
Additionally, the SyncE DPLL can be synchronized to the external sources, like a 1PPS or a 10M signal from the GNSS.
That's why the RFC proposes 2 interfaces:
- one for enabling redirected clock on a selected pin of the PHY
- one for the physical frequency lock of the DPLL
The connection with the PTP subsystem is that in most use cases I heard about SyncE is used as a physical frequency syntonization for PTP clocks.
Hence adding a DPLL monitoring there would solve 2 issues at the same time - monitoring of a GNSS-syntonized PTP clock and the SyncE syntonized one and would make a single point to monitor by the upper layer applications.
Let me know if that makes more sense now. We could add a separate SyncE and separate PTP DPLL monitoring interfaces, but in most cases they will point to the same device.
Regards
Maciek
These slides are very clear and nicely done!
( And they also confirm that (ab)using the PHC chardev ioctl for the DPLL
stuff is the wrong interface ;^)
The SyncE capable PHY is a PHY that can recover the physical clock,
at which the data symbols are transferred, (usually) divide it and
output it to the external PLL. It can also redirect the recovered
and divided clock to more than one pin.
Right, and as your slides show so clearly, the DPLL is connected to
the PHY, not to the time stamping unit with the PTP clock.
Since the 40.5.2 is not applicable to higher-speed ethernet which
don't use auto-negotiation, but rather the link training sequence
where the RX side always syncs its clock to the TX side.
I really want an interface that will also work with Gigabit and even
100 Megabit like the PHYTER (which does support SyncE).
The external DPLL tunes the frequency generated by a crystal to the frequency recovered by the PHY, and drives the outputs.
On the other end - the SyncE PHY uses the clock generated by the DPLL to transmit the data to the next element.
So I guess that this is an implementation detail of the higher speed PHYs.
That's why the RFC proposes 2 interfaces:
- one for enabling redirected clock on a selected pin of the PHY
- one for the physical frequency lock of the DPLL
The connection with the PTP subsystem is that in most use cases I
heard about SyncE is used as a physical frequency syntonization for
PTP clocks.
As your slides correctly show, SyncE is about distributing frequency
and not about Phase/ToD. Of course you can combine SyncE with PTP to
get both, provided that you disable frequency adjustment in the PTP
software stack (in linuxptp, this is the "nullf" servo). But SyncE in
fact predates PTP, and it can and should be configurable even on
interfaces that lack PHC altogther (or on kernels without PHC
enabled).
Let me know if that makes more sense now. We could add a separate
SyncE and separate PTP DPLL monitoring interfaces, but in most cases
they will point to the same device.
This is just a coincidence of the device you are working with. The
kernel really needs an interface that works with all kind of hardware
setups. Imagine a computer with discrete MACs with HW time
stamping/PHC and discrete PHYs with SyncE support. The PHC driver
won't have any connection to the PHY+DPLL.
Your API must be on the interface/MAC, with the possibility being
handled directly by the MAC driver (for integrated devices) or calling
into the PHY layers (phylib, phylink, and drivers/phy).
If you need a DPLL monitoring interface for your card, it ought to go
through the network interface to the MAC/PHY driver and then to the
DPLL itself. That way, it will work with different types of hardware.
Thanks,
Richard
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-22 02:31:04
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote:
Since the 40.5.2 is not applicable to higher-speed ethernet which
don't use auto-negotiation, but rather the link training sequence
where the RX side always syncs its clock to the TX side.
By "the RX side always syncs its clock to the TX side" do you mean the
RX channel synchronizes to the link partner's TX channel?
Wow, that brings back the 100 megabit scheme I guess. That's cool,
because the same basic idea applies to the PHYTER then.
Still we are doing to need a way for user space to query the HW
topology to discover whether a given ports may be syntonized from a
second port. I don't think your pin selection thing works unless user
space can tell what the pins are connected to.
Thanks,
Richard
From: Machnikowski, Maciej <hidden> Date: 2021-08-23 08:29:26
-----Original Message-----
From: Richard Cochran <richardcochran@gmail.com>
Sent: Sunday, August 22, 2021 4:31 AM
Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote:
quoted
Since the 40.5.2 is not applicable to higher-speed ethernet which
don't use auto-negotiation, but rather the link training sequence
where the RX side always syncs its clock to the TX side.
By "the RX side always syncs its clock to the TX side" do you mean the RX
channel synchronizes to the link partner's TX channel?
Wow, that brings back the 100 megabit scheme I guess. That's cool, because
the same basic idea applies to the PHYTER then.
Yes! Sounds very similar! :)
Still we are doing to need a way for user space to query the HW topology to
discover whether a given ports may be syntonized from a second port. I
don't think your pin selection thing works unless user space can tell what the
pins are connected to.
Thanks,
Richard
And a good catch! I'll update the RFC to add the query functionality and move the SyncE logic/pins to the netdev subsystem.
Thanks
From: Richard Cochran <richardcochran@gmail.com> Date: 2021-08-30 21:06:22
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote:
So to be able to control SyncE we need 2 interfaces:
- Interface to enable the recovered clock output at the given pin
- interface to monitor the DPLL to see if the clock that we got is valid, or not.
If it comes to ESMC (G.8264) messages, SyncE itself can run in 2 modes (slides 29/30 will give you more details):
- QL-Disabled - with no ESMC messages - it base on the local information from the PLL to make all decisions
- QL-Enabled - that adds ESMC and quality message transfer between the nodes.
How do you get the QL codes from this?
+enum if_eec_state {
+ IF_EEC_STATE_INVALID = 0,
+ IF_EEC_STATE_FREERUN,
+ IF_EEC_STATE_LOCKACQ,
+ IF_EEC_STATE_LOCKREC,
+ IF_EEC_STATE_LOCKED,
+ IF_EEC_STATE_HOLDOVER,
+ IF_EEC_STATE_OPEN_LOOP,
+ __IF_EEC_STATE_MAX,
+};
Thanks,
Richard
From: Machnikowski, Maciej <hidden> Date: 2021-08-31 09:29:34
-----Original Message-----
From: Richard Cochran <richardcochran@gmail.com>
Sent: Monday, August 30, 2021 11:06 PM
To: Machnikowski, Maciej <redacted>
Subject: Re: [RFC net-next 1/7] ptp: Add interface for acquiring DPLL state
On Fri, Aug 20, 2021 at 06:30:02PM +0000, Machnikowski, Maciej wrote:
quoted
So to be able to control SyncE we need 2 interfaces:
- Interface to enable the recovered clock output at the given pin
- interface to monitor the DPLL to see if the clock that we got is valid, or
not.
quoted
If it comes to ESMC (G.8264) messages, SyncE itself can run in 2 modes
(slides 29/30 will give you more details):
quoted
- QL-Disabled - with no ESMC messages - it base on the local information
from the PLL to make all decisions
quoted
- QL-Enabled - that adds ESMC and quality message transfer between the
nodes.
How do you get the QL codes from this?
+enum if_eec_state {
+ IF_EEC_STATE_INVALID = 0,
+ IF_EEC_STATE_FREERUN,
+ IF_EEC_STATE_LOCKACQ,
+ IF_EEC_STATE_LOCKREC,
+ IF_EEC_STATE_LOCKED,
+ IF_EEC_STATE_HOLDOVER,
+ IF_EEC_STATE_OPEN_LOOP,
+ __IF_EEC_STATE_MAX,
+};
This structure is for monitoring the lock state - or in other words - quality
of incoming sync signal.
The Locked state here means that the frequency used for transmitting the
data is syntonized with the input one. If something goes wrong, like the
frequency you recover from the link goes beyond the specified range or
the external signal is lost, the QL level will change accordingly.
The application layer running on top of this API needs to get the proper
QL level from the config file (just like the clockClass in PTP) and broadcast
it when the state is locked and switch to QL-DNU when you get out of
the lock state and expire preset hw-dependent holdover clock.
Also, if you are syntonizing to the SyncE clock you need to wait with
passing along QL-levels until the state reported by the EEC changes
to LOCKED.
Regards
Maciek