RE: DPAA Ethernet traffice troubles with Linux kernel

33 messages, 6 authors, 2018-02-05 · open the first message on its own page

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-15 16:38:28

-----Original Message-----
From: Linuxppc-dev [mailto:linuxppc-dev-
bounces+madalin.bucur=nxp.com@lists.ozlabs.org] On Behalf Of mad skateman
Sent: Wednesday, January 10, 2018 10:39 PM
To: linuxppc-dev@lists.ozlabs.org
Subject: DPAA Ethernet traffice troubles with Linux kernel

Hi linux devs,

Like mentioned in this thread
https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-January/167630.html
i also experience the exact same issues.
I am also trying to find out why the network traffic is not flowing
the way it should (out for example ).

My linux knowledge is very basic but i hope i can contribute anyway.

I am using the AmigaOne X5000 with a P5020
Most detailed technical information regarding this issue can be found
in the Thread by Jamie Krueger mentioned above.

In this screenshot, the ETH0 and ETH1 seem up and running (probed) ..
even due to the FSL_DPAA_MAC error messages that DMESG shows.
http://www.skateman.nl/wp-content/uploads/2018/01/Screenshot-at-2018-01-08-21_22_06_ETH_NIC_ERROR.png

http://www.skateman.nl/wp-content/uploads/2018/01/Screenshot-at-2018-01-08-22_16_28.png

I was able to use some tooling like ETHTOOL to adjust some settings
and check if the interface responded. This all seems fine.

Hope that someone can find a fix, so the Ethernet adapter can be used.

Thanks!!
Hi,

Please use text logs instead of pictures next time, it's easier to read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the issue
you encounter. Normally the unused interfaces should have status disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the device
back 2 back to a known good machine and determine what is broken - Rx/Tx?
Is there another software version that does work on these machines?

Madalin

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-15 16:59:24

On Thu, 1970-01-01 at 00:00 +0000, Madalin-cristian Bucur wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
-----Original Message-----
From: Linuxppc-dev [mailto:linuxppc-dev-
bounces+madalin.bucur=nxp.com@lists.ozlabs.org] On Behalf Of mad skateman
Sent: Wednesday, January 10, 2018 10:39 PM
To: linuxppc-dev@lists.ozlabs.org
Subject: DPAA Ethernet traffice troubles with Linux kernel

Hi linux devs,

Like mentioned in this thread
https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-January/167630.html
i also experience the exact same issues.
I am also trying to find out why the network traffic is not flowing
the way it should (out for example ).

My linux knowledge is very basic but i hope i can contribute anyway.

I am using the AmigaOne X5000 with a P5020
Most detailed technical information regarding this issue can be found
in the Thread by Jamie Krueger mentioned above.

In this screenshot, the ETH0 and ETH1 seem up and running (probed) ..
even due to the FSL_DPAA_MAC error messages that DMESG shows.
http://www.skateman.nl/wp-content/uploads/2018/01/Screenshot-at-2018-01-08-21_22_06_ETH_NIC_ERROR.png

http://www.skateman.nl/wp-content/uploads/2018/01/Screenshot-at-2018-01-08-22_16_28.png

I was able to use some tooling like ETHTOOL to adjust some settings
and check if the interface responded. This all seems fine.

Hope that someone can find a fix, so the Ethernet adapter can be used.

Thanks!!
Hi,

Please use text logs instead of pictures next time, it's easier to read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the issue
you encounter. Normally the unused interfaces should have status disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the device
back 2 back to a known good machine and determine what is broken - Rx/Tx?
Is there another software version that does work on these machines?
Hi, just saw this and thought of a small patch I just wrote for mdio bus, o idea
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
---
 drivers/of/of_mdio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 98258583abb0..76ff28a41dad 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -229,7 +229,8 @@ int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np)
 		else
 			rc = of_mdiobus_register_device(mdio, child, addr);
 		if (rc)
-			goto unregister;
+			pr_warn(FW_WARN
+				"%pOF: Failed to register MDIO device.\n", child);
 	}
 
 	if (!scanphys)
@@ -253,7 +254,8 @@ int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np)
 			if (of_mdiobus_child_is_phy(child)) {
 				rc = of_mdiobus_register_phy(mdio, child, addr);
 				if (rc)
-					goto unregister;
+					pr_warn(FW_WARN
+						"%pOF: Failed to register MDIO PHY.\n", child);
 			}
 		}
 	}
-- 
2.13.6

DPAA Ethernet traffice troubles with Linux kernel

From: Christian Zigotzky <hidden>
Date: 2018-01-15 19:03:55

Hi All,

I compiled the RC8 of kernel 4.15 with Joakim's patch for the AmigaOne 
X5000 today. Many thanks to Joakim for the mdio patch.

Please test it on your X5000.

Thanks,
Christian


On 15 January 2018 at 5:59PM, Joakim Tjernlund wrote:
quoted hunk
quoted
Hi,

Please use text logs instead of pictures next time, it's easier to read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the issue
you encounter. Normally the unused interfaces should have status disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the device
back 2 back to a known good machine and determine what is broken - Rx/Tx?
Is there another software version that does work on these machines?
Hi, just saw this and thought of a small patch I just wrote for mdio bus, o idea
if it is relevant but here goes:

 From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
---
  drivers/of/of_mdio.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 98258583abb0..76ff28a41dad 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -229,7 +229,8 @@ int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np)
  		else
  			rc = of_mdiobus_register_device(mdio, child, addr);
  		if (rc)
-			goto unregister;
+			pr_warn(FW_WARN
+				"%pOF: Failed to register MDIO device.\n", child);
  	}
  
  	if (!scanphys)
@@ -253,7 +254,8 @@ int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np)
  			if (of_mdiobus_child_is_phy(child)) {
  				rc = of_mdiobus_register_phy(mdio, child, addr);
  				if (rc)
-					goto unregister;
+					pr_warn(FW_WARN
+						"%pOF: Failed to register MDIO PHY.\n", child);
  			}
  		}
  	}

DPAA Ethernet traffice troubles with Linux kernel

From: Christian Zigotzky <hidden>
Date: 2018-01-15 19:10:00

Sorry, I have forgotten the download link. Please test it with the DPAA 
Ethernet.
Hi All,

I compiled the RC8 of kernel 4.15 with Joakim's patch for the AmigaOne 
X5000 today. Many thanks to Joakim for the mdio patch.

Download: http://www.xenosoft.de/uImage-4.15-rc8_with_mdio_patch.tar.gz

Please test it on your X5000.

Thanks,
Christian


On 15 January 2018 at 5:59PM, Joakim Tjernlund wrote:
quoted
quoted
Hi,

Please use text logs instead of pictures next time, it's easier to 
read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the 
issue
you encounter. Normally the unused interfaces should have status 
disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the 
device
back 2 back to a known good machine and determine what is broken - 
Rx/Tx?
Is there another software version that does work on these machines?
Hi, just saw this and thought of a small patch I just wrote for mdio 
bus, o idea
if it is relevant but here goes:

 From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
---
  drivers/of/of_mdio.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 98258583abb0..76ff28a41dad 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -229,7 +229,8 @@ int of_mdiobus_register(struct mii_bus *mdio, 
struct device_node *np)
          else
              rc = of_mdiobus_register_device(mdio, child, addr);
          if (rc)
-            goto unregister;
+            pr_warn(FW_WARN
+                "%pOF: Failed to register MDIO device.\n", child);
      }
        if (!scanphys)
@@ -253,7 +254,8 @@ int of_mdiobus_register(struct mii_bus *mdio, 
struct device_node *np)
              if (of_mdiobus_child_is_phy(child)) {
                  rc = of_mdiobus_register_phy(mdio, child, addr);
                  if (rc)
-                    goto unregister;
+                    pr_warn(FW_WARN
+                        "%pOF: Failed to register MDIO PHY.\n", child);
              }
          }
      }

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-15 20:21:14

Hi All,

I have been testing with the rc8 kernel as Christian sugested.

At this moment still the same issues..

Here some of my finding...

A dump of dmesg regarding the FSL_DPAA:

[    4.597949] libphy: Fixed MDIO Bus: probed
[    4.599095] libphy: Freescale PowerQUICC MII Bus: probed
[    4.609108] libphy: Freescale PowerQUICC MII Bus: probed
[    4.609995] libphy: Freescale PowerQUICC MII Bus: probed
[    4.610689] libphy: Freescale PowerQUICC MII Bus: probed
[    4.611623] libphy: Freescale PowerQUICC MII Bus: probed
[    4.612470] libphy: Freescale XGMAC MDIO Bus: probed
[    4.617710] fsl_dpaa_mac ffe4e6000.ethernet: FMan dTSEC version: 0x08240101
[    4.618335] fsl_dpaa_mac ffe4e6000.ethernet: FMan MAC address:
00:04:9f:01:02:03
[    4.618653] fsl_dpaa_mac ffe4e8000.ethernet: FMan dTSEC version: 0x08240101
[    4.619042] fsl_dpaa_mac ffe4e8000.ethernet: FMan MAC address:
00:04:9f:01:02:04
[    4.619294] fsl_dpaa_mac ffe4e0000.ethernet:
of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@e0000) failed
[    4.619505] fsl_dpaa_mac: probe of ffe4e0000.ethernet failed with error -22
[    4.619673] fsl_dpaa_mac ffe4e2000.ethernet:
of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@e2000) failed
[    4.619878] fsl_dpaa_mac: probe of ffe4e2000.ethernet failed with error -22
[    4.620048] fsl_dpaa_mac ffe4e4000.ethernet:
of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@e4000) failed
[    4.620258] fsl_dpaa_mac: probe of ffe4e4000.ethernet failed with error -22
[    4.625078] fsl_dpaa_mac ffe4f0000.ethernet:
of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@f0000) failed
[    4.630001] fsl_dpaa_mac: probe of ffe4f0000.ethernet failed with error -22
[    4.637015] fsl_dpaa_mac ffe4e6000.ethernet eth0: Probed interface eth0
[    4.643933] fsl_dpaa_mac ffe4e8000.ethernet eth1: Probed interface eth1

This all seems correct! The probed mac`s are the one`s as put in Uboot:
Ethaddr and Eth1addr

The Eth0 and Eth1 are also probed.

When eth0 is brought up... it is unable to get an ip adress from the
dhcp server.

eth0      Link encap:Ethernet  HWaddr 00:04:9f:01:02:03
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:510 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:102556 (102.5 KB)
          Memory:fe4e6000-fe4e6fff

When using mii-tool for a double check:

mii-tool -l eth0
eth0: negotiated 1000baseT-FD flow-control, link ok

and when using ethtool

ethtool eth0
Settings for eth0:
	Supported ports: [ MII ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Half 1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 3
	Transceiver: external
	Auto-negotiation: on
	Current message level: 0x00000037 (55)
			       drv probe link ifdown ifup
	Link detected: yes

RX packets always stay 0 ?

Will continiue to get more info...







On 1/15/18, Christian Zigotzky [off-list ref] wrote:
Sorry, I have forgotten the download link. Please test it with the DPAA
Ethernet.
quoted
Hi All,

I compiled the RC8 of kernel 4.15 with Joakim's patch for the AmigaOne
X5000 today. Many thanks to Joakim for the mdio patch.

Download: http://www.xenosoft.de/uImage-4.15-rc8_with_mdio_patch.tar.gz

Please test it on your X5000.

Thanks,
Christian


On 15 January 2018 at 5:59PM, Joakim Tjernlund wrote:
quoted
quoted
Hi,

Please use text logs instead of pictures next time, it's easier to
read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the
issue
you encounter. Normally the unused interfaces should have status
disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the
device
back 2 back to a known good machine and determine what is broken -
Rx/Tx?
Is there another software version that does work on these machines?
Hi, just saw this and thought of a small patch I just wrote for mdio
bus, o idea
if it is relevant but here goes:

 From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
---
  drivers/of/of_mdio.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 98258583abb0..76ff28a41dad 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -229,7 +229,8 @@ int of_mdiobus_register(struct mii_bus *mdio,
struct device_node *np)
          else
              rc = of_mdiobus_register_device(mdio, child, addr);
          if (rc)
-            goto unregister;
+            pr_warn(FW_WARN
+                "%pOF: Failed to register MDIO device.\n", child);
      }
        if (!scanphys)
@@ -253,7 +254,8 @@ int of_mdiobus_register(struct mii_bus *mdio,
struct device_node *np)
              if (of_mdiobus_child_is_phy(child)) {
                  rc = of_mdiobus_register_phy(mdio, child, addr);
                  if (rc)
-                    goto unregister;
+                    pr_warn(FW_WARN
+                        "%pOF: Failed to register MDIO PHY.\n", child);
              }
          }
      }

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-15 21:32:18

Some extra info:

When Ubuntu boots, Eth0 (192.168.22.44) is not brought up automaticly..

When i bring up eth0 by hand, its still not active..

root@X5000LNX:/home/skateman# ifconfig eth0 up
root@X5000LNX:/home/skateman# ping 192.168.22.44
connect: Network is unreachable

When i use mii-tool too Kick the tranceiver... it comes alive.. i can
ping the eth0 itself

root@X5000LNX:/home/skateman# mii-tool -R eth0
resetting the transceiver...
root@X5000LNX:/home/skateman# ping 192.168.22.44
PING 192.168.22.44 (192.168.22.44) 56(84) bytes of data.
64 bytes from 192.168.22.44: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 192.168.22.44: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 192.168.22.44: icmp_seq=3 ttl=64 time=0.047 ms
64 bytes from 192.168.22.44: icmp_seq=4 ttl=64 time=0.048 ms

eth0      Link encap:Ethernet  HWaddr 00:04:9f:01:02:03
          inet addr:192.168.22.44  Bcast:192.168.22.255  Mask:255.255.255.0
          inet6 addr: fe80::c84b:9f6b:f2f6:8933/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:7600 (7.6 KB)
          Memory:fe4e6000-fe4e6fff

Not realy sure why the Tranceiver is not brought up directly when linux starts..



On 1/15/18, Christian Zigotzky [off-list ref] wrote:
Sorry, I have forgotten the download link. Please test it with the DPAA
Ethernet.
quoted
Hi All,

I compiled the RC8 of kernel 4.15 with Joakim's patch for the AmigaOne
X5000 today. Many thanks to Joakim for the mdio patch.

Download: http://www.xenosoft.de/uImage-4.15-rc8_with_mdio_patch.tar.gz

Please test it on your X5000.

Thanks,
Christian


On 15 January 2018 at 5:59PM, Joakim Tjernlund wrote:
quoted
quoted
Hi,

Please use text logs instead of pictures next time, it's easier to
read.
The errors you see are related to missing MAC addresses for the unused
interfaces, you can ignore these are they are not relevant for the
issue
you encounter. Normally the unused interfaces should have status
disabled
in the device tree but there is not a big deal if they fail like that.
As I've advised Jamie on the other thread, please try to connect the
device
back 2 back to a known good machine and determine what is broken -
Rx/Tx?
Is there another software version that does work on these machines?
Hi, just saw this and thought of a small patch I just wrote for mdio
bus, o idea
if it is relevant but here goes:

 From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
---
  drivers/of/of_mdio.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 98258583abb0..76ff28a41dad 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -229,7 +229,8 @@ int of_mdiobus_register(struct mii_bus *mdio,
struct device_node *np)
          else
              rc = of_mdiobus_register_device(mdio, child, addr);
          if (rc)
-            goto unregister;
+            pr_warn(FW_WARN
+                "%pOF: Failed to register MDIO device.\n", child);
      }
        if (!scanphys)
@@ -253,7 +254,8 @@ int of_mdiobus_register(struct mii_bus *mdio,
struct device_node *np)
              if (of_mdiobus_child_is_phy(child)) {
                  rc = of_mdiobus_register_phy(mdio, child, addr);
                  if (rc)
-                    goto unregister;
+                    pr_warn(FW_WARN
+                        "%pOF: Failed to register MDIO PHY.\n", child);
              }
          }
      }

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-01-16 15:04:14

When i use mii-tool too Kick the tranceiver... it comes alive.. i can
ping the eth0 itself

root@X5000LNX:/home/skateman# mii-tool -R eth0
resetting the transceiver...
root@X5000LNX:/home/skateman# ping 192.168.22.44
PING 192.168.22.44 (192.168.22.44) 56(84) bytes of data.
64 bytes from 192.168.22.44: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 192.168.22.44: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 192.168.22.44: icmp_seq=3 ttl=64 time=0.047 ms
64 bytes from 192.168.22.44: icmp_seq=4 ttl=64 time=0.048 ms
What PHY driver are you using?

This smells a bit like an RGMII-ID problem.

     Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-01-16 15:16:47

Hi, just saw this and thought of a small patch I just wrote for mdio bus, o idea
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
Hi Joakim

You appear to be using an old kernel. Take a look at:

commit 95f566de0269a0c59fd6a737a147731302136429
Author: Madalin Bucur [off-list ref]
Date:   Tue Jan 9 14:43:34 2018 +0200

    of_mdio: avoid MDIO bus removal when a PHY is missing
    
    If one of the child devices is missing the of_mdiobus_register_phy()
    call will return -ENODEV. When a missing device is encountered the
    registration of the remaining PHYs is stopped and the MDIO bus will
    fail to register. Propagate all errors except ENODEV to avoid it.
    
    Signed-off-by: Madalin Bucur [off-list ref]
    Reviewed-by: Andrew Lunn [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]


    Andrew

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-16 17:07:59

-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Andrew Lunn
Sent: Tuesday, January 16, 2018 5:04 PM
To: mad skateman <redacted>
Cc: Christian Zigotzky <redacted>; Joakim Tjernlund
[off-list ref]; linuxppc-dev@lists.ozlabs.org; Madalin-
cristian Bucur [off-list ref]; netdev@vger.kernel.org
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel
quoted
When i use mii-tool too Kick the tranceiver... it comes alive.. i can
ping the eth0 itself

root@X5000LNX:/home/skateman# mii-tool -R eth0
resetting the transceiver...
root@X5000LNX:/home/skateman# ping 192.168.22.44
PING 192.168.22.44 (192.168.22.44) 56(84) bytes of data.
64 bytes from 192.168.22.44: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 192.168.22.44: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 192.168.22.44: icmp_seq=3 ttl=64 time=0.047 ms
64 bytes from 192.168.22.44: icmp_seq=4 ttl=64 time=0.048 ms
What PHY driver are you using?

This smells a bit like an RGMII-ID problem.

     Andrew
Hi Andrew,
From another thread[1] on the same topic:
I am not sure what PHY hardware/configuration you are using on the
DS and RDB platforms, but I can confirm that AmigaONE X5000/20
(Cyrus Motherboard with p5020 SoC), has dTSEC 4 and dTSEC 5
wired to two Micrel KSZ9021RN Gigabit Ethernet PHYs, using the
RGMII protocol.
[1] https://www.spinics.net/lists/netdev/msg478062.html

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-16 17:57:41

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
Hi, just saw this and thought of a small patch I just wrote for mdio bus, o idea
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
commit 95f566de0269a0c59fd6a737a147731302136429
Author: Madalin Bucur [off-list ref]
Date:   Tue Jan 9 14:43:34 2018 +0200

    of_mdio: avoid MDIO bus removal when a PHY is missing

    If one of the child devices is missing the of_mdiobus_register_phy()
    call will return -ENODEV. When a missing device is encountered the
    registration of the remaining PHYs is stopped and the MDIO bus will
    fail to register. Propagate all errors except ENODEV to avoid it.

    Signed-off-by: Madalin Bucur [off-list ref]
    Reviewed-by: Andrew Lunn [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]


    Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-16 18:16:24

Hi,

I have been looking deeper into my wireshark packet captures and found
something that could be helpfull.

I can see that the Ethernet NICS at least do something. ARP BROADCAST
traffic is seen.
But i also found some packets...which have LG BITs set to 1 .. when i think
0 should be the correct value..

This has something to do with the MAC adresses being non Authorative.. and
since whe can use Uboot and choose any Mac addr we want, this could make
sense.. More info about this
https://osqa-ask.wireshark.org/questions/59761/oui-lookup-tool-does-not-recognize-local-addresses

Logs like these appear: not the original.

Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Address: Broadcast (ff:ff:ff:ff:ff:ff)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this
is NOT the factory default)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)


Will try to get the picture more clear... but think about this..



On Tue, Jan 16, 2018 at 6:57 PM, Joakim Tjernlund <
Joakim.Tjernlund@infinera.com> wrote:
On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.
quoted
quoted
Hi, just saw this and thought of a small patch I just wrote for mdio
bus, o idea
quoted
quoted
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
quoted
commit 95f566de0269a0c59fd6a737a147731302136429
Author: Madalin Bucur [off-list ref]
Date:   Tue Jan 9 14:43:34 2018 +0200

    of_mdio: avoid MDIO bus removal when a PHY is missing

    If one of the child devices is missing the of_mdiobus_register_phy()
    call will return -ENODEV. When a missing device is encountered the
    registration of the remaining PHYs is stopped and the MDIO bus will
    fail to register. Propagate all errors except ENODEV to avoid it.

    Signed-off-by: Madalin Bucur [off-list ref]
    Reviewed-by: Andrew Lunn [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]


    Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-16 18:38:06

Hi,

I have been looking deeper into my wireshark packet captures and found
something that could be helpfull.

I can see using wireshark that the Ethernet NIC at least does something.
ARP BROADCAST traffic is seen.
But i also found some packets...which have LG BITs set to 1 .. when i think
0 should be the correct value..

This has something to do with the MAC adresses being locally administered
.. and since whe can use Uboot and choose any Mac addr we want, this could
make sense..

These types of Logs also apear in my Wireshark Capture files... ( these are
not my org. logs)

Ethernet II, Src: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8), Dst: Broadcast
(ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Address: Broadcast (ff:ff:ff:ff:ff:ff)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this
is NOT the factory default)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
Source: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8)
Address: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory
default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255
(255.255.255.255)

In the link below some similair Logs and problems regarding DHCP for
example.

Dave

https://www.microchip.com/forums/m/tm.aspx?m=956881&fp=1&p=2

On Tue, Jan 16, 2018 at 6:57 PM, Joakim Tjernlund <
Joakim.Tjernlund@infinera.com> wrote:
On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.
quoted
quoted
Hi, just saw this and thought of a small patch I just wrote for mdio
bus, o idea
quoted
quoted
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
quoted
commit 95f566de0269a0c59fd6a737a147731302136429
Author: Madalin Bucur [off-list ref]
Date:   Tue Jan 9 14:43:34 2018 +0200

    of_mdio: avoid MDIO bus removal when a PHY is missing

    If one of the child devices is missing the of_mdiobus_register_phy()
    call will return -ENODEV. When a missing device is encountered the
    registration of the remaining PHYs is stopped and the MDIO bus will
    fail to register. Propagate all errors except ENODEV to avoid it.

    Signed-off-by: Madalin Bucur [off-list ref]
    Reviewed-by: Andrew Lunn [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]


    Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-16 18:39:23

Hi,

I have been looking deeper into my wireshark packet captures and found
something that could be helpfull.

I can see using wireshark that the Ethernet NIC at least does something.
ARP BROADCAST traffic is seen.
But i also found some packets...which have LG BITs set to 1 .. when i think
0 should be the correct value..

This has something to do with the MAC adresses being locally administered
.. and since whe can use Uboot and choose any Mac addr we want, this could
make sense..

These types of Logs also apear in my Wireshark Capture files... ( these are
not my org. logs)

Ethernet II, Src: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8), Dst: Broadcast
(ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Address: Broadcast (ff:ff:ff:ff:ff:ff)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this
is NOT the factory default)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
Source: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8)
Address: Microchi_8f:c6:a8 (d8:80:39:8f:c6:a8)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory
default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255
(255.255.255.255)

In the link below some similair Logs and problems regarding DHCP for
example.

Dave

On Tue, Jan 16, 2018 at 6:57 PM, Joakim Tjernlund <
Joakim.Tjernlund@infinera.com> wrote:
On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.
quoted
quoted
Hi, just saw this and thought of a small patch I just wrote for mdio
bus, o idea
quoted
quoted
if it is relevant but here goes:

From fe0b98d54a79779482700676331b4d10a0f3cada Mon Sep 17 00:00:00 2001
From: Joakim Tjernlund <redacted>
Date: Sun, 14 Jan 2018 21:27:20 +0100
Subject: [PATCH] of_mdiobus_register: Continue after error

of_mdiobus_register unregister itself if one phy fails to register
which is bad for system having all its PHYs on the same MDIO bus.
Just log the error and continue with the remaining PHYs instead.

Signed-off-by: Joakim Tjernlund <redacted>
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
quoted
commit 95f566de0269a0c59fd6a737a147731302136429
Author: Madalin Bucur [off-list ref]
Date:   Tue Jan 9 14:43:34 2018 +0200

    of_mdio: avoid MDIO bus removal when a PHY is missing

    If one of the child devices is missing the of_mdiobus_register_phy()
    call will return -ENODEV. When a missing device is encountered the
    registration of the remaining PHYs is stopped and the MDIO bus will
    fail to register. Propagate all errors except ENODEV to avoid it.

    Signed-off-by: Madalin Bucur [off-list ref]
    Reviewed-by: Andrew Lunn [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]


    Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-01-16 20:53:47

quoted
You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old.
Development for 4.14 stopped somewhere around the beginning of
September. So there has been over 4 months of work since then.  We are
clearly interested in fixing bugs in that kernel, since it is the
current stable version. But when reporting bugs, please let is know if
the latest version of the network kernel,
it://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git has
the issue.
Seems like this patch hasn't been sent to 4.14.x.
If it fixes a bug for you, please request the fix is added to stable.

   Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Christian Zigotzky <hidden>
Date: 2018-01-17 05:54:48

FYI

Sent from my iPhone
On 17. Jan 2018, at 06:50, Christian Zigotzky [off-list ref] wrote:

Hi Skateman,

Fantastic! Many thanks for testing the RC8 of kernel 4.15 without PAMU support.

@All
Further information: http://forum.hyperion-entertainment.biz/viewtopic.php?f=58&p=43706#p43706

Cheers,
Christian

Sent from my iPhone
quoted
On 16. Jan 2018, at 23:05, mad skateman [off-list ref] wrote:

Fantastic Christian.. 

Your latest kernel makes the NIC work!!!

Few tweaks to be done... like the buffer space 

Brilliant!

quoted
On 16. Jan 2018, at 21:46, Christian Zigotzky [off-list ref] wrote:

FYI
quoted
On 16 January 2018 at 9:42PM, Christian Zigotzky wrote:
Hi All,

I compiled the RC8 of kernel 4.15 for the X5000 without PAMU support today.

Download: http://www.xenosoft.de/uImage_without_pamu.tar.gz

Please test it on your AmigaOne X5000.

Thanks,
Christian


On 16 January 2018 at 6:33PM, Madalin-cristian Bucur wrote:
quoted
quoted
The PAMU related errors may be relevant to the issue, if you have incorrect
settings you may have no traffic passing through. The PAMU configuration
should be made by the bootloader. Can you try to disable CONFIG_FSL_PAMU?

Madalin

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-17 11:47:41

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
quoted
You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old.
Development for 4.14 stopped somewhere around the beginning of
September. So there has been over 4 months of work since then.  We are
clearly interested in fixing bugs in that kernel, since it is the
current stable version. But when reporting bugs, please let is know if
the latest version of the network kernel,
it://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git has
the issue.
quoted
Seems like this patch hasn't been sent to 4.14.x.
If it fixes a bug for you, please request the fix is added to stable.
That doesn't work really, having users to hit the bug, debug it, fix it and then
find it fixed already in upstream, then specifically request it to be backported to stable. 
I don't need this fix to be backported, already got it. Someone else might though.

I would be interested in bug fixes upstream which fixes:

libphy: Freescale XGMAC MDIO Bus: probed
iommu: Adding device ffe488000.port to group 10
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e1000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe489000.port to group 22
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e3000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48a000.port to group 23
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e5000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48b000.port to group 24
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48c000.port to group 25
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e9000: Error while reading PHY0 reg at 3.3
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed

Every FMAN eth I/F with a fixed link fails mysteriously.
Custom board based on t1040rdb, been over the device tree and I cannot find any significant
changes.

 Jocke

Re: DPAA Ethernet traffice troubles with Linux kernel

From: mad skateman <hidden>
Date: 2018-01-17 12:06:45

After using the new compiled 4.14 rc8 kernel without PAMU support posted by
Christian Zigotzky the X5000 can use the Network interface with some minor
issues.

I had to give the Eth0 a manual IP due to not responding on DHCP requests.

I can ping my Gateway, and even DNS queries work..

But when i start Netsurf (or generate to much packets)... all traffic
dies.... due to No buffer space available..

64 bytes from 192.168.22.66: icmp_seq=81 ttl=255 time=0.317 ms
64 bytes from 192.168.22.66: icmp_seq=82 ttl=255 time=0.317 ms
64 bytes from 192.168.22.66: icmp_seq=83 ttl=255 time=0.314 ms
64 bytes from 192.168.22.66: icmp_seq=84 ttl=255 time=0.321 ms
64 bytes from 192.168.22.66: icmp_seq=85 ttl=255 time=0.323 ms
64 bytes from 192.168.22.66: icmp_seq=86 ttl=255 time=0.312 ms
64 bytes from 192.168.22.66: icmp_seq=87 ttl=255 time=0.331 ms
64 bytes from 192.168.22.66: icmp_seq=88 ttl=255 time=0.338 ms
64 bytes from 192.168.22.66: icmp_seq=89 ttl=255 time=0.334 ms
64 bytes from 192.168.22.66: icmp_seq=90 ttl=255 time=0.349 ms
64 bytes from 192.168.22.66: icmp_seq=91 ttl=255 time=0.324 ms
64 bytes from 192.168.22.66: icmp_seq=92 ttl=255 time=0.320 ms
64 bytes from 192.168.22.66: icmp_seq=93 ttl=255 time=0.320 ms
64 bytes from 192.168.22.66: icmp_seq=94 ttl=255 time=0.338 ms
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available

A workaround to keep Eth0 alive a bit longer.... is the following command

ip link set eth0 qlen 10000

We are making progress!!


On Wed, Jan 17, 2018 at 12:47 PM, Joakim Tjernlund <
Joakim.Tjernlund@infinera.com> wrote:
On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.
quoted
quoted
quoted
You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old.
Development for 4.14 stopped somewhere around the beginning of
September. So there has been over 4 months of work since then.  We are
clearly interested in fixing bugs in that kernel, since it is the
current stable version. But when reporting bugs, please let is know if
the latest version of the network kernel,
it://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git has
the issue.
quoted
Seems like this patch hasn't been sent to 4.14.x.
If it fixes a bug for you, please request the fix is added to stable.
That doesn't work really, having users to hit the bug, debug it, fix it
and then
find it fixed already in upstream, then specifically request it to be
backported to stable.
I don't need this fix to be backported, already got it. Someone else might
though.

I would be interested in bug fixes upstream which fixes:

libphy: Freescale XGMAC MDIO Bus: probed
iommu: Adding device ffe488000.port to group 10
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e1000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe489000.port to group 22
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e3000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48a000.port to group 23
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e5000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48b000.port to group 24
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48c000.port to group 25
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e9000: Error while reading PHY0 reg at 3.3
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed

Every FMAN eth I/F with a fixed link fails mysteriously.
Custom board based on t1040rdb, been over the device tree and I cannot
find any significant
changes.

 Jocke

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-01-17 13:43:44

That doesn't work really, having users to hit the bug, debug it, fix it and then
find it fixed already in upstream, then specifically request it to be backported to stable. 
I don't need this fix to be backported, already got it. Someone else might though.
The "someone else might though" is a big point of asking for it to
added to stable. The other reason is it means one less patch you need
to maintain in your build.
I would be interested in bug fixes upstream which fixes:
Did you try upstream? Does it give the same errors?

    Andrew

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-17 14:11:43

-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Tuesday, January 16, 2018 7:58 PM
To: andrew@lunn.ch
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
Hi Joakim,

These errors are issued when trying to probe the second time the same
MAC node. The issue was introduced by this commit:

commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.

Madalin

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-17 14:15:57

-----Original Message-----
From: Andrew Lunn [mailto:andrew@lunn.ch]
Sent: Wednesday, January 17, 2018 3:44 PM
To: Joakim Tjernlund <redacted>
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel
quoted
That doesn't work really, having users to hit the bug, debug it, fix it
and then
quoted
find it fixed already in upstream, then specifically request it to be
backported to stable.
quoted
I don't need this fix to be backported, already got it. Someone else
might though.

The "someone else might though" is a big point of asking for it to
added to stable. The other reason is it means one less patch you need
to maintain in your build.
I've sent that patch [1] for net but I guess the timing was wrong and
it was merged to net-next.
quoted
I would be interested in bug fixes upstream which fixes:
Did you try upstream? Does it give the same errors?

    Andrew
[1] https://patchwork.kernel.org/patch/10146119/

Madalin

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-17 14:25:24

-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Madalin-cristian Bucur
Sent: Wednesday, January 17, 2018 4:16 PM
To: Andrew Lunn <andrew@lunn.ch>; Joakim Tjernlund
[off-list ref]
Cc: linuxppc-dev@lists.ozlabs.org; netdev@vger.kernel.org;
madskateman@gmail.com; David S . Miller [off-list ref]
Subject: RE: DPAA Ethernet traffice troubles with Linux kernel
quoted
-----Original Message-----
From: Andrew Lunn [mailto:andrew@lunn.ch]
Sent: Wednesday, January 17, 2018 3:44 PM
To: Joakim Tjernlund <redacted>
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel
quoted
That doesn't work really, having users to hit the bug, debug it, fix
it
quoted
and then
quoted
find it fixed already in upstream, then specifically request it to be
backported to stable.
quoted
I don't need this fix to be backported, already got it. Someone else
might though.

The "someone else might though" is a big point of asking for it to
added to stable. The other reason is it means one less patch you need
to maintain in your build.
I've sent that patch [1] for net but I guess the timing was wrong and
it was merged to net-next.
quoted
quoted
I would be interested in bug fixes upstream which fixes:
Did you try upstream? Does it give the same errors?

    Andrew
[1] https://patchwork.kernel.org/patch/10146119/

Madalin
Hi Dave,

Can you please add the fix [1] to stable?

Thank you,
Madalin

RE: DPAA Ethernet traffice troubles with Linux kernel

From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: 2018-01-17 14:43:19

-----Original Message-----
From: Madalin-cristian Bucur
Sent: Wednesday, January 17, 2018 4:25 PM
To: David S . Miller <davem@davemloft.net>
Cc: linuxppc-dev@lists.ozlabs.org; netdev@vger.kernel.org;
madskateman@gmail.com; 'Madalin-cristian Bucur' [off-list ref];
Andrew Lunn [off-list ref]; Joakim Tjernlund
[off-list ref]
Subject: RE: DPAA Ethernet traffice troubles with Linux kernel
quoted
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Madalin-cristian Bucur
Sent: Wednesday, January 17, 2018 4:16 PM
To: Andrew Lunn <andrew@lunn.ch>; Joakim Tjernlund
[off-list ref]
Cc: linuxppc-dev@lists.ozlabs.org; netdev@vger.kernel.org;
madskateman@gmail.com; David S . Miller [off-list ref]
Subject: RE: DPAA Ethernet traffice troubles with Linux kernel
quoted
-----Original Message-----
From: Andrew Lunn [mailto:andrew@lunn.ch]
Sent: Wednesday, January 17, 2018 3:44 PM
To: Joakim Tjernlund <redacted>
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel
quoted
That doesn't work really, having users to hit the bug, debug it, fix
it
quoted
and then
quoted
find it fixed already in upstream, then specifically request it to
be
quoted
quoted
backported to stable.
quoted
I don't need this fix to be backported, already got it. Someone else
might though.

The "someone else might though" is a big point of asking for it to
added to stable. The other reason is it means one less patch you need
to maintain in your build.
I've sent that patch [1] for net but I guess the timing was wrong and
it was merged to net-next.
quoted
quoted
I would be interested in bug fixes upstream which fixes:
Did you try upstream? Does it give the same errors?

    Andrew
[1] https://patchwork.kernel.org/patch/10146119/

Madalin
Hi Dave,

Can you please add the fix [1] to stable?

Thank you,
Madalin
Sorry,

I've provided the wrong link towards the patch (v1 instead of v3),
here's the correct one:

https://patchwork.kernel.org/patch/10151969/

Madalin

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-17 15:00:12

On Thu, 1970-01-01 at 00:00 +0000, Madalin-cristian Bucur wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Tuesday, January 16, 2018 7:58 PM
To: andrew@lunn.ch
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
Hi Joakim,

These errors are issued when trying to probe the second time the same
MAC node. The issue was introduced by this commit:

commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.

Madalin
Ahh, now it starts to look better, reverting "fsl/man: Inherit parent device and of_node" on 4.14 gives:
libphy: Fixed MDIO Bus: probed
tun: Universal TUN/TAP device driver, 1.6
libphy: Freescale XGMAC MDIO Bus: probed
iommu: Adding device ffe488000.port to group 10
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e1000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe489000.port to group 22
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e3000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48a000.port to group 23
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e5000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48b000.port to group 24
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48c000.port to group 25
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e9000: Error while reading PHY0 reg at 3.3
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac ffe4e0000.ethernet: FMan MEMAC
fsl_mac ffe4e0000.ethernet: FMan MAC address: 00:06:9c:0b:06:1f
fsl_dpa dpaa-ethernet.0 eth0: Probed interface eth0
fsl_dpa dpaa-ethernet.1 eth1: Probed interface eth1
fsl_dpa dpaa-ethernet.2 eth2: Probed interface eth2
fsl_dpa dpaa-ethernet.3 eth3: Probed interface eth3
fsl_dpa dpaa-ethernet.4 eth4: Probed interface eth4

Still some minor errors: mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
but this is going the right way(I have not had a chance to try if they work due
to external modules not ported/ready yet)

The other patch series is still to be tested though but I already now wanted 
to stress the importance of getting all upstream fixes into stable, ASAP.
You now what they are, I have no idea.

Thanks 
       Jocke

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-18 09:04:20

On Thu, 1970-01-01 at 00:00 +0000, Joakim Tjernlund wrote:
On Thu, 1970-01-01 at 00:00 +0000, Madalin-cristian Bucur wrote:
quoted
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Tuesday, January 16, 2018 7:58 PM
To: andrew@lunn.ch
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
Hi Joakim,

These errors are issued when trying to probe the second time the same
MAC node. The issue was introduced by this commit:

commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.

Madalin
Ahh, now it starts to look better, reverting "fsl/man: Inherit parent device and of_node" on 4.14 gives:
libphy: Fixed MDIO Bus: probed
tun: Universal TUN/TAP device driver, 1.6
libphy: Freescale XGMAC MDIO Bus: probed
iommu: Adding device ffe488000.port to group 10
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e1000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe489000.port to group 22
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e3000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48a000.port to group 23
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e5000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48b000.port to group 24
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48c000.port to group 25
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e9000: Error while reading PHY0 reg at 3.3
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac ffe4e0000.ethernet: FMan MEMAC
fsl_mac ffe4e0000.ethernet: FMan MAC address: 00:06:9c:0b:06:1f
fsl_dpa dpaa-ethernet.0 eth0: Probed interface eth0
fsl_dpa dpaa-ethernet.1 eth1: Probed interface eth1
fsl_dpa dpaa-ethernet.2 eth2: Probed interface eth2
fsl_dpa dpaa-ethernet.3 eth3: Probed interface eth3
fsl_dpa dpaa-ethernet.4 eth4: Probed interface eth4

Still some minor errors: mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
but this is going the right way(I have not had a chance to try if they work due
to external modules not ported/ready yet)

The other patch series is still to be tested though but I already now wanted 
to stress the importance of getting all upstream fixes into stable, ASAP.
You now what they are, I have no idea.
FYI, applied http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=* on 4.14.x
and I still see:
libphy: Fixed MDIO Bus: probed
tun: Universal TUN/TAP device driver, 1.6
libphy: Freescale XGMAC MDIO Bus: probed
iommu: Adding device ffe488000.port to group 10
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e1000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe489000.port to group 22
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e3000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48a000.port to group 23
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e5000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48b000.port to group 24
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e7000: Error while reading PHY0 reg at 3.3
iommu: Adding device ffe48c000.port to group 25
libphy: Freescale XGMAC MDIO Bus: probed
mdio_bus ffe4e9000: Error while reading PHY0 reg at 3.3

Other than that, FMAN appears to be working.

 Jocke

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-19 08:00:58

On Thu, 1970-01-01 at 00:00 +0000, Madalin-cristian Bucur wrote:
quoted
-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Tuesday, January 16, 2018 7:58 PM
To: andrew@lunn.ch
Subject: Re: DPAA Ethernet traffice troubles with Linux kernel

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
quoted
Hi Joakim

You appear to be using an old kernel. Take a look at:
Not really, I am using 4.14.x and I don't think that is old. Seems like
this
patch hasn't been sent to 4.14.x.

I wonder if I might be missing something else, we just moved to 4.14 and
notic that all
our fixed PHYs are non functioning:
fsl_mac ffe4e2000.ethernet: FMan MEMAC
fsl_mac ffe4e2000.ethernet: FMan MAC address: 00:06:9c:0b:06:20
fsl_mac dpaa-ethernet.0: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.0 failed with error -16
fsl_mac ffe4e4000.ethernet: FMan MEMAC
fsl_mac ffe4e4000.ethernet: FMan MAC address: 00:06:9c:0b:06:21
fsl_mac dpaa-ethernet.1: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.1 failed with error -16
fsl_mac ffe4e6000.ethernet: FMan MEMAC
fsl_mac ffe4e6000.ethernet: FMan MAC address: 00:06:9c:0b:06:22
fsl_mac dpaa-ethernet.2: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.2 failed with error -16
fsl_mac ffe4e8000.ethernet: FMan MEMAC
fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:06:9c:0b:06:23
fsl_mac dpaa-ethernet.3: __devm_request_mem_region(mac) failed
fsl_mac: probe of dpaa-ethernet.3 failed with error -16

Feels like FMAN still think there are real PHYs there ?
Hi Joakim,

These errors are issued when trying to probe the second time the same
MAC node. The issue was introduced by this commit:

commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.
Looking at the above patch seriers I see it is in state Accepted and has been there
since 2017-10-16
That seems like a awful long to wait in before getting into Linux, is there something
holding these patches back ?

 Jocke 

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-01-19 13:22:57

quoted
commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.
Looking at the above patch seriers I see it is in state Accepted and has been there
since 2017-10-16
That seems like a awful long to wait in before getting into Linux, is there something
holding these patches back ?
They are in Linux, have been since October 16th. But at the moment,
they are only in v4.15, not v4.14.

These patches probably don't fit the stable rules, for getting them
added to v4.14.

https://github.com/torvalds/linux/blob/master/Documentation/process/stable-kernel-rules.rst

What is needed is a minimal fix. Or just wait until Sunday, when there
is a good chance v4.15 will be released.

   Andrew

Re: DPAA Ethernet traffice troubles with Linux kernel

From: Joakim Tjernlund <hidden>
Date: 2018-01-19 13:42:54

On Thu, 1970-01-01 at 00:00 +0000, Andrew Lunn wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

quoted
quoted
commit 4d8ee1935bcd666360311dfdadeee235d682d69a
Author: Florian Fainelli [off-list ref]
Date: Tue Aug 22 15:24:47 2017 -0700
fsl/man: Inherit parent device and of_node

and was later addressed by this patch set:

http://patchwork.ozlabs.org/project/netdev/list/?series=8462&state=*

Even with these errors printed, all is working fine, it's just the
second probing that fails. Adding the latter patches or reverting
the one above makes the errors prints dissapear.
Looking at the above patch seriers I see it is in state Accepted and has been there
since 2017-10-16
That seems like a awful long to wait in before getting into Linux, is there something
holding these patches back ?
They are in Linux, have been since October 16th. But at the moment,
they are only in v4.15, not v4.14.
Now I see them in 4.15, must have looked in the wrong branch.
These patches probably don't fit the stable rules, for getting them
added to v4.14.

https://github.com/torvalds/linux/blob/master/Documentation/process/stable-kernel-rules.rst
Stuff needs to work, whatever needed to make that happen is allowed. Even backporting
some new infra structure if need be to simplify fixing bugs.
What is needed is a minimal fix. Or just wait until Sunday, when there
is a good chance v4.15 will be released.
You seem to think everyone always upgrade to linux latest but this is not so.
We do product development here and appreciate the stable kernels so we can work in
peace and not chasing the latest kernel.

 Jocke

PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Christian Zigotzky <hidden>
Date: 2018-02-04 16:47:13

Hello,

The PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the 
first networking updates [1] for the kernel 4.16.

Error messages:

[    0.634241] libphy: pasemi gpio mdio bus: probed
[    0.634749] pasemi gpio mdio bus: Cannot register as MDIO bus, err -38
[    2.311496] pasemi_mac 0000:00:14.0: runtime IRQ mapping not provided 
by arch
[    2.311554] pasemi_mac 0000:00:14.1: runtime IRQ mapping not provided 
by arch
[    2.311599] pasemi_mac 0000:00:14.2: runtime IRQ mapping not provided 
by arch
[    2.311641] pasemi_mac 0000:00:14.3: runtime IRQ mapping not provided 
by arch
[    2.312276] pasemi_mac 0000:00:15.0: runtime IRQ mapping not provided 
by arch
[    2.312903] pasemi_mac 0000:00:15.1: runtime IRQ mapping not provided 
by arch
[    3.817420] i2c-pasemi 0000:00:1c.0: runtime IRQ mapping not provided 
by arch
[    3.817616] i2c-pasemi 0000:00:1c.1: runtime IRQ mapping not provided 
by arch
[    3.817809] i2c-pasemi 0000:00:1c.2: runtime IRQ mapping not provided 
by arch
[    4.299984] pasemi_edac 0000:00:04.0: runtime IRQ mapping not 
provided by arch
[    4.300281] pasemi_edac 0000:00:05.0: runtime IRQ mapping not 
provided by arch
[   39.633565] pasemi_mac 0000:00:14.3: PHY init failed: -19.
[   39.633569] pasemi_mac 0000:00:14.3: Defaulting to 1Gbit full duplex

I figured out that the problematic code is in the mdio bus changes of 
the networking updates. [1]

I found the problematic code in the file 'drivers/net/phy/mdio_bus.c'. I 
created a patch which solves the problem with the PA Semi PWRficient 
Gigabit Ethernet. (attached)

Could you please check the changes in the file 'drivers/net/phy/mdio_bus.c'?

Thanks,
Christian

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b2fe5fa68642860e7de76167c3111623aa0d5de1

Re: PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-02-04 17:16:31

On Sun, Feb 04, 2018 at 05:47:03PM +0100, Christian Zigotzky wrote:
Hello,

The PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first
networking updates [1] for the kernel 4.16.

Error messages:

[    0.634241] libphy: pasemi gpio mdio bus: probed
[    0.634749] pasemi gpio mdio bus: Cannot register as MDIO bus, err -38
-38 is ENOSYS.
quoted hunk
--- a/drivers/net/phy/mdio_bus.c	2018-02-03 17:34:46.973045321 +0100
+++ b/drivers/net/phy/mdio_bus.c	2018-02-04 11:03:14.909093360 +0100
@@ -47,41 +47,11 @@
 
 #include "mdio-boardinfo.h"
 
-static int mdiobus_register_gpiod(struct mdio_device *mdiodev)
-{
-	struct gpio_desc *gpiod = NULL;
-
-	/* Deassert the optional reset signal */
-	if (mdiodev->dev.of_node)
-		gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
-					       "reset-gpios", 0, GPIOD_OUT_LOW,
-					       "PHY reset");
So i think you don't have GPIOLIB enabled. Hence you are hitting

http://elixir.free-electrons.com/linux/latest/source/include/linux/gpio/consumer.h#L470

static inline
struct gpio_desc *fwnode_get_named_gpiod(struct fwnode_handle *fwnode,
					 const char *propname, int index,
					 enum gpiod_flags dflags,
					 const char *label)
{
	return ERR_PTR(-ENOSYS);
}

So rather than just deleting all this code, breaking other platforms
that need this gpio, lets try a real fix. Please try this. If it
works, i will formally submit it.

   Andrew
From a4210ba306948497d7360927c1e532eb903c58b2 Mon Sep 17 00:00:00 2001
From: Andrew Lunn <andrew@lunn.ch>
Date: Sun, 4 Feb 2018 11:09:20 -0600
Subject: [PATCH] net: phy: Handle not having GPIO enabled in the kernel

If CONFIG_GPIOLIB is disabled, fwnode_get_named_gpiod() becomes a stub
function, which return -ENOSYS. Handle this in the same way as
-ENOENT, i.e. assume there is no GPIO used to reset the PHYs.

Reported-by: Christian Zigotzky <redacted>
Signed-off-by: Andrew Lunn <andrew@lunn.ch>
---
 drivers/net/phy/mdio_bus.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/phy/mdio_bus.c b/drivers/net/phy/mdio_bus.c
index 88272b3ac2e2..24b5511222c8 100644
--- a/drivers/net/phy/mdio_bus.c
+++ b/drivers/net/phy/mdio_bus.c
@@ -56,7 +56,8 @@ static int mdiobus_register_gpiod(struct mdio_device *mdiodev)
 		gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
 					       "reset-gpios", 0, GPIOD_OUT_LOW,
 					       "PHY reset");
-	if (PTR_ERR(gpiod) == -ENOENT)
+	if (PTR_ERR(gpiod) == -ENOENT ||
+	    PTR_ERR(gpiod) == -ENOSYS)
 		gpiod = NULL;
 	else if (IS_ERR(gpiod))
 		return PTR_ERR(gpiod);
-- 
2.15.1

Re: PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Florian Fainelli <f.fainelli@gmail.com>
Date: 2018-02-04 20:02:04


On 02/04/2018 09:16 AM, Andrew Lunn wrote:
On Sun, Feb 04, 2018 at 05:47:03PM +0100, Christian Zigotzky wrote:
quoted
Hello,

The PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first
networking updates [1] for the kernel 4.16.

Error messages:

[    0.634241] libphy: pasemi gpio mdio bus: probed
[    0.634749] pasemi gpio mdio bus: Cannot register as MDIO bus, err -38
-38 is ENOSYS.
quoted
--- a/drivers/net/phy/mdio_bus.c	2018-02-03 17:34:46.973045321 +0100
+++ b/drivers/net/phy/mdio_bus.c	2018-02-04 11:03:14.909093360 +0100
@@ -47,41 +47,11 @@
 
 #include "mdio-boardinfo.h"
 
-static int mdiobus_register_gpiod(struct mdio_device *mdiodev)
-{
-	struct gpio_desc *gpiod = NULL;
-
-	/* Deassert the optional reset signal */
-	if (mdiodev->dev.of_node)
-		gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
-					       "reset-gpios", 0, GPIOD_OUT_LOW,
-					       "PHY reset");
So i think you don't have GPIOLIB enabled. Hence you are hitting

http://elixir.free-electrons.com/linux/latest/source/include/linux/gpio/consumer.h#L470

static inline
struct gpio_desc *fwnode_get_named_gpiod(struct fwnode_handle *fwnode,
					 const char *propname, int index,
					 enum gpiod_flags dflags,
					 const char *label)
{
	return ERR_PTR(-ENOSYS);
}

So rather than just deleting all this code, breaking other platforms
that need this gpio, lets try a real fix. Please try this. If it
works, i will formally submit it.

   Andrew

From a4210ba306948497d7360927c1e532eb903c58b2 Mon Sep 17 00:00:00 2001
From: Andrew Lunn <andrew@lunn.ch>
Date: Sun, 4 Feb 2018 11:09:20 -0600
Subject: [PATCH] net: phy: Handle not having GPIO enabled in the kernel

If CONFIG_GPIOLIB is disabled, fwnode_get_named_gpiod() becomes a stub
function, which return -ENOSYS. Handle this in the same way as
-ENOENT, i.e. assume there is no GPIO used to reset the PHYs.

Reported-by: Christian Zigotzky <redacted>
Signed-off-by: Andrew Lunn <andrew@lunn.ch>
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
Fixes: bafbdd527d56 ("phylib: Add device reset GPIO support")
quoted hunk
---
 drivers/net/phy/mdio_bus.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/phy/mdio_bus.c b/drivers/net/phy/mdio_bus.c
index 88272b3ac2e2..24b5511222c8 100644
--- a/drivers/net/phy/mdio_bus.c
+++ b/drivers/net/phy/mdio_bus.c
@@ -56,7 +56,8 @@ static int mdiobus_register_gpiod(struct mdio_device *mdiodev)
 		gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
 					       "reset-gpios", 0, GPIOD_OUT_LOW,
 					       "PHY reset");
-	if (PTR_ERR(gpiod) == -ENOENT)
+	if (PTR_ERR(gpiod) == -ENOENT ||
+	    PTR_ERR(gpiod) == -ENOSYS)
 		gpiod = NULL;
 	else if (IS_ERR(gpiod))
 		return PTR_ERR(gpiod);
-- 
Florian

PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Christian Zigotzky <hidden>
Date: 2018-02-05 09:38:44

Hello Andrew,

Many thanks for your patch. I compiled the latest git kernel today and 
the PA Semi PWRficient Gigabit Ethernet works with your patch.

Have a nice week!

Thanks,
Christian


On 04 February 2018 at 6:16PM, Andrew Lunn wrote:
 > On Sun, Feb 04, 2018 at 05:47:03PM +0100, Christian Zigotzky wrote:
 >> Hello,
 >>
 >> The PA Semi PWRficient Gigabit Ethernet doesn't work anymore since 
the first
 >> networking updates [1] for the kernel 4.16.
 >>
 >> Error messages:
 >>
 >> [    0.634241] libphy: pasemi gpio mdio bus: probed
 >> [    0.634749] pasemi gpio mdio bus: Cannot register as MDIO bus, 
err -38
 >
 > -38 is ENOSYS.
 >
 >> --- a/drivers/net/phy/mdio_bus.c    2018-02-03 17:34:46.973045321 +0100
 >> +++ b/drivers/net/phy/mdio_bus.c    2018-02-04 11:03:14.909093360 +0100
 >> @@ -47,41 +47,11 @@
 >>
 >>  #include "mdio-boardinfo.h"
 >>
 >> -static int mdiobus_register_gpiod(struct mdio_device *mdiodev)
 >> -{
 >> -    struct gpio_desc *gpiod = NULL;
 >> -
 >> -    /* Deassert the optional reset signal */
 >> -    if (mdiodev->dev.of_node)
 >> -        gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
 >> -                           "reset-gpios", 0, GPIOD_OUT_LOW,
 >> -                           "PHY reset");
 >
 > So i think you don't have GPIOLIB enabled. Hence you are hitting
 >
 > 
http://elixir.free-electrons.com/linux/latest/source/include/linux/gpio/consumer.h#L470
 >
 > static inline
 > struct gpio_desc *fwnode_get_named_gpiod(struct fwnode_handle *fwnode,
 >                      const char *propname, int index,
 >                      enum gpiod_flags dflags,
 >                      const char *label)
 > {
 >     return ERR_PTR(-ENOSYS);
 > }
 >
 > So rather than just deleting all this code, breaking other platforms
 > that need this gpio, lets try a real fix. Please try this. If it
 > works, i will formally submit it.
 >
 >    Andrew
 >
 > >From a4210ba306948497d7360927c1e532eb903c58b2 Mon Sep 17 00:00:00 2001
 > From: Andrew Lunn [off-list ref]
 > Date: Sun, 4 Feb 2018 11:09:20 -0600
 > Subject: [PATCH] net: phy: Handle not having GPIO enabled in the kernel
 >
 > If CONFIG_GPIOLIB is disabled, fwnode_get_named_gpiod() becomes a stub
 > function, which return -ENOSYS. Handle this in the same way as
 > -ENOENT, i.e. assume there is no GPIO used to reset the PHYs.
 >
 > Reported-by: Christian Zigotzky [off-list ref]
 > Signed-off-by: Andrew Lunn [off-list ref]
 > ---
 >  drivers/net/phy/mdio_bus.c | 3 ++-
 >  1 file changed, 2 insertions(+), 1 deletion(-)
 >
 > diff --git a/drivers/net/phy/mdio_bus.c b/drivers/net/phy/mdio_bus.c
 > index 88272b3ac2e2..24b5511222c8 100644
 > --- a/drivers/net/phy/mdio_bus.c
 > +++ b/drivers/net/phy/mdio_bus.c
 > @@ -56,7 +56,8 @@ static int mdiobus_register_gpiod(struct 
mdio_device *mdiodev)
 >          gpiod = fwnode_get_named_gpiod(&mdiodev->dev.of_node->fwnode,
 >                             "reset-gpios", 0, GPIOD_OUT_LOW,
 >                             "PHY reset");
 > -    if (PTR_ERR(gpiod) == -ENOENT)
 > +    if (PTR_ERR(gpiod) == -ENOENT ||
 > +        PTR_ERR(gpiod) == -ENOSYS)
 >          gpiod = NULL;
 >      else if (IS_ERR(gpiod))
 >          return PTR_ERR(gpiod);

Re: PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Andrew Lunn <andrew@lunn.ch>
Date: 2018-02-05 14:29:06

On Mon, Feb 05, 2018 at 10:38:34AM +0100, Christian Zigotzky wrote:
Hello Andrew,

Many thanks for your patch. I compiled the latest git kernel today and the
PA Semi PWRficient Gigabit Ethernet works with your patch.
Great.

Can i add a tested-by:

Thanks
	Andrew

Re: PA Semi PWRficient Gigabit Ethernet doesn't work anymore since the first networking updates for the kernel 4.16

From: Christian Zigotzky <hidden>
Date: 2018-02-05 15:27:11

Yes, you can.

Christian

On 05 February 2018 at 3:29PM, Andrew Lunn wrote:
On Mon, Feb 05, 2018 at 10:38:34AM +0100, Christian Zigotzky wrote:
quoted
Hello Andrew,

Many thanks for your patch. I compiled the latest git kernel today and the
PA Semi PWRficient Gigabit Ethernet works with your patch.
Great.

Can i add a tested-by:

Thanks
	Andrew
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help