From: Alessandro B Maurici <hidden> Date: 2021-11-23 02:55:57
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler, and, due to
the commit c10a485c3de5 ("phy: phy_ethtool_ksettings_get: Lock the phy for
consistency"), this will cause a lockup.
As that mutex call is needed for consistency, we need to release the lock,
if previously locked, before calling the handler to prevent issues.
Signed-off-by: Alessandro B Maurici <redacted>
---
drivers/net/phy/phy.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-11-23 04:18:16
On Mon, Nov 22, 2021 at 11:55:48PM -0300, Alessandro B Maurici wrote:
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler
I think we need to take a step back here and answer the question, why
does it call phy_ethtool_get_link_ksettings in the link change
handler. I'm not aware of any other MAC driver which does this.
Andrew
From: Alessandro B Maurici <hidden> Date: 2021-11-23 04:49:57
On Tue, 23 Nov 2021 05:18:14 +0100
Andrew Lunn [off-list ref] wrote:
On Mon, Nov 22, 2021 at 11:55:48PM -0300, Alessandro B Maurici wrote:
quoted
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler
I think we need to take a step back here and answer the question, why
does it call phy_ethtool_get_link_ksettings in the link change
handler. I'm not aware of any other MAC driver which does this.
Andrew
I agree, the use in the lan743x seems related to the PTP, that driver seems
to be the only one using it, at least in the Linus tree.
I think that driver could be patched as there are other ways to do it,
but my take on the problem itself is that the PHY device interface opens
a way to break the flow and this behavior does not seem to be documented,
so, instead of documenting a possible harmful interface while in the callback,
we should just get rid of the problem itself, and calling a callback without
any locks held seems to be a good alternative.
This is also a non critical performance path and the additional code
would not impact much, of course it makes the stuff less nice to look at.
The patch also has an additional check for the lock, since there is a
function that is not calling the lock explicitly and has a warn if the lock
is not held at the start, so I put it there to be extra safe.
Alessandro
On Tue, 23 Nov 2021 05:18:14 +0100
Andrew Lunn [off-list ref] wrote:
quoted
On Mon, Nov 22, 2021 at 11:55:48PM -0300, Alessandro B Maurici wrote:
quoted
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler
I think we need to take a step back here and answer the question, why
does it call phy_ethtool_get_link_ksettings in the link change
handler. I'm not aware of any other MAC driver which does this.
Andrew
I agree, the use in the lan743x seems related to the PTP, that driver seems
to be the only one using it, at least in the Linus tree.
I think that driver could be patched as there are other ways to do it,
but my take on the problem itself is that the PHY device interface opens
a way to break the flow and this behavior does not seem to be documented,
so, instead of documenting a possible harmful interface while in the callback,
we should just get rid of the problem itself, and calling a callback without
any locks held seems to be a good alternative.
This is also a non critical performance path and the additional code
would not impact much, of course it makes the stuff less nice to look at.
The patch also has an additional check for the lock, since there is a
function that is not calling the lock explicitly and has a warn if the lock
is not held at the start, so I put it there to be extra safe.
Alessandro
Seeing the following code snippet in lan743x_phy_link_status_change()
I wonder why it doesn't use phydev->speed and phydev->duplex directly.
The current code seems to include unneeded overhead.
phy_ethtool_get_link_ksettings(netdev, &ksettings);
local_advertisement =
linkmode_adv_to_mii_adv_t(phydev->advertising);
remote_advertisement =
linkmode_adv_to_mii_adv_t(phydev->lp_advertising);
lan743x_phy_update_flowcontrol(adapter,
ksettings.base.duplex,
local_advertisement,
remote_advertisement);
lan743x_ptp_update_latency(adapter, ksettings.base.speed);
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-11-23 14:09:31
On Tue, Nov 23, 2021 at 01:49:46AM -0300, Alessandro B Maurici wrote:
On Tue, 23 Nov 2021 05:18:14 +0100
Andrew Lunn [off-list ref] wrote:
quoted
On Mon, Nov 22, 2021 at 11:55:48PM -0300, Alessandro B Maurici wrote:
quoted
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler
I think we need to take a step back here and answer the question, why
does it call phy_ethtool_get_link_ksettings in the link change
handler. I'm not aware of any other MAC driver which does this.
Andrew
I agree, the use in the lan743x seems related to the PTP, that driver seems
to be the only one using it, at least in the Linus tree.
I think that driver could be patched as there are other ways to do it,
but my take on the problem itself is that the PHY device interface opens
a way to break the flow and this behavior does not seem to be documented,
so, instead of documenting a possible harmful interface while in the callback,
we should just get rid of the problem itself, and calling a callback without
any locks held seems to be a good alternative.
That is a really bad alternative. It is only because the lock is held
can the MAC driver actually trust anything passed to it. The callback
needs phydev->speed, phydev->duplex, etc, and they can change at any
time when the lock is not held. The values can be inconsistent with
each other, etc, unless the lock is held.
The callback has always had the lock held, so is safe. However,
recently a few bugs have been reported and fixed for functions like
phy_ethtool_get_link_ksettings() and phy_ethtool_set_link_ksettings()
where they have accessed phydev members without the lock and got
inconsistent values in race condition. These are hard race conditions
to reproduce, but a deadlock like this is very obvious, easy to fix. I
would also say that _ethtool_ in the function name is also a good hit
this is intended to be used for an ethtool callback.
Lets remove the inappropriate use of phy_ethtool_get_link_ksettings()
here.
Andrew
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-11-23 14:11:33
Seeing the following code snippet in lan743x_phy_link_status_change()
I wonder why it doesn't use phydev->speed and phydev->duplex directly.
The current code seems to include unneeded overhead.
Yes, that is the change i would make. When adding the extra locks i
missed that a driver was doing something like this. I will check all
other callers to see if they are using it in odd contexts.
Andrew
From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2021-11-23 14:14:24
On Tue, Nov 23, 2021 at 03:09:04PM +0100, Andrew Lunn wrote:
On Tue, Nov 23, 2021 at 01:49:46AM -0300, Alessandro B Maurici wrote:
quoted
On Tue, 23 Nov 2021 05:18:14 +0100
Andrew Lunn [off-list ref] wrote:
quoted
On Mon, Nov 22, 2021 at 11:55:48PM -0300, Alessandro B Maurici wrote:
quoted
From: Alessandro B Maurici <redacted>
Releases the phy lock before calling phy_link_change to avoid any worker
thread lockup. Some network drivers(eg Microchip's LAN743x), make a call to
phy_ethtool_get_link_ksettings inside the link change handler
I think we need to take a step back here and answer the question, why
does it call phy_ethtool_get_link_ksettings in the link change
handler. I'm not aware of any other MAC driver which does this.
Andrew
I agree, the use in the lan743x seems related to the PTP, that driver seems
to be the only one using it, at least in the Linus tree.
I think that driver could be patched as there are other ways to do it,
but my take on the problem itself is that the PHY device interface opens
a way to break the flow and this behavior does not seem to be documented,
so, instead of documenting a possible harmful interface while in the callback,
we should just get rid of the problem itself, and calling a callback without
any locks held seems to be a good alternative.
That is a really bad alternative. It is only because the lock is held
can the MAC driver actually trust anything passed to it. The callback
needs phydev->speed, phydev->duplex, etc, and they can change at any
time when the lock is not held. The values can be inconsistent with
each other, etc, unless the lock is held.
The callback has always had the lock held, so is safe. However,
recently a few bugs have been reported and fixed for functions like
phy_ethtool_get_link_ksettings() and phy_ethtool_set_link_ksettings()
where they have accessed phydev members without the lock and got
inconsistent values in race condition. These are hard race conditions
to reproduce, but a deadlock like this is very obvious, easy to fix. I
would also say that _ethtool_ in the function name is also a good hit
this is intended to be used for an ethtool callback.
Lets remove the inappropriate use of phy_ethtool_get_link_ksettings()
here.
From: Alessandro B Maurici <hidden> Date: 2021-11-23 15:58:21
On Tue, 23 Nov 2021 15:09:04 +0100
Andrew Lunn [off-list ref] wrote:
The callback has always had the lock held, so is safe. However,
recently a few bugs have been reported and fixed for functions like
phy_ethtool_get_link_ksettings() and phy_ethtool_set_link_ksettings()
where they have accessed phydev members without the lock and got
inconsistent values in race condition. These are hard race conditions
to reproduce, but a deadlock like this is very obvious, easy to fix. I
would also say that _ethtool_ in the function name is also a good hit
this is intended to be used for an ethtool callback.
Lets remove the inappropriate use of phy_ethtool_get_link_ksettings()
here.
Andrew
Yes, I was under the impression because the lan743x driver used that way,
this was an expected use case, and that why the patch, but you are 100%
correct that the phy_dev information sent to the call back would be
unprotected if used with that patch. My mistake.
Alessandro
From: Alessandro B Maurici <hidden> Date: 2021-11-23 16:06:47
On Tue, 23 Nov 2021 15:11:22 +0100
Andrew Lunn [off-list ref] wrote:
Yes, that is the change i would make. When adding the extra locks i
missed that a driver was doing something like this. I will check all
other callers to see if they are using it in odd contexts.
Andrew
Andrew, this kinda of implementation is really hard to get in a fast review,
fortunately I happen to be testing one lan743x board with a 5.10.79 kernel
that had the new locks in place, and noticed that really fast, but I wrongly
assumed that call was okayish since the driver was on stable.
If you need to do some testing I will still have the hardware with me for
some time.
Alessandro
On Tue, 23 Nov 2021 15:11:22 +0100
Andrew Lunn [off-list ref] wrote:
quoted
Yes, that is the change i would make. When adding the extra locks i
missed that a driver was doing something like this. I will check all
other callers to see if they are using it in odd contexts.
Andrew
Andrew, this kinda of implementation is really hard to get in a fast review,
fortunately I happen to be testing one lan743x board with a 5.10.79 kernel
that had the new locks in place, and noticed that really fast, but I wrongly
assumed that call was okayish since the driver was on stable.
If you need to do some testing I will still have the hardware with me for
some time.
Alessandro
Great that you have test hw, could you please test the following patch?
The duplex argument of lan743x_phy_update_flowcontrol() seems to be some
leftover, it isn't used and can be removed.
From: Alessandro B Maurici <hidden> Date: 2021-11-23 22:31:23
On Tue, 23 Nov 2021 21:32:56 +0100
Heiner Kallweit [off-list ref] wrote:
quoted hunk
Great that you have test hw, could you please test the following patch?
The duplex argument of lan743x_phy_update_flowcontrol() seems to be some
leftover, it isn't used and can be removed.