Hello,
I've been trying to wrap my head around Ethernet auto-negotiation,
vs actual / real packets seen at the MAC layer. I found the relevant
Wikipedia article to be fairly informative:
https://en.wikipedia.org/wiki/Autonegotiation
The reason I care is that my Ethernet HW does not allow changing the
flow control setting once the MAC has started (more specifically, once
RX DMA has been enabled).
In nb8800_open(), the code currently works in this order:
nb8800_start_rx(dev);
phy_start(phydev);
The first line enables the MAC (and DMA).
The second enables the PHY and starts auto-negotiation.
This is a problem: I would like for PHY auto-negotiation to be
/complete/ before I enable the MAC.
What is the recommended way to wait for the PHY?
AFAICT, the PHY layer calls back into the eth driver through the
adjust_link() callback registered through of_phy_connect().
It seems like this might be a good place to enable the MAC?
(When some other conditions are true.)
Regards.
From: Andrew Lunn <andrew@lunn.ch> Date: 2017-12-06 16:59:07
On Wed, Dec 06, 2017 at 05:39:00PM +0100, Mason wrote:
Hello,
I've been trying to wrap my head around Ethernet auto-negotiation,
vs actual / real packets seen at the MAC layer. I found the relevant
Wikipedia article to be fairly informative:
https://en.wikipedia.org/wiki/Autonegotiation
The reason I care is that my Ethernet HW does not allow changing the
flow control setting once the MAC has started (more specifically, once
RX DMA has been enabled).
In nb8800_open(), the code currently works in this order:
nb8800_start_rx(dev);
phy_start(phydev);
The first line enables the MAC (and DMA).
The second enables the PHY and starts auto-negotiation.
This is a problem: I would like for PHY auto-negotiation to be
/complete/ before I enable the MAC.
What is the recommended way to wait for the PHY?
AFAICT, the PHY layer calls back into the eth driver through the
adjust_link() callback registered through of_phy_connect().
It seems like this might be a good place to enable the MAC?
That probably works, but you might have a few corner cases to handle.
I'm not sure changes at the PHY always transition through down. So you
could for example get a callback saying the link is up, 1Gbps, then a
second call saying it has dropped to 100Mbps, if your
cables/connectors are bad.
If your hardware has problems, it might be safest to stop everything
in the callback, make configuration changes, and they start everything
back up. A link negotiation change is not something you expect to
happen often. So making it slow but robust is O.K.
Andrew
On Wed, Dec 06, 2017 at 05:39:00PM +0100, Mason wrote:
quoted
I've been trying to wrap my head around Ethernet auto-negotiation,
vs actual / real packets seen at the MAC layer. I found the relevant
Wikipedia article to be fairly informative:
https://en.wikipedia.org/wiki/Autonegotiation
The reason I care is that my Ethernet HW does not allow changing the
flow control setting once the MAC has started (more specifically, once
RX DMA has been enabled).
In nb8800_open(), the code currently works in this order:
nb8800_start_rx(dev);
phy_start(phydev);
The first line enables the MAC (and DMA).
The second enables the PHY and starts auto-negotiation.
This is a problem: I would like for PHY auto-negotiation to be
/complete/ before I enable the MAC.
What is the recommended way to wait for the PHY?
quoted
AFAICT, the PHY layer calls back into the eth driver through the
adjust_link() callback registered through of_phy_connect().
It seems like this might be a good place to enable the MAC?
That probably works, but you might have a few corner cases to handle.
I'm not sure changes at the PHY always transition through down. So you
could for example get a callback saying the link is up, 1Gbps, then a
second call saying it has dropped to 100Mbps, if your
cables/connectors are bad.
If your hardware has problems, it might be safest to stop everything
in the callback, make configuration changes, and they start everything
back up. A link negotiation change is not something you expect to
happen often. So making it slow but robust is O.K.
What you've described is, in fact, the existing implementation! ;-)
nb8800_pause_config() checks for netif_running() and, when true,
tries to disable everything, make the change, then re-enable
everything.
The problem with this is the following mind-boggling quirk of
the hardware: once RX DMA is enabled, there is no supported
way to disable it! Thus, I'm trying to find a clean way to set
the control flow parameter BEFORE enabling the MAC.
Regards.
From: Andrew Lunn <andrew@lunn.ch> Date: 2017-12-06 18:26:37
The problem with this is the following mind-boggling quirk of
the hardware: once RX DMA is enabled, there is no supported
way to disable it! Thus, I'm trying to find a clean way to set
the control flow parameter BEFORE enabling the MAC.
There is no solution. I can pull the cable out, and plug it into a
different computer. The first could use flow control, the second not.
If you cannot disable RX DMA, you should probably disable unloading of
the module. Also kexec.
Andrew
The problem with this is the following mind-boggling quirk of
the hardware: once RX DMA is enabled, there is no supported
way to disable it! Thus, I'm trying to find a clean way to set
the control flow parameter BEFORE enabling the MAC.
There is no solution. I can pull the cable out, and plug it into a
different computer. The first could use flow control, the second not.
If you cannot disable RX DMA, you should probably disable unloading of
the module. Also kexec.
It is not possible to /selectively/ disable RX DMA, but after a reset,
DMA is disabled. So the solution I propose is this:
When we detect link down, we put the chip in reset, so that we will
repeat initialization when the link comes back up.
Mans has been reluctant to adopt this approach, but this is what
I implemented in my forked version. I'm trying to push my changes
upstream, to minimize the differences.
Regards.
By chip, you mean the Ethernet controller? Not the whole SoC?
Doh! Yes. Let me rephrase.
When we detect link down, we put the ethernet HW block in reset,
and repeat initialization when the link comes back up.
Hmmm, however, at the moment, I only reset on an administrative
(user-requested) link down, i.e. through ndo_stop. I would probably
have to handle cable unplug/replug events as well.
Or just consider the quirk to make flow control too complicated
to implement correctly...
Regards.
By chip, you mean the Ethernet controller? Not the whole SoC?
Doh! Yes. Let me rephrase.
When we detect link down, we put the ethernet HW block in reset,
and repeat initialization when the link comes back up.
Hmmm, however, at the moment, I only reset on an administrative
(user-requested) link down, i.e. through ndo_stop. I would probably
have to handle cable unplug/replug events as well.
Or just consider the quirk to make flow control too complicated
to implement correctly...
I suppose your procedure is fine, but don't you have a better way to
resolve that by trying to place a special RX DMA ring entry that allows
your RX DMA not to be entirely stopped, but intentionally looped through
a buffer that you control? As long as you can stop the Ethernet MAC RX,
working with such a limitation is probably fine, but this really sounds
like a huge pain in the butt and a major HW flaw.
--
Florian
When we detect link down, we put the ethernet HW block in reset,
and repeat initialization when the link comes back up.
Hmmm, however, at the moment, I only reset on an administrative
(user-requested) link down, i.e. through ndo_stop. I would probably
have to handle cable unplug/replug events as well.
Or just consider the quirk to make flow control too complicated
to implement correctly...
I suppose your procedure is fine, but don't you have a better way to
resolve that by trying to place a special RX DMA ring entry that allows
your RX DMA not to be entirely stopped, but intentionally looped through
a buffer that you control? As long as you can stop the Ethernet MAC RX,
working with such a limitation is probably fine, but this really sounds
like a huge pain in the butt and a major HW flaw.
Could you elaborate a bit on your suggestion?
(Special ring entry, looped through a buffer under my control)
Is this a typical thing to do to stop DMA?
Currently the driver tries to stop DMA in nb8800_dma_stop(),
which does the following:
http://elixir.free-electrons.com/linux/latest/source/drivers/net/ethernet/aurora/nb8800.c#L881
1) poll until TX finishes (I assume the system no longer accepts new
frames to send at this point)
2) set the EOC (end of chain) bit on all descriptors (could there be
a problem if we receive a frame at that moment? Don't we need some
kind of lock?)
3) disable address filtering (need to check what this does)
4) enable loop-back mode
5) send up to 5 "fake" packets in order to hit an EOC descriptor
The reason I'm trying to move away from this method is that it doesn't
work on our new SoC; and when pressed, the HW dev said it had never been
supported. (Also I find it somewhat hackish, but that's a matter of taste.)
Regards.
When we detect link down, we put the ethernet HW block in reset,
and repeat initialization when the link comes back up.
Hmmm, however, at the moment, I only reset on an administrative
(user-requested) link down, i.e. through ndo_stop. I would probably
have to handle cable unplug/replug events as well.
Or just consider the quirk to make flow control too complicated
to implement correctly...
I suppose your procedure is fine, but don't you have a better way to
resolve that by trying to place a special RX DMA ring entry that allows
your RX DMA not to be entirely stopped, but intentionally looped through
a buffer that you control? As long as you can stop the Ethernet MAC RX,
working with such a limitation is probably fine, but this really sounds
like a huge pain in the butt and a major HW flaw.
Could you elaborate a bit on your suggestion?
(Special ring entry, looped through a buffer under my control)
Is this a typical thing to do to stop DMA?
What I was thinking is something along the lines of what step 5 does
already, without requiring the need to loop back and sending 5 packets,
just placing an EOC descriptor, and forcing a reduction of the RX ring
size such that you hit that descriptor right away, and have DMA be
contained within that single entry in order to safely stop it. Question
is of course, how to take it out of that state considering that usually
the RX DMA produces status bits (unlike TX where the CPU produces those
bits).
Currently the driver tries to stop DMA in nb8800_dma_stop(),
which does the following:
http://elixir.free-electrons.com/linux/latest/source/drivers/net/ethernet/aurora/nb8800.c#L881
1) poll until TX finishes (I assume the system no longer accepts new
frames to send at this point)
2) set the EOC (end of chain) bit on all descriptors (could there be
a problem if we receive a frame at that moment? Don't we need some
kind of lock?)
If you have disabled the RX interrupt and quiesced NAPI, this should be
safe.
3) disable address filtering (need to check what this does)
4) enable loop-back mode
5) send up to 5 "fake" packets in order to hit an EOC descriptor
That's quite unusual.
The reason I'm trying to move away from this method is that it doesn't
work on our new SoC; and when pressed, the HW dev said it had never been
supported. (Also I find it somewhat hackish, but that's a matter of taste.)
Having any HW state machine requiring X number of clock cycles to
guarantee a full transition to a stopped state is not unusual, however,
the fact that you need to send 5 packets to guarantee an EOC descriptor
is hit is completely unusual. Ideally there is a single bit that tells
the DMA engine: you are enabled, do your thing, or you are now disabled,
and you must stop all accesses to DRAM *now*.
So what would be the correct way to quiesce that controller according to
your HW folks?
--
Florian
+ Mans
On 09/12/2017 19:49, Florian Fainelli wrote:
Having any HW state machine requiring X number of clock cycles to
guarantee a full transition to a stopped state is not unusual, however,
the fact that you need to send 5 packets to guarantee an EOC descriptor
is hit is completely unusual. Ideally there is a single bit that tells
the DMA engine: you are enabled, do your thing, or you are now disabled,
and you must stop all accesses to DRAM *now*.
So what would be the correct way to quiesce that controller according to
your HW folks?
He (it's a single person) offered to run some RTL-level simulations,
but then moved on to more important tasks. At some point he wrote:
If you reset the DMA enable in the middle of a DMA, the hardware
state machine doesn't return to the IDLE state if it has more
descriptors to process. It should be noted that the RX DMA has been
designed by our IP vendor with no intention of stopping the DMA in
the middle of its operation.
While the documentation for the IP states:
Receive DMA Channel Disabling
When the entire receive frame has been read from the Receive FIFO and
sent over the AMBA bus, the DMA operation ends, and the Receive DMA
Channel is automatically disabled. To do this, hardware resets the
Enable bit in the Receive Channel Control Register to "0" after the
last data has been read from the Receive FIFO and sent over the AMBA
bus.
When operating in descriptor mode, upon completion of a receive frame
DMA operation, if the descriptor chain has not ended when a receive
frame DMA operation completes, the next receive frame DMA operation
begins. The last descriptor in a descriptor chain is indicated by
having its End Of Chain- EOC, flag set to "1". If this EOC flag is
"0", to begin the next receive frame DMA operation, the next
descriptor is automatically retrieved and used to configure the
Receive DMA Channel. The Receive DMA Channel is then automatically
re-enabled and the next receive frame DMA operation begins.
In descriptor mode, an AMBA bus error can occur when reading receive
descriptor data. If this happens, receive descriptor processing ends
and the Receive DMA Channel is turned off. The Descriptor Error bit
in the Receive Status Register is set to "1".
I don't see how sending "fake" packets through the loop back would be
considered "resetting the DMA enable in the middle of a DMA".
(I'm afraid the HW dev didn't grasp what the driver is doing.)
I suppose I should test forcefully setting the enable bit to 0 in
the driver, and see if hell breaks loose.
Regards.
+ Mans
On 09/12/2017 19:49, Florian Fainelli wrote:
quoted
Having any HW state machine requiring X number of clock cycles to
guarantee a full transition to a stopped state is not unusual, however,
the fact that you need to send 5 packets to guarantee an EOC descriptor
is hit is completely unusual. Ideally there is a single bit that tells
the DMA engine: you are enabled, do your thing, or you are now disabled,
and you must stop all accesses to DRAM *now*.
That's how sane hardware works. This hardware is not sane.
quoted
So what would be the correct way to quiesce that controller according to
your HW folks?
He (it's a single person) offered to run some RTL-level simulations,
but then moved on to more important tasks. At some point he wrote:
quoted
If you reset the DMA enable in the middle of a DMA, the hardware
state machine doesn't return to the IDLE state if it has more
descriptors to process. It should be noted that the RX DMA has been
designed by our IP vendor with no intention of stopping the DMA in
the middle of its operation.
While the documentation for the IP states:
quoted
Receive DMA Channel Disabling
When the entire receive frame has been read from the Receive FIFO and
sent over the AMBA bus, the DMA operation ends, and the Receive DMA
Channel is automatically disabled. To do this, hardware resets the
Enable bit in the Receive Channel Control Register to "0" after the
last data has been read from the Receive FIFO and sent over the AMBA
bus.
When operating in descriptor mode, upon completion of a receive frame
DMA operation, if the descriptor chain has not ended when a receive
frame DMA operation completes, the next receive frame DMA operation
begins. The last descriptor in a descriptor chain is indicated by
having its End Of Chain- EOC, flag set to "1". If this EOC flag is
"0", to begin the next receive frame DMA operation, the next
descriptor is automatically retrieved and used to configure the
Receive DMA Channel. The Receive DMA Channel is then automatically
re-enabled and the next receive frame DMA operation begins.
In descriptor mode, an AMBA bus error can occur when reading receive
descriptor data. If this happens, receive descriptor processing ends
and the Receive DMA Channel is turned off. The Descriptor Error bit
in the Receive Status Register is set to "1".
I don't see how sending "fake" packets through the loop back would be
considered "resetting the DMA enable in the middle of a DMA".
(I'm afraid the HW dev didn't grasp what the driver is doing.)
I suppose I should test forcefully setting the enable bit to 0 in
the driver, and see if hell breaks loose.
You can't. When the enable bit is 1, writes to that register are
ignored. It goes back to 0 automatically when the hw runs out of
descriptors.
--
Måns Rullgård