From: Martin Blumenstingl <martin.blumenstingl@googlemail.com> Date: 2021-03-24 20:05:26
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If my description is too vague then please let me know about any
specific questions you have.
Also if there's an existing driver that "does things right" I am happy
to look at that one.
Thank you and best regards,
Martin
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-24 20:14:37
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If my description is too vague then please let me know about any
specific questions you have.
Also if there's an existing driver that "does things right" I am happy
to look at that one.
Is this question specific in any way to DSA?
Many Ethernet drivers are multi-queue. Some map one queue per CPU, some
use mqprio to map one or more queues per priority.
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
quoted
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.
You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
--
Florian
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-24 22:22:11
Hi Florian,
On Wed, Mar 24, 2021 at 02:09:02PM -0700, Florian Fainelli wrote:
On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
quoted
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
quoted
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.
You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
What are the benefits of mapping packets to TX queues of the DSA master
from the DSA layer?
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
quoted
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.
You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
Hi,
The PMAC in the xrx200 has 4 TX queues and 8 RX queues. We can not map
one queue to each port as there are more ports than queues. I am also
unsure if the DSL part which is using an out of tree driver uses some of
these DMA resources.
Is it possible to configure a mapping between a DSA bridge and a queue
on the mater device with tc from user space? We could expose these 4 TX
queues on the mac driver to Linux and then Linux configure somehow a
mapping between ports or bridges and queues.
Hauke
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
quoted
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.
You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
Hi,
The PMAC in the xrx200 has 4 TX queues and 8 RX queues. We can not map
one queue to each port as there are more ports than queues. I am also
unsure if the DSL part which is using an out of tree driver uses some of
these DMA resources.
bcmsysport.c for the "Lite" version has the same kind of limitation, so
what it does is limit the number of DSA slave queues to continue to
support a 1:1 mapping.
Is it possible to configure a mapping between a DSA bridge and a queue
on the mater device with tc from user space? We could expose these 4 TX
queues on the mac driver to Linux and then Linux configure somehow a
mapping between ports or bridges and queues.
There is not a way to establish a mapping from user-space between the
DSA slave queues and the DSA master queues, I am not too sure what would
be the right way to do that but it would likely require changes to the
'tc' tool and ndo_setup_tc() most likely.
--
Florian
Hi Florian,
On Wed, Mar 24, 2021 at 02:09:02PM -0700, Florian Fainelli wrote:
quoted
On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
quoted
Hi Martin,
On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
quoted
Hello,
the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).
Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue
With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.
My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?
If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.
You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
What are the benefits of mapping packets to TX queues of the DSA master
from the DSA layer?
For systemport and bcm_sf2 this was explained in this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d
in a nutshell, the switch hardware can return the queue status back to
the systemport's transmit DMA such that it can automatically pace the TX
completion interrupts. To do that we need to establish a mapping between
the DSA slave and master that is comprised of the switch port number and
TX queue number, and tell the HW to inspect the congestion status of
that particular port and queue.
What this is meant to address is a "lossless" (within the SoC at least)
behavior when you have user ports that are connected at a speed lower
than that of your internal connection to the switch typically Gigabit or
more. If you send 1Gbits/sec worth of traffic down to a port that is
connected at 100Mbits/sec there will be roughly 90% packet loss unless
you have a way to pace the Ethernet controller's transmit DMA, which
then ultimately limits the TX completion of the socket buffers so things
work nicely. I believe that per queue flow control was evaluated before
and an out of band mechanism was preferred but I do not remember the
details of that decision to use ACB.
--
Florian
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-25 01:21:47
On Wed, Mar 24, 2021 at 04:07:47PM -0700, Florian Fainelli wrote:
quoted
What are the benefits of mapping packets to TX queues of the DSA master
from the DSA layer?
For systemport and bcm_sf2 this was explained in this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d
in a nutshell, the switch hardware can return the queue status back to
the systemport's transmit DMA such that it can automatically pace the TX
completion interrupts. To do that we need to establish a mapping between
the DSA slave and master that is comprised of the switch port number and
TX queue number, and tell the HW to inspect the congestion status of
that particular port and queue.
What this is meant to address is a "lossless" (within the SoC at least)
behavior when you have user ports that are connected at a speed lower
than that of your internal connection to the switch typically Gigabit or
more. If you send 1Gbits/sec worth of traffic down to a port that is
connected at 100Mbits/sec there will be roughly 90% packet loss unless
you have a way to pace the Ethernet controller's transmit DMA, which
then ultimately limits the TX completion of the socket buffers so things
work nicely. I believe that per queue flow control was evaluated before
and an out of band mechanism was preferred but I do not remember the
details of that decision to use ACB.
Interesting system design.
Just to clarify, this port to queue mapping is completely optional, right?
You can send packets to a certain switch port through any TX queue of
the systemport?
On Wed, Mar 24, 2021 at 04:07:47PM -0700, Florian Fainelli wrote:
quoted
quoted
What are the benefits of mapping packets to TX queues of the DSA master
from the DSA layer?
For systemport and bcm_sf2 this was explained in this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d
in a nutshell, the switch hardware can return the queue status back to
the systemport's transmit DMA such that it can automatically pace the TX
completion interrupts. To do that we need to establish a mapping between
the DSA slave and master that is comprised of the switch port number and
TX queue number, and tell the HW to inspect the congestion status of
that particular port and queue.
What this is meant to address is a "lossless" (within the SoC at least)
behavior when you have user ports that are connected at a speed lower
than that of your internal connection to the switch typically Gigabit or
more. If you send 1Gbits/sec worth of traffic down to a port that is
connected at 100Mbits/sec there will be roughly 90% packet loss unless
you have a way to pace the Ethernet controller's transmit DMA, which
then ultimately limits the TX completion of the socket buffers so things
work nicely. I believe that per queue flow control was evaluated before
and an out of band mechanism was preferred but I do not remember the
details of that decision to use ACB.
Interesting system design.
Just to clarify, this port to queue mapping is completely optional, right?
You can send packets to a certain switch port through any TX queue of
the systemport?
Absolutely, the Broadcom tags allow you to steer traffic towards any TX
queue of any switch port. If ring inspection is disabled then there is
no flow control applied whatsoever.
--
Florian
From: Martin Blumenstingl <martin.blumenstingl@googlemail.com> Date: 2021-03-25 19:30:29
Hello Florian, Vladimir, Hauke,
first of all: thank you very much for this very informative discussion!
On Thu, Mar 25, 2021 at 4:08 AM Florian Fainelli [off-list ref] wrote:
[...]
quoted
Just to clarify, this port to queue mapping is completely optional, right?
You can send packets to a certain switch port through any TX queue of
the systemport?
Absolutely, the Broadcom tags allow you to steer traffic towards any TX
queue of any switch port. If ring inspection is disabled then there is
no flow control applied whatsoever.
I think with the Lantiq PMAC and GSWIP the situation is very similar
skb_set_queue_mapping from tag_brcm.c seems very interesting.
a per-CPU queue is also very interesting since these Lantiq SoCs have
two MIPS cores
Many thanks to all of you again!
Best regards,
Martin