Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
Thanks and best regards
Frieder
[1]: https://community.nxp.com/t5/i-MX-Processors/i-MX8MM-Ethernet-TX-Bandwidth-Fluctuations/m-p/1242467#M170563
From: Dave Taht <hidden> Date: 2021-05-06 14:56:19
I am a big fan of bql - is that implemented on this driver?
cd /sys/class/net/your_device_name/queues/tx-0/byte_queue_limits/
cat limit
see also bqlmon from github
is fq_codel running on the ethernet interface? the iperf bidir test
does much better with that in place rather than a fifo. tc -s qdisc
show dev your_device
Also I tend to run tests using the flent tool, which will yield more
data. Install netperf and irtt on the target, flent, netperf, irtt on
the test driver box...
flent -H the-target-ip -x --socket-stats -t whateveryouaretesting rrul
# the meanest bidir test there
flent-gui *.gz
On Thu, May 6, 2021 at 7:47 AM Frieder Schrempf
[off-list ref] wrote:
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
Thanks and best regards
Frieder
[1]: https://community.nxp.com/t5/i-MX-Processors/i-MX8MM-Ethernet-TX-Bandwidth-Fluctuations/m-p/1242467#M170563
From: Adam Ford <hidden> Date: 2021-05-06 19:25:28
On Thu, May 6, 2021 at 9:51 AM Frieder Schrempf
[off-list ref] wrote:
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
I have seen a similar regression on linux-next on both Mini and Nano.
I thought I broke something, but it returned to normal after a reboot.
However, with a 1Gb connection, I was running at ~450 Mbs which is
consistent with what you were seeing with a 100Mb link.
adam
From: Tim Harvey <tharvey@gateworks.com> Date: 2021-05-07 15:36:33
On Thu, May 6, 2021 at 12:20 PM Adam Ford [off-list ref] wrote:
On Thu, May 6, 2021 at 9:51 AM Frieder Schrempf
[off-list ref] wrote:
quoted
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
I have seen a similar regression on linux-next on both Mini and Nano.
I thought I broke something, but it returned to normal after a reboot.
However, with a 1Gb connection, I was running at ~450 Mbs which is
consistent with what you were seeing with a 100Mb link.
adam
Frieder,
I've noticed this as well on our designs with IMX8MN+DP83867 and
IMX8MM+KSZ9897S. I also notice it with IMX8MN+DP83867. I have noticed
it on all kernels I've tested and it appears to latch back and forth
every few times I run a 10s iperf3 between 50% and 100% line speed.
I have no idea what it is but glad you are asking and hope someone
knows how to fix it!
Best Regards,
Tim
Hi Dave,
thanks for the input. I really don't know much about the networking stack, so at the moment I can only provide the values requested below, without knowing what it really means.
What's so strange is, that the performance is actually good in general and only "snaps" to the "bad" state and back after some time or after repeated test runs.
And by the way, the ethernet driver in use is the FEC driver at drivers/net/ethernet/freescale/fec_main.c.
On 06.05.21 16:53, Dave Taht wrote:
I am a big fan of bql - is that implemented on this driver?
cd /sys/class/net/your_device_name/queues/tx-0/byte_queue_limits/
cat limit
see also bqlmon from github
is fq_codel running on the ethernet interface? the iperf bidir test
does much better with that in place rather than a fifo. tc -s qdisc
show dev your_device
~# tc -s qdisc show dev eth0
RTNETLINK answers: Operation not supported
Dump terminated
Best regards
Frieder
Also I tend to run tests using the flent tool, which will yield more
data. Install netperf and irtt on the target, flent, netperf, irtt on
the test driver box...
flent -H the-target-ip -x --socket-stats -t whateveryouaretesting rrul
# the meanest bidir test there
flent-gui *.gz
On Thu, May 6, 2021 at 7:47 AM Frieder Schrempf
[off-list ref] wrote:
On Thu, May 6, 2021 at 9:51 AM Frieder Schrempf
[off-list ref] wrote:
quoted
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
I have seen a similar regression on linux-next on both Mini and Nano.
I thought I broke something, but it returned to normal after a reboot.
However, with a 1Gb connection, I was running at ~450 Mbs which is
consistent with what you were seeing with a 100Mb link.
Thanks for your response. If you say "regression" does this mean that you had some previous version where this issue didn't occur? As for me, I can see it on 5.4 and 5.10, but I didn't try it with anything else so far.
Best regards
Frieder
On Thu, May 6, 2021 at 12:20 PM Adam Ford [off-list ref] wrote:
quoted
On Thu, May 6, 2021 at 9:51 AM Frieder Schrempf
[off-list ref] wrote:
quoted
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
I have seen a similar regression on linux-next on both Mini and Nano.
I thought I broke something, but it returned to normal after a reboot.
However, with a 1Gb connection, I was running at ~450 Mbs which is
consistent with what you were seeing with a 100Mb link.
adam
Frieder,
I've noticed this as well on our designs with IMX8MN+DP83867 and
IMX8MM+KSZ9897S. I also notice it with IMX8MN+DP83867. I have noticed
it on all kernels I've tested and it appears to latch back and forth
every few times I run a 10s iperf3 between 50% and 100% line speed.
I have no idea what it is but glad you are asking and hope someone
knows how to fix it!
Thanks for providing that information. Yes, the latching effect between "slow" and normal speed now and then is exactly what I'm seeing, too. Good to know that this is something not only happening at my end!
Best regards
Frieder
From: Adam Ford <hidden> Date: 2021-05-10 13:12:30
On Mon, May 10, 2021 at 7:52 AM Frieder Schrempf
[off-list ref] wrote:
Hi Adam,
On 06.05.21 21:20, Adam Ford wrote:
quoted
On Thu, May 6, 2021 at 9:51 AM Frieder Schrempf
[off-list ref] wrote:
quoted
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
I have seen a similar regression on linux-next on both Mini and Nano.
I thought I broke something, but it returned to normal after a reboot.
However, with a 1Gb connection, I was running at ~450 Mbs which is
consistent with what you were seeing with a 100Mb link.
Thanks for your response. If you say "regression" does this mean that you had some previous version where this issue didn't occur? As for me, I can see it on 5.4 and 5.10, but I didn't try it with anything else so far.
I have not seen this in the 4.19 kernel that NXP provided, but I have
seen this intermittently in the 5.10, so I called it a regression.
adam
From: Dave Taht <hidden> Date: 2021-05-10 15:11:29
On Mon, May 10, 2021 at 5:49 AM Frieder Schrempf
[off-list ref] wrote:
Hi Dave,
thanks for the input. I really don't know much about the networking stack, so at the moment I can only provide the values requested below, without knowing what it really means.
What's so strange is, that the performance is actually good in general and only "snaps" to the "bad" state and back after some time or after repeated test runs.
And by the way, the ethernet driver in use is the FEC driver at drivers/net/ethernet/freescale/fec_main.c.
It doesn't look (from a quick grep) that that driver ever got BQL support.
davet@Georges-MacBook-Pro freescale % grep sent_queue *.c
gianfar.c: netdev_tx_sent_queue(txq, bytes_sent);
ucc_geth.c: netdev_sent_queue(dev, skb->len);
If you really care about bidirectional throughput, having enormous
fifo buffers buried deep in the driver has a tendency to hurt that a
lot and has the symptoms you describe, however not as persistent, so I
would suspect another bug involving gso or gro to start with...
BUT: I note that the effort in implementing bql and testing the packet
size accounting usually shows up other problems in the tx/rx ring, GRO
or NAPI code, and thus is worthwhile exercise that might find where
things are getting stuck.
It doesn't appear your kernel has fq_codel qdisc support, either,
which means big dumb fifos at that layer, drastically affecting bidir
throughput. also.
Since the nxp team is cc'd this is a preso I'd given broadcom back in 2018:
http://www.taht.net/~d/broadcom_aug9_2018.pdf
And the relevant lwn articles from, like, 2011:
https://lwn.net/Articles/454390/https://lwn.net/Articles/469652/
If someone wants to send me a board to play with...
On 06.05.21 16:53, Dave Taht wrote:
quoted
I am a big fan of bql - is that implemented on this driver?
cd /sys/class/net/your_device_name/queues/tx-0/byte_queue_limits/
cat limit
see also bqlmon from github
is fq_codel running on the ethernet interface? the iperf bidir test
does much better with that in place rather than a fifo. tc -s qdisc
show dev your_device
~# tc -s qdisc show dev eth0
RTNETLINK answers: Operation not supported
Dump terminated
Best regards
Frieder
quoted
Also I tend to run tests using the flent tool, which will yield more
data. Install netperf and irtt on the target, flent, netperf, irtt on
the test driver box...
flent -H the-target-ip -x --socket-stats -t whateveryouaretesting rrul
# the meanest bidir test there
flent-gui *.gz
On Thu, May 6, 2021 at 7:47 AM Frieder Schrempf
[off-list ref] wrote:
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at my side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini
boards. It happens quite often that the measured bandwidth in TX direction
drops from its expected/nominal value to something like 50% (for 100M) or ~67%
(for 1G) connections.
So far we reproduced this with two different hardware designs using two
different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel
versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a short
p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info (with 'nc -l
-p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g.
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
There is nothing else running on the system in parallel. Some more info is also
available in this post: [1].
If there's anyone around who has an idea on what might be the reason for this,
please let me know!
Or maybe someone would be willing to do a quick test on his own hardware.
That would also be highly appreciated!
Thanks and best regards
Frieder
[1]:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommu
nity.nxp.com%2Ft5%2Fi-MX-Processors%2Fi-MX8MM-Ethernet-TX-Bandwidth-
Fluctuations%2Fm-p%2F1242467%23M170563&data=04%7C01%7Cqiang
qing.zhang%40nxp.com%7C5d4866d4565e4cbc36a008d9109da0ff%7C686ea1d
3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637559091463792932%7CUnkno
wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
WwiLCJXVCI6Mn0%3D%7C1000&sdata=ygcThQOLIzp0lzhXacRLjSjnjm1FEj
YSxakXwZtxde8%3D&reserved=0
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce on L5.10, and can't reproduce on L5.4.
According to your description, you can reproduce this issue both L5.4 and L5.10? So I need confirm with you.
Best Regards,
Joakim Zhang
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at my side, I will try
my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini
boards. It happens quite often that the measured bandwidth in TX
direction drops from its expected/nominal value to something like 50%
(for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using
two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different
kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a
short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info
(with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g.
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M
link).
quoted
There is nothing else running on the system in parallel. Some more
info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason
for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware.
That would also be highly appreciated!
Thanks and best regards
Frieder
[1]:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcomm
u
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce on L5.10, and can't reproduce on L5.4.
According to your description, you can reproduce this issue both L5.4 and L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and 5.10 but both kernels were official mainline kernels and **not** from the linux-imx downstream tree.
Maybe there is some problem in the mainline tree and it got included in the NXP release kernel starting from L5.10?
Best regards
Frieder
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at my side, I will try
my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini
boards. It happens quite often that the measured bandwidth in TX
direction drops from its expected/nominal value to something like 50%
(for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using
two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different
kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a
short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info
(with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good' (e.g.
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M
link).
quoted
There is nothing else running on the system in parallel. Some more
info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason
for this, please let me know!
Or maybe someone would be willing to do a quick test on his own hardware.
That would also be highly appreciated!
Thanks and best regards
Frieder
[1]:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcomm
u
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and 5.10 but both
kernels were official mainline kernels and **not** from the linux-imx
downstream tree.
Ok.
Maybe there is some problem in the mainline tree and it got included in the
NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after adding AVB support in mainline.
ENET IP is not a _real_ multiple queues per my understanding, queue 0 is for best effort. And the queue 1&2 is for AVB stream whose default bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps for 1Gbps). When transmitting packets, net core will select queues randomly, which caused the tx bandwidth fluctuations. So you can change to use single queue if you care more about tx bandwidth. Or you can refer to NXP internal implementation.
e.g.
I hope this can help you :)
Best Regards,
Joakim Zhang
Best regards
Frieder
quoted
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at my
side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our
i.MX8M-Mini boards. It happens quite often that the measured
bandwidth in TX direction drops from its expected/nominal value to
something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using
two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different
kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a
short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info
(with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good'
(e.g.
quoted
quoted
quoted
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for
100M
link).
quoted
There is nothing else running on the system in parallel. Some more
info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason
for this, please let me know!
Or maybe someone would be willing to do a quick test on his own
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and 5.10 but both
kernels were official mainline kernels and **not** from the linux-imx
downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got included in the
NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after adding AVB support in mainline.
ENET IP is not a _real_ multiple queues per my understanding, queue 0 is for best effort. And the queue 1&2 is for AVB stream whose default bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps for 1Gbps). When transmitting packets, net core will select queues randomly, which caused the tx bandwidth fluctuations. So you can change to use single queue if you care more about tx bandwidth. Or you can refer to NXP internal implementation.
e.g.
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
Also if there was a way to present it as two interfaces, rather than
one, that would allow for a specific avb device to be
presented.
Or:
Is there a standard means of signalling down the stack via the IP
layer (a dscp? a setsockopt?) that the AVB queue is requested?
Best Regards,
Joakim Zhang
quoted
Best regards
Frieder
quoted
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at my
side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our
i.MX8M-Mini boards. It happens quite often that the measured
bandwidth in TX direction drops from its expected/nominal value to
something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs using
two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different
kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
To measure the throughput we simply run iperf3 on the target (with a
short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the info
(with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good'
(e.g.
quoted
quoted
quoted
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for
100M
link).
quoted
There is nothing else running on the system in parallel. Some more
info is also available in this post: [1].
If there's anyone around who has an idea on what might be the reason
for this, please let me know!
Or maybe someone would be willing to do a quick test on his own
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce
on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and 5.10
but both kernels were official mainline kernels and **not** from the
linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got included
in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding, queue 0 is for
best effort. And the queue 1&2 is for AVB stream whose default bandwidth
fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps for 1Gbps).
When transmitting packets, net core will select queues randomly, which
caused the tx bandwidth fluctuations. So you can change to use single queue if
you care more about tx bandwidth. Or you can refer to NXP internal
implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be helpful to have on
all queues.
There is no BQL support in this driver, and BQL may improve throughput further, but should not be the root cause of this reported issue.
Also if there was a way to present it as two interfaces, rather than one, that
would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP layer (a dscp?
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue 1&2 based on VLAN-ID.
Best Regards,
Joakim Zhang
quoted
Best Regards,
Joakim Zhang
quoted
Best regards
Frieder
quoted
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at
my side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our
i.MX8M-Mini boards. It happens quite often that the measured
bandwidth in TX direction drops from its expected/nominal value
to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs
using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two
different kernel versions (v5.4 and v5.10) and link speeds of 100M and
1G.
quoted
quoted
quoted
quoted
quoted
To measure the throughput we simply run iperf3 on the target
(with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the
info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good'
(e.g.
quoted
quoted
quoted
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s
for 100M
link).
quoted
There is nothing else running on the system in parallel. Some
more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the
reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce
on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and 5.10
but both kernels were official mainline kernels and **not** from the
linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got included
in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding, queue 0 is for
best effort. And the queue 1&2 is for AVB stream whose default bandwidth
fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps for 1Gbps).
When transmitting packets, net core will select queues randomly, which
caused the tx bandwidth fluctuations. So you can change to use single queue if
you care more about tx bandwidth. Or you can refer to NXP internal
implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be helpful to have on
all queues.
There is no BQL support in this driver, and BQL may improve throughput further, but should not be the root cause of this reported issue.
quoted
Also if there was a way to present it as two interfaces, rather than one, that
would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP layer (a dscp?
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue 1&2 based on VLAN-ID.
I had to look up what AVB even means, but from my current understanding it doesn't seem right that for non-AVB packets the driver picks any of the three queues in a random fashion while at the same time knowing that queue 1 and 2 have a 50% limitation on the bandwidth. Shouldn't there be some way to prefer queue 0 without needing the user to set it up or even arbitrarily limiting the number of queues as proposed above?
Best Regards,
Joakim Zhang
quoted
quoted
Best Regards,
Joakim Zhang
quoted
Best regards
Frieder
quoted
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at
my side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our
i.MX8M-Mini boards. It happens quite often that the measured
bandwidth in TX direction drops from its expected/nominal value
to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs
using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two
different kernel versions (v5.4 and v5.10) and link speeds of 100M and
1G.
quoted
quoted
quoted
quoted
quoted
To measure the throughput we simply run iperf3 on the target
(with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the
info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good'
(e.g.
quoted
quoted
quoted
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s
for 100M
link).
quoted
There is nothing else running on the system in parallel. Some
more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the
reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce
on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and
5.10 but both kernels were official mainline kernels and **not**
from the linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got
included in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after
adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding, queue
0 is for
best effort. And the queue 1&2 is for AVB stream whose default
bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps
for 1Gbps).
quoted
quoted
When transmitting packets, net core will select queues randomly,
which caused the tx bandwidth fluctuations. So you can change to use
single queue if you care more about tx bandwidth. Or you can refer to
NXP internal implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
There is no BQL support in this driver, and BQL may improve throughput
further, but should not be the root cause of this reported issue.
quoted
quoted
Also if there was a way to present it as two interfaces, rather than
one, that would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP layer (a
dscp?
quoted
quoted
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue 1&2
based on VLAN-ID.
I had to look up what AVB even means, but from my current understanding it
doesn't seem right that for non-AVB packets the driver picks any of the three
queues in a random fashion while at the same time knowing that queue 1 and 2
have a 50% limitation on the bandwidth. Shouldn't there be some way to prefer
queue 0 without needing the user to set it up or even arbitrarily limiting the
number of queues as proposed above?
-----Original Message-----
From: Joakim Zhang <redacted>
Sent: 2021年5月12日 19:59
To: Frieder Schrempf <redacted>; dl-linux-imx
[off-list ref]; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: RE: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi Frieder,
Sorry, I missed this mail before, I can reproduce this issue at
my side, I will try my best to look into this issue.
Best Regards,
Joakim Zhang
quoted
-----Original Message-----
From: Frieder Schrempf <redacted>
Sent: 2021年5月6日 22:46
To: dl-linux-imx <redacted>; netdev@vger.kernel.org;
linux-arm-kernel@lists.infradead.org
Subject: i.MX8MM Ethernet TX Bandwidth Fluctuations
Hi,
we observed some weird phenomenon with the Ethernet on our
i.MX8M-Mini boards. It happens quite often that the measured
bandwidth in TX direction drops from its expected/nominal value
to something like 50% (for 100M) or ~67% (for 1G) connections.
So far we reproduced this with two different hardware designs
using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two
different kernel versions (v5.4 and v5.10) and link speeds of
100M and
1G.
quoted
quoted
quoted
quoted
quoted
To measure the throughput we simply run iperf3 on the target
(with a short p2p connection to the host PC) like this:
iperf3 -c 192.168.1.10 --bidir
But even something more simple like this can be used to get the
info (with 'nc -l -p 1122 > /dev/null' running on the host):
dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
The results fluctuate between each test run and are sometimes 'good'
(e.g.
quoted
quoted
quoted
~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s
for 100M
link).
quoted
There is nothing else running on the system in parallel. Some
more info is also available in this post: [1].
If there's anyone around who has an idea on what might be the
reason for this, please let me know!
Or maybe someone would be willing to do a quick test on his own
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can reproduce
on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and
5.10 but both kernels were official mainline kernels and **not**
from the linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got
included in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist after
adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding, queue
0 is for
best effort. And the queue 1&2 is for AVB stream whose default
bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and 500Mbps
for 1Gbps).
quoted
quoted
When transmitting packets, net core will select queues randomly,
which caused the tx bandwidth fluctuations. So you can change to use
single queue if you care more about tx bandwidth. Or you can refer to
NXP internal implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
There is no BQL support in this driver, and BQL may improve throughput
further, but should not be the root cause of this reported issue.
quoted
quoted
Also if there was a way to present it as two interfaces, rather than
one, that would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP layer (a
dscp?
quoted
quoted
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue 1&2
based on VLAN-ID.
I had to look up what AVB even means, but from my current understanding it
doesn't seem right that for non-AVB packets the driver picks any of the three
queues in a random fashion while at the same time knowing that queue 1 and 2
have a 50% limitation on the bandwidth. Shouldn't there be some way to prefer
queue 0 without needing the user to set it up or even arbitrarily limiting the
number of queues as proposed above?
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can
reproduce on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and
5.10 but both kernels were official mainline kernels and **not**
from the linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got
included in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist
after adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding,
queue
0 is for
best effort. And the queue 1&2 is for AVB stream whose default
bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and
500Mbps
for 1Gbps).
quoted
quoted
When transmitting packets, net core will select queues randomly,
which caused the tx bandwidth fluctuations. So you can change to
use single queue if you care more about tx bandwidth. Or you can
refer to NXP internal implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
There is no BQL support in this driver, and BQL may improve
throughput
further, but should not be the root cause of this reported issue.
quoted
quoted
Also if there was a way to present it as two interfaces, rather
than one, that would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP
layer (a
dscp?
quoted
quoted
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue
1&2
based on VLAN-ID.
I had to look up what AVB even means, but from my current
understanding it doesn't seem right that for non-AVB packets the
driver picks any of the three queues in a random fashion while at the
same time knowing that queue 1 and 2 have a 50% limitation on the
bandwidth. Shouldn't there be some way to prefer queue 0 without
needing the user to set it up or even arbitrarily limiting the number of
queues as proposed above?
quoted
Yes, I think we can. I look into NXP local implementation, there is a
g30bn3ok%3D&reserved=0
This is the version for L5.4 kernel.
Yes, this looks like it could solve the issue. Would you mind preparing a patch to
upstream the change in [1]? I would be happy to test (at least the non-AVB
case) and review.
Yes, I can have a try. I saw this patch has been staying in downstream tree for many years, and I don't know the history.
Anyway, I will try to upstream first to see if anyone has comments.
Best Regards,
Joakim Zhang
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can
reproduce on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and
5.10 but both kernels were official mainline kernels and **not**
from the linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got
included in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist
after adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding,
queue
0 is for
best effort. And the queue 1&2 is for AVB stream whose default
bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and
500Mbps
for 1Gbps).
quoted
quoted
When transmitting packets, net core will select queues randomly,
which caused the tx bandwidth fluctuations. So you can change to
use single queue if you care more about tx bandwidth. Or you can
refer to NXP internal implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
There is no BQL support in this driver, and BQL may improve
throughput
further, but should not be the root cause of this reported issue.
quoted
quoted
Also if there was a way to present it as two interfaces, rather
than one, that would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP
layer (a
dscp?
quoted
quoted
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into queue
1&2
based on VLAN-ID.
I had to look up what AVB even means, but from my current
understanding it doesn't seem right that for non-AVB packets the
driver picks any of the three queues in a random fashion while at the
same time knowing that queue 1 and 2 have a 50% limitation on the
bandwidth. Shouldn't there be some way to prefer queue 0 without
needing the user to set it up or even arbitrarily limiting the number of
queues as proposed above?
quoted
Yes, I think we can. I look into NXP local implementation, there is a
g30bn3ok%3D&reserved=0
This is the version for L5.4 kernel.
Yes, this looks like it could solve the issue. Would you mind preparing a patch to
upstream the change in [1]? I would be happy to test (at least the non-AVB
case) and review.
Yes, I can have a try. I saw this patch has been staying in downstream tree for many years, and I don't know the history.
Anyway, I will try to upstream first to see if anyone has comments.
Thanks, that would be great. Please put me on cc if you send the patch.
Just for the record:
When I set fsl,num-tx-queues = <1>, I do see that the bandwidth-drops don't occur anymore. When I instead apply the queue selection patch from the downstream kernel, I also see that queue 0 is always picked for my untagged traffic. In both cases bandwidth stays just as high as expected (> 900 Mbit/s).
Hi Frieder,
For NXP release kernel, I tested on i.MX8MQ/MM/MP, I can
reproduce on
L5.10, and can't reproduce on L5.4.
quoted
According to your description, you can reproduce this issue
both
L5.4 and
L5.10? So I need confirm with you.
Thanks for looking into this. I could reproduce this on 5.4 and
5.10 but both kernels were official mainline kernels and
**not** from the linux-imx downstream tree.
Ok.
quoted
Maybe there is some problem in the mainline tree and it got
included in the NXP release kernel starting from L5.10?
No, this much looks like a known issue, it should always exist
after adding
AVB support in mainline.
quoted
ENET IP is not a _real_ multiple queues per my understanding,
queue
0 is for
best effort. And the queue 1&2 is for AVB stream whose default
bandwidth fraction is 0.5 in driver. (i.e. 50Mbps for 100Mbps and
500Mbps
for 1Gbps).
quoted
quoted
When transmitting packets, net core will select queues randomly,
which caused the tx bandwidth fluctuations. So you can change to
use single queue if you care more about tx bandwidth. Or you can
refer to NXP internal implementation.
- fsl,num-tx-queues = <3>;
- fsl,num-rx-queues = <3>;
+ fsl,num-tx-queues = <1>;
+ fsl,num-rx-queues = <1>;
status = "disabled";
};
};
I hope this can help you :)
Patching out the queues is probably not the right thing.
for starters... Is there BQL support in this driver? It would be
helpful to have on all queues.
There is no BQL support in this driver, and BQL may improve
throughput
further, but should not be the root cause of this reported issue.
quoted
quoted
Also if there was a way to present it as two interfaces, rather
than one, that would allow for a specific avb device to be presented.
Or:
Is there a standard means of signalling down the stack via the IP
layer (a
dscp?
quoted
quoted
a setsockopt?) that the AVB queue is requested?
AFAIK, AVB is scope of VLAN, so we can queue AVB packets into
queue
1&2
based on VLAN-ID.
I had to look up what AVB even means, but from my current
understanding it doesn't seem right that for non-AVB packets the
driver picks any of the three queues in a random fashion while at
the same time knowing that queue 1 and 2 have a 50% limitation on
the bandwidth. Shouldn't there be some way to prefer queue 0
without needing the user to set it up or even arbitrarily limiting
the number of
queues as proposed above?
quoted
Yes, I think we can. I look into NXP local implementation, there is
a
g30bn3ok%3D&reserved=0
This is the version for L5.4 kernel.
Yes, this looks like it could solve the issue. Would you mind
preparing a patch to upstream the change in [1]? I would be happy to
test (at least the non-AVB
case) and review.
Yes, I can have a try. I saw this patch has been staying in downstream tree
for many years, and I don't know the history.
quoted
Anyway, I will try to upstream first to see if anyone has comments.
Thanks, that would be great. Please put me on cc if you send the patch.
Sure :-)
Best Regards,
Joakim Zhang
Just for the record:
When I set fsl,num-tx-queues = <1>, I do see that the bandwidth-drops don't
occur anymore. When I instead apply the queue selection patch from the
downstream kernel, I also see that queue 0 is always picked for my untagged
traffic. In both cases bandwidth stays just as high as expected (> 900 Mbit/s).