Frieder Schrempf reported a TX throuthput issue [1], it happens quite often
that the measured bandwidth in TX direction drops from its expected/nominal
value to something like ~50% (for 100M) or ~67% (for 1G) connections.
[1] https://lore.kernel.org/linux-arm-kernel/421cc86c-b66f-b372-32f7-21e59f9a98bc@kontron.de/
The issue becomes clear after digging into it, Net core would select
queues when transmitting packets. Since FEC have not impletemented
ndo_select_queue callback yet, so it will call netdev_pick_tx to select
queues randomly.
For i.MX6SX ENET IP with AVB support, driver default enables this
feature. According to the setting of QOS/RCMRn/DMAnCFG registers, AVB
configured to Credit-based scheme, 50% bandwidth of each queue 1&2.
With below tests let me think more:
1) With FEC_QUIRK_HAS_AVB quirk, can reproduce TX bandwidth fluctuations issue.
2) Without FEC_QUIRK_HAS_AVB quirk, can't reproduce TX bandwidth fluctuations issue.
The related difference with or w/o FEC_QUIRK_HAS_AVB quirk is that, whether we
program FTYPE field of TxBD or not. As I describe above, AVB feature is
enabled by default. With FEC_QUIRK_HAS_AVB quirk, frames in queue 0
marked as non-AVB, and frames in queue 1&2 marked as AVB Class A&B. It's
unreasonable if frames in queue 1&2 are not required to be time-sensitive.
So when Net core select tx queues ramdomly, Credit-based scheme would work
and lead to TX bandwidth fluctuated. On the other hand, w/o
FEC_QUIRK_HAS_AVB quirk, frames in queue 1&2 are all marked as non-AVB, so
Credit-based scheme would not work.
Till now, how can we fix this TX throughput issue? Yes, please remove
FEC_QUIRK_HAS_AVB quirk if you suffer it from time-nonsensitive networking.
However, this quirk is used to indicate i.MX6SX, other setting depends
on it. So this patch adds a new quirk FEC_QUIRK_HAS_MULTI_QUEUES to
represent i.MX6SX, it is safe for us remove FEC_QUIRK_HAS_AVB quirk
now.
FEC_QUIRK_HAS_AVB quirk is set by default in the driver, and users may
not know much about driver details, they would waste effort to find the
root cause, that is not we want. The following patch is a implementation
to fix it and users don't need to modify the driver.
Tested-by: Frieder Schrempf <redacted>
Reported-by: Frieder Schrempf <redacted>
Signed-off-by: Joakim Zhang <redacted>
---
drivers/net/ethernet/freescale/fec.h | 5 +++++
drivers/net/ethernet/freescale/fec_main.c | 11 ++++++-----
2 files changed, 11 insertions(+), 5 deletions(-)
From: Fugang Duan <redacted>
As we know that AVB is enabled by default, and the ENET IP design is
queue 0 for best effort, queue 1&2 for AVB Class A&B. Bandwidth of each
queue 1&2 set in driver is 50%, TX bandwidth fluctuated when selecting
tx queues randomly with FEC_QUIRK_HAS_AVB quirk available.
This patch adds ndo_select_queue callback to select queues for
transmitting to fix this issue. It will always return queue 0 if this is
not a vlan packet, and return queue 1 or 2 based on priority of vlan
packet.
You may complain that in fact we only use single queue for trasmitting
if we are not targeted to VLAN. Yes, but seems we have no choice, since
AVB is enabled when the driver probed, we can't switch this feature
dynamicly. After compare multiple queues to single queue, TX throughput
almost no improvement.
One way we can implemet is to configure the driver to multiple queues
with Round-robin scheme by default. Then add ndo_setup_tc callback to
enable/disable AVB feature for users. Unfortunately, ENET AVB IP seems
not follow the standard 802.1Qav spec. We only can program
DMAnCFG[IDLE_SLOPE] field to calculate bandwidth fraction. And idle
slope is restricted to certain valus (a total of 19). It's far away from
CBS QDisc implemented in Linux TC framework. If you strongly suggest to do
this, I think we only can support limited numbers of bandwidth and reject
others, but it's really urgly and wried.
With this patch, VLAN tagged packets route to queue 0/1/2 based on vlan
priority; VLAN untagged packets route to queue 0.
Tested-by: Frieder Schrempf <redacted>
Reported-by: Frieder Schrempf <redacted>
Signed-off-by: Fugang Duan <redacted>
Signed-off-by: Joakim Zhang <redacted>
---
drivers/net/ethernet/freescale/fec_main.c | 32 +++++++++++++++++++++++
1 file changed, 32 insertions(+)
This patch set intends to fix TX bandwidth fluctuations, any feedback would be appreciated.
---
ChangeLogs:
V1: remove RFC tag, RFC discussions please turn to below:
https://lore.kernel.org/lkml/YK0Ce5YxR2WYbrAo@lunn.ch/T/
V2: change functions to be static in this patch set. And add the
t-b tag.
V2: change functions to be static in this patch set. And add the
t-b tag.
Please fix these warnings in patch #2:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchw
ork.hopto.org%2Fstatic%2Fnipa%2F498729%2F12315211%2Fbuild_allmodconfi
g_warn%2Fsummary&data=04%7C01%7Cqiangqing.zhang%40nxp.com%
7C45b786c85a294b3ea9ec08d92d17090c%7C686ea1d3bc2b4c6fa92cd99c5c30
1635%7C0%7C0%7C637590399225645759%7CUnknown%7CTWFpbGZsb3d8ey
JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
7C1000&sdata=eb4Ujx2FKOnu5M41j1RnPLCBoEoGfQiCO%2BO5MzIsXyE
%3D&reserved=0
I can't reproduce these warnings with " make ARCH=arm64 allmodconfig", could you please show me the command you used? Thanks.
Best Regards,
Joakim Zhang
-----Original Message-----
From: Andrew Lunn <andrew@lunn.ch>
Sent: 2021年6月15日 20:28
To: Joakim Zhang <redacted>
Cc: David Miller <davem@davemloft.net>; kuba@kernel.org;
frieder.schrempf@kontron.de; netdev@vger.kernel.org;
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V2 net-next 0/2] net: fec: fix TX bandwidth fluctuations
quoted
I can't reproduce these warnings with " make ARCH=arm64 allmodconfig",
could you please show me the command you used? Thanks.
Try adding W=1
Thanks.
I try below build options, also can't reproduce this issue, so really don't know how to fix it.
make ARCH=arm64 distclean
make ARCH=arm64 allmodconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1 / make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=2 / make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=3
I saw many unrelated warnings...
Best Regards,
Joakim Zhang
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-06-16 13:06:30
I try below build options, also can't reproduce this issue, so really don't know how to fix it.
make ARCH=arm64 distclean
make ARCH=arm64 allmodconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1 / make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=2 / make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=3
I saw many unrelated warnings...
Then it could be sparse. Install sparse and use C=1.
Andrew
-----Original Message-----
From: Andrew Lunn <andrew@lunn.ch>
Sent: 2021年6月16日 21:06
To: Joakim Zhang <redacted>
Cc: David Miller <davem@davemloft.net>; kuba@kernel.org;
frieder.schrempf@kontron.de; netdev@vger.kernel.org;
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V2 net-next 0/2] net: fec: fix TX bandwidth fluctuations
quoted
I try below build options, also can't reproduce this issue, so really don't know
how to fix it.
quoted
make ARCH=arm64 distclean
make ARCH=arm64 allmodconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1 / make -j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=2 / make -j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=3
quoted
I saw many unrelated warnings...
Then it could be sparse. Install sparse and use C=1.
After applying the patch #2, I tried to use C=1 yesterday, I double check it today, still no warnings. Anything I missing?
$ make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1,C=1
CALL scripts/atomic/check-atomics.sh
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
CHK kernel/kheaders_data.tar.xz
CC [M] drivers/net/ethernet/freescale/fec_main.o
LD [M] drivers/net/ethernet/freescale/fec.o
MODPOST modules-only.symvers
GEN Module.symvers
CC [M] drivers/net/ethernet/freescale/fec.mod.o
LD [M] drivers/net/ethernet/freescale/fec.ko
Best Regards,
Joakim Zhang
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-06-17 13:04:26
On Thu, Jun 17, 2021 at 11:40:58AM +0000, Joakim Zhang wrote:
Hi Andrew,
quoted
-----Original Message-----
From: Andrew Lunn <andrew@lunn.ch>
Sent: 2021年6月16日 21:06
To: Joakim Zhang <redacted>
Cc: David Miller <davem@davemloft.net>; kuba@kernel.org;
frieder.schrempf@kontron.de; netdev@vger.kernel.org;
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V2 net-next 0/2] net: fec: fix TX bandwidth fluctuations
quoted
I try below build options, also can't reproduce this issue, so really don't know
how to fix it.
quoted
make ARCH=arm64 distclean
make ARCH=arm64 allmodconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1 / make -j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=2 / make -j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=3
quoted
I saw many unrelated warnings...
Then it could be sparse. Install sparse and use C=1.
After applying the patch #2, I tried to use C=1 yesterday, I double check it today, still no warnings. Anything I missing?
$ make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1,C=1
CALL scripts/atomic/check-atomics.sh
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
CHK kernel/kheaders_data.tar.xz
CC [M] drivers/net/ethernet/freescale/fec_main.o
LD [M] drivers/net/ethernet/freescale/fec.o
MODPOST modules-only.symvers
GEN Module.symvers
CC [M] drivers/net/ethernet/freescale/fec.mod.o
LD [M] drivers/net/ethernet/freescale/fec.ko
Best Regards,
Joakim Zhang
-----Original Message-----
From: Andrew Lunn <andrew@lunn.ch>
Sent: 2021年6月17日 21:04
To: Joakim Zhang <redacted>
Cc: David Miller <davem@davemloft.net>; kuba@kernel.org;
frieder.schrempf@kontron.de; netdev@vger.kernel.org;
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V2 net-next 0/2] net: fec: fix TX bandwidth fluctuations
On Thu, Jun 17, 2021 at 11:40:58AM +0000, Joakim Zhang wrote:
quoted
Hi Andrew,
quoted
-----Original Message-----
From: Andrew Lunn <andrew@lunn.ch>
Sent: 2021年6月16日 21:06
To: Joakim Zhang <redacted>
Cc: David Miller <davem@davemloft.net>; kuba@kernel.org;
frieder.schrempf@kontron.de; netdev@vger.kernel.org;
linux-kernel@vger.kernel.org
Subject: Re: [PATCH V2 net-next 0/2] net: fec: fix TX bandwidth
fluctuations
quoted
I try below build options, also can't reproduce this issue, so
really don't know
how to fix it.
quoted
make ARCH=arm64 distclean
make ARCH=arm64 allmodconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1 / make
-j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=2 / make -j8
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=3
quoted
I saw many unrelated warnings...
Then it could be sparse. Install sparse and use C=1.
After applying the patch #2, I tried to use C=1 yesterday, I double check it
today, still no warnings. Anything I missing?
quoted
$ make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- W=1,C=1
CALL scripts/atomic/check-atomics.sh
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
CHK kernel/kheaders_data.tar.xz
CC [M] drivers/net/ethernet/freescale/fec_main.o
LD [M] drivers/net/ethernet/freescale/fec.o
MODPOST modules-only.symvers
GEN Module.symvers
CC [M] drivers/net/ethernet/freescale/fec.mod.o
LD [M] drivers/net/ethernet/freescale/fec.ko
Best Regards,
Joakim Zhang
quoted
Andrew
If you look at
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchw
ork.hopto.org%2Fstatic%2Fnipa%2F498729%2F12315211%2Fbuild_32bit%2Fst
dout&data=04%7C01%7Cqiangqing.zhang%40nxp.com%7Ce966882aff15
41ed54c108d931906ad2%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
%7C637595318610458379%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&s
data=0o34i6pkvqBN1hPApz5Ja8CHjtqn8iwQB8whxg0p8Rw%3D&reserved
=0
you see:
Kernel: arch/x86/boot/bzImage is ready (#9396)
So it is building for 32 bit x86. So try
make -j8 ARCH=i386 W=1 C=1
Assuming your host is an x86 machine.
Much thanks.
$ git am 0002-net-fec-add-ndo_select_queue-to-fix-TX-bandwidth-flu.patch
Applying: net: fec: add ndo_select_queue to fix TX bandwidth fluctuations
$ make -j8 ARCH=i386 W=1,C=1
CALL scripts/atomic/check-atomics.sh
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
CHK kernel/kheaders_data.tar.xz
CC [M] drivers/net/ethernet/freescale/fec_main.o
LD [M] drivers/net/ethernet/freescale/fec.o
MODPOST modules-only.symvers
Kernel: arch/x86/boot/bzImage is ready (#1)
GEN Module.symvers
CC [M] drivers/net/ethernet/freescale/fec.mod.o
LD [M] drivers/net/ethernet/freescale/fec.ko
Unfortunately, I still can't see warnings after changing to build x86 image, a strange phenomenon, "W=1 C=1" seems not work, "W=1,C=1" can work.
I also save all of the build logs to double check, there is no build warnings related to FEC driver.
Best Regards,
Joakim Zhang