From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2021-05-14 13:00:23
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.
So, the maximum that userspace can set appears to be 9888. If this is
set, then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888
So clearly this is too large, and we should not be allowing userspace
to set this large a MTU.
At this point, it seems to be impossible to regain the previous speed of
the interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes
[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
[ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes
[ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender
[ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiver
Whereas before the test, it was:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec
[ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec
[ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec
[ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec
[ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec
[ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec
[ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver
(this is on the server end of iperf3, the others are the client end,
but the results were pretty very similar to that.)
So, clearly something bad has happened to the buffer management as a
result of raising the MTU so high.
As the end which has suffered this issue is the mcbin VM host, I'm not
currently in a position I can reboot it without cause major disruption
to my network. However, thoughts on this (and... can others reproduce
it) would be useful.
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
From: Marcin Wojtas <hidden> Date: 2021-05-14 14:26:06
Hi Russell,
pt., 14 maj 2021 o 15:00 Russell King (Oracle) [off-list ref]
napisał(a):
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.
So, the maximum that userspace can set appears to be 9888. If this is
set, then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888
So clearly this is too large, and we should not be allowing userspace
to set this large a MTU.
At this point, it seems to be impossible to regain the previous speed of
the interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes
[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
[ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes
[ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender
[ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiver
Whereas before the test, it was:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec
[ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec
[ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec
[ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec
[ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec
[ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec
[ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver
(this is on the server end of iperf3, the others are the client end,
but the results were pretty very similar to that.)
So, clearly something bad has happened to the buffer management as a
result of raising the MTU so high.
As the end which has suffered this issue is the mcbin VM host, I'm not
currently in a position I can reboot it without cause major disruption
to my network. However, thoughts on this (and... can others reproduce
it) would be useful.
Thank your for the information. I will take a look after the weekend.
To be aligned - what exactly kernel baseline are you using?
Best regards,
Marcin
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-05-14 14:35:18
On Fri, May 14, 2021 at 02:00:18PM +0100, Russell King (Oracle) wrote:
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.
From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2021-05-14 17:19:51
On Fri, May 14, 2021 at 04:25:48PM +0200, Marcin Wojtas wrote:
Thank your for the information. I will take a look after the weekend.
To be aligned - what exactly kernel baseline are you using?
That was with 5.11 with these additional mvpp2 patches:
net: mvpp2: add TX FC firmware check
net: mvpp2: set 802.3x GoP Flow Control mode
net: mvpp2: add PPv23 RX FIFO flow control
net: mvpp2: add BM protection underrun feature support
net: mvpp2: add ethtool flow control configuration support
net: mvpp2: add RXQ flow control configurations
net: mvpp2: enable global flow control
net: mvpp2: add FCA RXQ non occupied descriptor threshold
net: mvpp2: add FCA periodic timer configurations
net: mvpp2: increase BM pool and RXQ size
net: mvpp2: add PPv23 version definition
net: mvpp2: always compare hw-version vs MVPP21
net: mvpp2: add CM3 SRAM memory map
dts: marvell: add CM3 SRAM memory to cp11x ethernet device tree
doc: marvell: add CM3 address space and PPv2.3 description
net: marvell: Fixed two spellings,controling to controlling and oen to one
net: mvpp2: prs: improve ipv4 parse flow
I'll also try to work out what's happening, but I think we need to find
out what the correct value for dev->max_mtu should be. That's all rather
convoluted:
dev->max_mtu = MVPP2_BM_JUMBO_PKT_SIZE;
#define MVPP2_BM_JUMBO_PKT_SIZE MVPP2_RX_MAX_PKT_SIZE(MVPP2_BM_JUMBO_FRAME_SIZE)
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
#define MVPP2_RX_MAX_PKT_SIZE(total_size) \
((total_size) - MVPP2_SKB_HEADROOM - MVPP2_SKB_SHINFO_SIZE)
The maximum settable MTU on eth0 (9888) disagrees with the comment
"frame size 9856" by 32 bytes.
I haven't checked to see whether 9856 works yet, because that will
first need me to reboot the machine... which I'll do over the weekend.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2021-05-14 17:21:42
On Fri, May 14, 2021 at 04:35:12PM +0200, Andrew Lunn wrote:
On Fri, May 14, 2021 at 02:00:18PM +0100, Russell King (Oracle) wrote:
quoted
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.
Oh, there's more values given elsewhere that disagree, see my reply to
Marcin. I would not be surprised if this was all "confused" about
what the proper value is. Certainly the comment you mention above
disagrees with what is in mvpp2.h
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
From: Stefan Chulski <hidden> Date: 2021-05-18 06:09:30
-----Original Message-----
From: Russell King <linux@armlinux.org.uk>
Sent: Friday, May 14, 2021 4:00 PM
To: Stefan Chulski <redacted>
Cc: Marcin Wojtas <redacted>; netdev@vger.kernel.org
Subject: [EXT] mvpp2: incorrect max mtu?
External Email
----------------------------------------------------------------------
Hi all,
While testing out the 10G speeds on my Macchiatobin platforms, the first
thing I notice is that they only manage about 1Gbps at a MTU of 1500.
As expected, this increases when the MTU is increased - a MTU of 9000
works, and gives a useful performance boost.
Then comes the obvious question - what is the maximum MTU.
#define MVPP2_BM_JUMBO_FRAME_SIZE 10432 /* frame size 9856 */
So, one may assume that 9856 is the maximum. However:
# ip li set dev eth0 mtu 9888
# ip li set dev eth0 mtu 9889
Error: mtu greater than device maximum.
So, the maximum that userspace can set appears to be 9888. If this is set,
then, while running iperf3, we get:
mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error),
size=9888
So clearly this is too large, and we should not be allowing userspace to set
this large a MTU.
At this point, it seems to be impossible to regain the previous speed of the
interface by lowering the MTU. Here is a MTU of 9000:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec 40 17.5 KBytes
[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 2.00-3.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 3.00-4.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
[ 5] 4.00-5.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 5.00-6.00 sec 1.28 MBytes 10.7 Mbits/sec 39 8.74 KBytes
[ 5] 6.00-7.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 39 8.74 KBytes
[ 5] 8.00-9.00 sec 1.13 MBytes 9.45 Mbits/sec 36 17.5 KBytes
[ 5] 9.00-10.00 sec 1.13 MBytes 9.45 Mbits/sec 39 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.9 MBytes 9.99 Mbits/sec 379 sender
[ 5] 0.00-10.00 sec 11.7 MBytes 9.80 Mbits/sec receiver
Whereas before the test, it was:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 773 MBytes 6.49 Gbits/sec
[ 5] 3.00-4.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 4.00-5.00 sec 779 MBytes 6.54 Gbits/sec
[ 5] 5.00-6.00 sec 784 MBytes 6.58 Gbits/sec
[ 5] 6.00-7.00 sec 777 MBytes 6.52 Gbits/sec
[ 5] 7.00-8.00 sec 774 MBytes 6.50 Gbits/sec
[ 5] 8.00-9.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 9.00-10.00 sec 774 MBytes 6.49 Gbits/sec
[ 5] 10.00-10.00 sec 3.07 MBytes 5.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.47 GBytes 6.41 Gbits/sec receiver
(this is on the server end of iperf3, the others are the client end, but the
results were pretty very similar to that.)
So, clearly something bad has happened to the buffer management as a
result of raising the MTU so high.
As the end which has suffered this issue is the mcbin VM host, I'm not
currently in a position I can reboot it without cause major disruption to my
network. However, thoughts on this (and... can others reproduce
it) would be useful.
Thanks.
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released).
Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.
Stefan.
From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2021-05-18 09:41:48
On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released).
Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.
So this must mean that setting dev->max_mtu is incorrect.
From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
same value that dev->max_mtu is set to, but max_mtu is the data
payload size in the ethernet frame, which doesn't include the hardware
ethernet header.
So, should max_mtu be set to 14 bytes less? Or should it be set to
9856? Less 14 bytes? Or what?
It is really confusing that we have these definitions that state e.g.
that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
It's not clear why it's different like that - why the additional 576
octets.
All of this could do with some explanation in the driver - would it be
possible to add some kind of documentation, or at least make the
definitions around packet and frame size more understandable please?
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
From: Stefan Chulski <hidden> Date: 2021-05-18 10:26:09
-----Original Message-----
From: Russell King <linux@armlinux.org.uk>
Sent: Tuesday, May 18, 2021 12:42 PM
To: Stefan Chulski <redacted>
Cc: Marcin Wojtas <redacted>; netdev@vger.kernel.org
Subject: Re: [EXT] mvpp2: incorrect max mtu?
On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
quoted
Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size
and it drained buffer pool(Buffers never released).
quoted
Received packet should be less than value set in
MVPP2_POOL_BUF_SIZE_REG for long pool.
So this must mean that setting dev->max_mtu is incorrect.
From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
same value that dev->max_mtu is set to, but max_mtu is the data payload
size in the ethernet frame, which doesn't include the hardware ethernet
header.
So, should max_mtu be set to 14 bytes less? Or should it be set to 9856? Less
14 bytes? Or what?
Yes, look like dev->max_mtuis incorrect. It should be 9856, MVPP2_POOL_BUF_SIZE_REG, Pool Buffer Size is in 32 units.
But before changing it I prefer to test different sizes. Currently, I don't have a connected board.
I will connect board and perform some tests. JUMBO_FRAME_SIZE is the size of the buffer, but packet offset should be taken into account.
It is really confusing that we have these definitions that state e.g.
that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
It's not clear why it's different like that - why the additional 576 octets.
All of this could do with some explanation in the driver - would it be possible
to add some kind of documentation, or at least make the definitions around
packet and frame size more understandable please?
Thanks.
Yes, definition or comments should be improved.
Regards,
Stefan.
From: Stefan Chulski <hidden> Date: 2021-05-23 10:54:18
quoted
-----Original Message-----
From: Russell King <linux@armlinux.org.uk>
Sent: Tuesday, May 18, 2021 12:42 PM
To: Stefan Chulski <redacted>
Cc: Marcin Wojtas <redacted>; netdev@vger.kernel.org
Subject: Re: [EXT] mvpp2: incorrect max mtu?
On Tue, May 18, 2021 at 06:09:12AM +0000, Stefan Chulski wrote:
quoted
Look like PPv2 tried scatter frame since it was larger than Jumbo
buffer size
and it drained buffer pool(Buffers never released).
quoted
Received packet should be less than value set in
MVPP2_POOL_BUF_SIZE_REG for long pool.
So this must mean that setting dev->max_mtu is incorrect.
From what I can see, the value programmed into that register would be
MVPP2_BM_JUMBO_PKT_SIZE which I believe is 9888. This is currently the
same value that dev->max_mtu is set to, but max_mtu is the data
payload size in the ethernet frame, which doesn't include the hardware
ethernet header.
So, should max_mtu be set to 14 bytes less? Or should it be set to
9856? Less
14 bytes? Or what?
Yes, look like dev->max_mtuis incorrect. It should be 9856,
MVPP2_POOL_BUF_SIZE_REG, Pool Buffer Size is in 32 units.
But before changing it I prefer to test different sizes. Currently, I don't have a
connected board.
I will connect board and perform some tests. JUMBO_FRAME_SIZE is the size
of the buffer, but packet offset should be taken into account.
quoted
It is really confusing that we have these definitions that state e.g.
that JUMBO_FRAME_SIZE is 10432 but the frame size comment says 9856.
It's not clear why it's different like that - why the additional 576 octets.
All of this could do with some explanation in the driver - would it be
possible to add some kind of documentation, or at least make the
definitions around packet and frame size more understandable please?
Thanks.
Yes, definition or comments should be improved.
Regards,
Stefan.
We probably should split this to 3 issues.
1. BM drain issue. I can post patch witch should fix it. So after lowering the MTU you won't have issue.
2. Regarding MTU max, here we actually talking about max MRU not MTU.
MRU set by mvpp2_xlg_max_rx_size_set/mvpp2_gmac_max_rx_size_set port->pkt_size - MVPP2_MH_SIZE(2 Bytes).
port->pkt_size = MTU + MVPP2_MH_SIZE(2 Bytes) + VLAN tag (4 Bytes) + ETH_HLEN(14 Bytes) + ETH_FCS_LEN(4 Bytes) and from some reason aligned to cache_line_size().
So if we set MTU = 9888, we actually get MRU 9918.
3. Fix all max MTU comments in code.
Regards,
Stefan.