From: Yachen Liu <hidden> Date: 2023-09-14 14:02:36
Hello,
I've noticed a potential issue with USB4NET when used in conjunction
with Linux bridging. I'm uncertain if it's a bug or a configuration
oversight on my part.
In essence, when the device at the other end of Thunderbolt sends
packets using the TSO mechanism (default behavior in macOS), the Linux
thunderbolt0 interface correctly receives this large packet. However,
the packet isn't properly forwarded to another device via bridging.
Detailed Description:
The test environment consists of three systems:
A: Mac Mini (M2): macOS Sonoma 14.0 RC
B: Proxmox VE 8.0. Kernel release: 6.2.16-3-pve, acting as the Host system.
C: Debian. A Guest system running within B.
System A and B are connected via USB4, while System C is a virtual
machine within B. On B, thunderbolt0 and tap102i0 are bridged to
establish a connection between A and C.
During an iperf3 speed test between A and B, I've achieved
bi-directional speeds of around 18Gbps. Between B and C, the speeds
are 100Gbps+ at their peak, with a minimum of 28Gbps.
However, when performing an iperf3 speed test between A and C, the
direction from C to A shows about 18Gbps, but from A to C, the speed
drops to just tens of Kbps, essentially making it unusable.
If tested using UDP, both directions achieve roughly 5Gbps. (I suspect
some buffer issue in macOS limiting the speed).
After various tests and investigations, I found that by setting
macOS's net.inet.tcp.tso to 0 (disabling TSO), speeds from A to C
improved to around 10Gbps.
Packet capture via tcpdump revealed that macOS writes large packets
(over 10000B) to Thunderbolt Networking using TSO. These packets are
correctly captured on thunderbolt0, but are missing from tap102i0,
resulting in significant packet loss.
Since ethtool doesn't support the thunderbolt0 device, further testing
has been hindered.
I'm unsure if this is a bug, or if it could be resolved via
configuration. If more information is needed, I am more than willing
to assist further with tests.
Thank you.
Warm regards,
Yachen Liu
From: Mika Westerberg <mika.westerberg@linux.intel.com> Date: 2023-09-14 15:52:50
Hi,
On Thu, Sep 14, 2023 at 10:02:24PM +0800, Yachen Liu wrote:
Hello,
I've noticed a potential issue with USB4NET when used in conjunction
with Linux bridging. I'm uncertain if it's a bug or a configuration
oversight on my part.
In essence, when the device at the other end of Thunderbolt sends
packets using the TSO mechanism (default behavior in macOS), the Linux
thunderbolt0 interface correctly receives this large packet. However,
the packet isn't properly forwarded to another device via bridging.
Detailed Description:
The test environment consists of three systems:
A: Mac Mini (M2): macOS Sonoma 14.0 RC
B: Proxmox VE 8.0. Kernel release: 6.2.16-3-pve, acting as the Host system.
C: Debian. A Guest system running within B.
System A and B are connected via USB4, while System C is a virtual
machine within B. On B, thunderbolt0 and tap102i0 are bridged to
establish a connection between A and C.
During an iperf3 speed test between A and B, I've achieved
bi-directional speeds of around 18Gbps. Between B and C, the speeds
are 100Gbps+ at their peak, with a minimum of 28Gbps.
However, when performing an iperf3 speed test between A and C, the
direction from C to A shows about 18Gbps, but from A to C, the speed
drops to just tens of Kbps, essentially making it unusable.
If tested using UDP, both directions achieve roughly 5Gbps. (I suspect
some buffer issue in macOS limiting the speed).
After various tests and investigations, I found that by setting
macOS's net.inet.tcp.tso to 0 (disabling TSO), speeds from A to C
improved to around 10Gbps.
Packet capture via tcpdump revealed that macOS writes large packets
(over 10000B) to Thunderbolt Networking using TSO. These packets are
correctly captured on thunderbolt0, but are missing from tap102i0,
resulting in significant packet loss.
Since ethtool doesn't support the thunderbolt0 device, further testing
has been hindered.
I'm unsure if this is a bug, or if it could be resolved via
configuration. If more information is needed, I am more than willing
to assist further with tests.
Thank you for the report. To be honest, I'm not sure how the interface
should in this case and is there perhaps some way for a NIC driver to
pass TSO packets up so that they would be passed in the same way to the
tap interface. Is there any example driver where this works? Like if you
use ethernet instead of TBT/USB4 does that work with TSO packets?
From: Yachen Liu <hidden> Date: 2023-09-14 16:07:59
Hi,
I just did a simple test. When using a standard network card, I can
capture big packets on all three interfaces (eth/br/tap), which can
exceed 50000B at most, and the MTU of the three devices is all just
1500.
I don't know much about Linux's networking architecture. I guess it
should be that when bridging is done, it checks whether the interface
supports GSO and then does special processing. However, for
Thunderbolt devices, no GSO feature was detected so no processing was
done causing problems.
Note: ethtool -k can read whether a general network card has enabled
GSO but it doesn't work for thunderbolt0 interface.
On Thu, Sep 14, 2023 at 11:52 PM Mika Westerberg
[off-list ref] wrote:
Hi,
On Thu, Sep 14, 2023 at 10:02:24PM +0800, Yachen Liu wrote:
quoted
Hello,
I've noticed a potential issue with USB4NET when used in conjunction
with Linux bridging. I'm uncertain if it's a bug or a configuration
oversight on my part.
In essence, when the device at the other end of Thunderbolt sends
packets using the TSO mechanism (default behavior in macOS), the Linux
thunderbolt0 interface correctly receives this large packet. However,
the packet isn't properly forwarded to another device via bridging.
Detailed Description:
The test environment consists of three systems:
A: Mac Mini (M2): macOS Sonoma 14.0 RC
B: Proxmox VE 8.0. Kernel release: 6.2.16-3-pve, acting as the Host system.
C: Debian. A Guest system running within B.
System A and B are connected via USB4, while System C is a virtual
machine within B. On B, thunderbolt0 and tap102i0 are bridged to
establish a connection between A and C.
During an iperf3 speed test between A and B, I've achieved
bi-directional speeds of around 18Gbps. Between B and C, the speeds
are 100Gbps+ at their peak, with a minimum of 28Gbps.
However, when performing an iperf3 speed test between A and C, the
direction from C to A shows about 18Gbps, but from A to C, the speed
drops to just tens of Kbps, essentially making it unusable.
If tested using UDP, both directions achieve roughly 5Gbps. (I suspect
some buffer issue in macOS limiting the speed).
After various tests and investigations, I found that by setting
macOS's net.inet.tcp.tso to 0 (disabling TSO), speeds from A to C
improved to around 10Gbps.
Packet capture via tcpdump revealed that macOS writes large packets
(over 10000B) to Thunderbolt Networking using TSO. These packets are
correctly captured on thunderbolt0, but are missing from tap102i0,
resulting in significant packet loss.
Since ethtool doesn't support the thunderbolt0 device, further testing
has been hindered.
I'm unsure if this is a bug, or if it could be resolved via
configuration. If more information is needed, I am more than willing
to assist further with tests.
Thank you for the report. To be honest, I'm not sure how the interface
should in this case and is there perhaps some way for a NIC driver to
pass TSO packets up so that they would be passed in the same way to the
tap interface. Is there any example driver where this works? Like if you
use ethernet instead of TBT/USB4 does that work with TSO packets?
From: Andrew Lunn <andrew@lunn.ch> Date: 2023-09-15 12:53:50
On Thu, Sep 14, 2023 at 10:02:24PM +0800, Yachen Liu wrote:
Hello,
I've noticed a potential issue with USB4NET when used in conjunction
with Linux bridging. I'm uncertain if it's a bug or a configuration
oversight on my part.
In essence, when the device at the other end of Thunderbolt sends
packets using the TSO mechanism (default behavior in macOS), the Linux
thunderbolt0 interface correctly receives this large packet. However,
the packet isn't properly forwarded to another device via bridging.
Maybe three is a concept mix up here. TSO is TCP Segmentation Offload.
The network stack gives the NIC a large buffer and a template
Ethernet, IP and TCP header. The NIC is then supposed to send MTU
sized frames on the wire, performing segmentation and appending real
headers to each Frame.
We also have Jumbo packets. These are a single frames which are bigger
than he standard 1500 MTU.
You can also combine these. So the stack could for example pass a 64K
buffer to the NIC for TSO, and the MTU could be 8K. So you would
expect to see 8 frames on the wire.
When running tcpdump, what do you actually see? Is TSO actually
happening? Are the frames the size of the MTU or less?
Andrew