Re: SMC-R throughput drops for specific message sizes

Dear Gerd,

Thank you for directing this matter to the most relevant group and 
individuals. Your support is greatly appreciated. We're actively delving deeper into the issue to gain further insights as well.
I'll ensure to keep this thread updated with any new findings as we progress. 
Should you need any additional information, please feel free to reach out.

Mit freundlichen Grüßen / Best regards

Alexandros Nikolaou


Bosch Service Solutions Magdeburg GmbH | Otto-von-Guericke-Str. 13 | 39104 Magdeburg | GERMANY | [www.boschservicesolutions.com]www.boschservicesolutions.com
Alexandros.Nikolaou@de.bosch.com


Sitz: Magdeburg, Registergericht: Amtsgericht Stendal, HRB 24039

Geschäftsführung: Robert Mulatz, Daniel Meyer

From: Wen Gu <guwen@linux.alibaba.com>
Sent: Wednesday, December 13, 2023 14:38
To: Gerd Bayer <gbayer@linux.ibm.com>; Nikolaou Alexandros (SO/PAF1-Mb) <redacted>; D . Wythe <alibuda@linux.alibaba.com>; Tony Lu <tonylu@linux.alibaba.com>; Nils Hoppmann <redacted>
Cc: linux-s390@vger.kernel.org <redacted>; netdev <redacted>; Wenjia Zhang <redacted>; Jan Karcher <redacted>; Dust Li <dust.li@linux.alibaba.com>
Subject: Re: SMC-R throughput drops for specific message sizes
 


On 2023/12/13 20:17, Gerd Bayer wrote:

Hi Nikolaou,

thank you for providing more details about your setup.

On Wed, 2023-12-06 at 15:28 +0000, Nikolaou Alexandros (SO/PAF1-Mb)
wrote:

quoted

Dear Wenjia,

while Wenjia is out, I'm writing primarily to getting some more folks'
attention to this topic. Furthermore, I'm moving the discussion to the
netdev mailing list where SMC discussions usually take place.

quoted

Thanks for getting back to me. Some further details on the
experiments are:
 
- The tests had been conducted on a one-to-one connection between two
Mellanox-powered (mlx5, ConnectX-5) PCs.
- Attached you may find the client log of the qperf output. You may
notice that for the majority of message size values, the bandwidth is
around 3.2GB/s which matches the maximum throughput of the
mellanox NICs.
According to a periodic regular pattern though, with the first
occurring at a message size of 473616 – 522192 (with a step of
12144kB), the 3.2GB/s throughput drops substantially. The
corresponding commands for these drops are
server: smc_run qperf
client: smc_run qperf -v -uu -H worker1 -m 473616 tcp_bw
- Our smc version (3E92E1460DA96BE2B2DDC2F, smc-tools-1.2.2) does not
provide us with the smcr info, smc_rnics -a and smcr -d
stats commands. As an alternative, you may also find attached the
output of ibv_devinfo -v.
- Buffer size:
sudo sysctl -w net.ipv4.tcp_rmem="4096 1048576 6291456"
sudo sysctl -w net.ipv4.tcp_wmem="4096 1048576 6291456"
- MTU size: 9000
 
Should you require further information, please let me know.

Wenjia and I belong to a group of Linux on Z developers that maintains
the SMC protocol on s390 mainframe systems. Nils Hoppmann is our expert
for performance and might be able to shed some light on his experiences
with throughput drops for particular SMC message sizes. Our experience
is heavily biased towards IBM Z systems, though - with their distinct
cache and PCI root-complex hardware designs.

Over the last few years there's a group around D. Wythe, Wen Gu and
Tony Lu who adopted and extended the SMC protocol for use-cases on x86
architectures. I address them here explicitly, soliciting feedback on
their experiences.

Certainly. Our team will take a closer look into this matter as well.
We intend to review the thread thoroughly and conduct an analysis within
our environment. Updates and feedback will be provided in this thread.

All in all there are several moving parts involved here, that could
play a role:
- firmware level of your Mellanox/NVidia NICs,
- platform specific hardware designs re. cache and root-complexes,
interrupt distribution, ...
- exact code level of the device drivers and the SMC protocol

This is just a heads-up, that there may be requests to try things with
newer code levels ;)

Thank you,
Gerd

--
Gerd Bayer
Linux on IBM Z Development - IBM Germany R&D