Another strange issue:

$ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c
172.17.0.2
------------------------------------------------------------
Client connecting to 172.17.0.2, TCP port 5001
TCP window size:  128 KByte (default)

Increasing the window size may improve the results.  E.g. on my systems I go from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB.

------------------------------------------------------------
[  3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  12.2 GBytes  10.4 Gbits/sec

$ iperf -c 172.17.0.2
------------------------------------------------------------
Client connecting to 172.17.0.2, TCP port 5001
TCP window size:  648 KByte (default)
------------------------------------------------------------
[  3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  14.5 GBytes  12.5 Gbits/sec

rsocket slower than IPoIB ?

This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising.  Does iperf use sendfile()?

My results with iperf (version 2.0.5) over ipoib (default configurations) vary considerably based on the TCP window size.  (Note that this is a 40 Gbps link.)  Results summarized:

TCP window size: 27.9 KByte (default)
 [  3]  0.0-10.0 sec  12.8 GBytes  11.0 Gbits/sec

TCP window size:  416 KByte (WARNING: requested  500 KByte)
[  3]  0.0-10.0 sec  8.19 GBytes  7.03 Gbits/sec

TCP window size:  250 KByte (WARNING: requested  125 KByte)
 [  3]  0.0-10.0 sec  4.99 GBytes  4.29 Gbits/sec

I'm guessing that there are some settings I can change to increase the ipoib performance on my systems.  Using rspreload, I get:

LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103
TCP window size:  512 KByte (default)
[  3]  0.0-10.0 sec  28.3 GBytes  24.3 Gbits/sec

It seems that ipoib bandwidth should be close to rsockets, similar to what you see.  I also don't understand the effect that the TCP window size is having on the results.  The smallest window gives the best bandwidth for ipoib?!

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html