On 26/09/2023 09:24, Alexandra Winter wrote:

On 25.09.23 17:18, Dust Li wrote:

quoted

quoted

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module
may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC.
If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be
we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
    Currently, If we want to disable a SMC-R device to communicate with
    others, we need to refer to 'ip link set dev xxx down' to disable the
    netdevice, then Infiniband subsystem will notify SMC that the state of
    the IB device has changed. We cannot explicitly choose not to use some
    specific IB/RoCE devices without disable totally.
    If the loopback device need to support enable/disable itself, I
    think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
    Now, we have to relay on IB/RoCE devices' hardware statistics to see
    how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC
and userspace RDMA applications. If SMC-R and userspace RDMA applications
run at the same time, we can't enable the device to run userspace RDMA
applications while block it from running SMC. For statistics, we cannot
tell how many packets/bytes were sent by SMC and how many were sent by
userspace RDMA applications.

So I think those are better to support in the SMC layer.

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling
of these requirements in the smc layer would be best. Especially, if we want
to add virtio-ism support soon. There we will face the same issues again.
Let's hear what others think about this.

Thanks you Sandy for bringing it up and Dust Li & Wen Gu for your thoughts.
I agree that such a runtime switch is needed and also that this generic 
handling would be good in the smc layer.