On 2025/3/20 0:48, Jason Gunthorpe wrote:

On Fri, Mar 07, 2025 at 01:01:50AM +0200, Nikolay Aleksandrov wrote:

quoted

Hi all,
This patch-set introduces minimal Ultra Ethernet driver infrastructure and
the lowest Ultra Ethernet sublayer - the Packet Delivery Sublayer (PDS),
which underpins the entire communication model of the Ultra Ethernet
Transport[1] (UET). Ultra Ethernet is a new RDMA transport designed for
efficient AI and HPC communication.

I was away while this discussion happened so I've gone through and
read the threads, looked at the patches and I don't think I've changed
my view since I talked to Enfabrica privately on this topic almost a
year ago.

I do not agree with creating a new subsystem (or whatever you are
calling drivers/ultraeth) for a single RDMA protocol and see nothing
new here to change my mind. I would likely NAK the direction I see in
this RFC, as I have other past attempts to build RDMA HW interfaces
outside of the RDMA subystem.

Since none of that past discussion seems to have been acknowledged or
rebutted in this series I will repeat the main points:

1) I'm aware of something like 5-7 new protocols that are competing
   for the same market as Ultra Ethernet. We can't give everyone and
   their dog a new subsystem (or whatever) and all the maintainability
   negatives that come with that. As a matter of maintainability we
   need to see consolidation here, not fragmentation!

   Yes, UE is a consortium driven standard, which is unique and a big
   positive, but I don't believe anyone can say for certain what
   direction the industry is going to go in. Many consortium standards
   have failed to get adoption in the past even with a large number of
   member companies.

   Nor can we know what concepts in UE are going to be copied into
   other competing RDMA transports. See my other remarks on job key
   for an example. Prematurely siloing stuff in drivers/ultraeth is
   very much the wrong technical direction for maintainability.

   That said, I think UE should be in the kernel and have a fair
   chance to compete for market share. Just in a maintainable and
   appropriate way while the industry evolves.

2) Due to the above, I'm pretty confident we will see RDMA NICs
   supporting a lot of different protocols. In fact they already do.

   From a kernel maintainability perspective we really want one RDMA
   driver leveraging as much common infrastructure between the
   protocols as possible. We do not want to see a single HW driver
   further split up needlessly to other subsystems, that would be a
   big maintainability downside.

   To put a clear point on this, mlx5 has been gaining new protocols
   and fitting into the existing driver model for a number of years
   now. In fact there is speculation that UE could be implemented in
   mlx5 RDMA with minimal kernel changes. There would be no reason to
   try to mess up the driver to also interact with this stuff in
   drivers/ultraeth as seems to be proposed here.

   I think other HW will be similar. UE isn't so radically different
   that every HW path will need to diverge from classical RDMA. Nor is
   is so dissimilar to other competing proposals. We don't want
   artificial differences we want to create things that can be re-used
   when appropriate.

   Leon's response to Bart is correct, we already have similar
   examples of almost everything UE does. Bart is also correct that
   verbs would be a PITA, but RDMA userspace has moved beyond verbs
   limitations years ago now. Alot of mlx5 stuff is not using verbs
   today, for instance. EFA and other examples use extensive stuff
   beyond verbs.

Regarding to reuse the existing rdma subsystem for a new protocol:
Currently EFA seems to be layering a RDM layer on top of the SRD
transport layer, see [1], and RDM layer is implemented by software in
the libfabric while SRD seems to be implemented by hardware, which
provides 'Scalable Reliable Datagram' service through the QP type
of EFA_QP_DRIVER_TYPE_SRD.

I am not sure if layers like SRD and RDM are clean layering from
protocol design perspective.
But if the hardware implement both SRD and RDM layer in hardware,
then there might be two types of object need managing, SRD object
might be shared between different applications, and RDM object
need to be created based on a SRD object.

As the existing rdma subsystem doesn't seems to support the above
use case yet and as we are discussing a possible new subsystem or
updating existing subsystem to support new protocol here, it would
be good to discuss if it is possible to support the above case or
another new subsystem is needed for that use case too.

1. https://github.com/ofiwg/libfabric/blob/main/prov/efa/docs/efa_rdm_protocol_v4.md