Re: [RFC PATCH 00/28] INFINIBAND NETWORK BLOCK DEVICE (IBNBD)

This series introduces IBNBD/IBTRS kernel modules.

IBNBD (InfiniBand network block device) allows for an RDMA transfer of block IO
over InfiniBand network. The driver presents itself as a block device on client
side and transmits the block requests in a zero-copy fashion to the server-side
via InfiniBand. The server part of the driver converts the incoming buffers back
into BIOs and hands them down to the underlying block device. As soon as IO
responses come back from the drive, they are being transmitted back to the
client.

Hi Jack, Danil and Roman,

I met Danil and Roman last week at Vault, and I think you guys
are awesome, thanks a lot for open-sourcing your work! However,
I have a couple of issues here, some are related to the code and
some are fundamental actually.

- Is there room for this ibnbd? If we were to take every block driver
   that was submitted without sufficient justification, it'd be very
   hard to maintain. What advantage (if any) does this buy anyone over
   existing rdma based protocols (srp, iser, nvmf)? I'm really (*really*)
   not sold on this one...

- To me, the fundamental design that the client side owns a pool of
   buffers that it issues writes too, seems inferior than the
   one taken in iser/nvmf (immediate data). IMO, the ibnbd design has
   scalability issues both in terms of server side resources, client
   side contention and network congestion (on infiniband the latter is
   less severe).

- I suggest that for your next post, you provide a real-life use-case
   where each of the existing drivers can't suffice, and by can't
   suffice I mean that it has a fundamental issue with it, not something
   that requires a fix. With that our feedback can be much more concrete
   and (at least on my behalf) more open to accept it.

- I'm not exactly sure why you would suggest that your implementation
   supports only infiniband if you use rdma_cm for address resolution,
   nor I understand why you emphasize feature (2) below, nor why even
   in the presence of rdma_cm you have ibtrs_ib_path? (confused...)
   iWARP needs a bit more attention if you don't use the new generic
   interfaces though...

- I honestly do not understand why you need *19057* LOC to implement
   a rdma based block driver. Thats almost larger than all of our
   existing block drivers combined... First glance at the code provides
   some explanations, (1) you have some strange code that has no business
   in a block driver like ibtrs_malloc/ibtrs_zalloc (yikes) or
   open-coding various existing logging routines, (2) you are for some
   reason adding a second tag allocation scheme (why?), (3) you are open
   coding a lot of stuff that we added to the stack in the past months...
   (4) you seem to over-layer your code for reasons that I do not
   really understand. And I didn't really look deep at all into the
   code, just to get the feel of it, and it seems like it needs a lot
   of work before it can even be considered upstream ready.

We design and implement this solution based on our need for Cloud Computing,
the key features are:
- High throughput and low latency due to:
1) Only two rdma messages per IO

Where exactly did you witnessed latency that was meaningful by having
another rdma message on the wire? That's only for writes, anyway, and
we have first data bursts for that..

2) Simplified client side server memory management
3) Eliminated SCSI sublayer

That's hardly an advantage given all we are losing without it...

...

Cheers,
Sagi.