Re: bifurcated driver

Also CC netdev, this thread may interest network folks.

Le 06/11/2014 09:13, Alex Markuze a écrit :

Danny sums up the issue perfectly IMHO.
While both verbs and DPDK aim to provide generic user space networking, the
similarities end there.
verbs and RDMA HW are closely coupled and behave differently then standard
eth nics and are not related to netdev mechanisms.

Or, welcome to this discussion.

Those interested can read the IB spec's (+1K pages) available from
openfabrics*.
*https://www.openfabrics.org/index.php




On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny [off-list ref] wrote:

quoted

I roughly read libibverbs related code and relevant infiniband/rdma
documents, and found though
many concepts in libibverbs looks similar to bifurcated driver, but there
are still lots of differences as
illustrated below based on my understanding:

1) Queue pair defined in RDMA specification are abstract concept, where
the queue pairs term used in
   bifurcated driver are rx/tx queue pairs in the NIC.
2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver
(no NIC control), while libibverbs
   as a user space library rather than driver offloads certain operations
to kernel driver and NIC by invoking
   "verbs" APIs.
3) Libibverbs invokes infiniband specific system calls to allow
user/kernel space communication based on
   "verbs" defined in infiniband/RDMA spec, while bifurcated driver build
on top of af_packet module
   and new socket options to do things like hw queue split-off , map
certain pages on I/O space to user space
   operations, etc.
4) There is a specific embedded MMU unit in Infiniband/RDMA to provides
memory protection, while
   bifurcated driver uses IOMMU rather than NIC to provide memory
protection.

IMHO, libibverbs and corresponding kernel modules/drivers are specifically
designed and implemented for
direct access to RDMA hardware from userspace, and it highly depends on
"verbs" related system calls
supported by infiniband/rdma mechanism in kernel, rather than netdev
mechanism that bifurcated driver
solution depends on.

quoted

-----Original Message-----
From: Vincent JARDIN [mailto:vincent.jardin-pdR9zngts4EAvxtiuMwx3w@public.gmane.org]
Sent: Thursday, November 06, 2014 9:31 AM
To: Zhou, Danny
Cc: Thomas Monjalon; dev-VfR2kkLFssw@public.gmane.org; Fastabend, John R; Or Gerlitz
Subject: Re: [dpdk-dev] bifurcated driver

+Or

On 05/11/2014 23:48, Zhou, Danny wrote:

quoted

Hi Thomas,

Thanks for sharing the links to ibverbs, I will take a close look at

it and compare it to bifurcated driver. My take

quoted

quoted

after a rough review is that idea is very much similar, but bifurcated

driver implementation is generic for any

quoted

quoted

Ethernet device based on existing af_packet mechanism, with extension

of exchanging the messages between

quoted

quoted

user space and kernel space driver.

I have an internal document to summary the pros and cons of below

solutions, except for ibvers, but

quoted

quoted

will be adding it shortly.

- igb_uio
- uio_pci_generic
- VFIO
- bifurcated driver

Short answers to your questions:

quoted

    - upstream status

Adding IOMMU based memory protection and generic descriptor

description support now, into version 2

quoted

quoted

kernel patches.

quoted

    - usable with kernel netdev

af_packet based, and relevant patchset will be submitted to netdev for

sure.

quoted

quoted

quoted

    - usable in a vm

No, it does no coexist with SRIOV for number of reasons. but if you

pass-through a PF to a VM, it works perfect.

quoted

quoted

quoted

    - usable for Ethernet

It could work with all Ethernet NICs, as flow director is available

and NIC driver support new net_ops to split off

quoted

quoted

queue pairs for user space.

quoted

    - hardware requirements

No specific hardware requirements. All mainstream NICs have multiple

qpairs and flow director support.

quoted

quoted

quoted

    - security protection

Leverage IOMMU to provide memory protection on Intel platform. Other

archs provide similar memory protection

quoted

quoted

mechanism, so we only use arch-agnostic DMA memory allocation APIs in

kernel to support memory protection.

quoted

quoted

quoted

    - performance

DPDK native performance on user space queues, as long as drop_en is

enabled to avoid head-of-line blocking.

quoted

quoted

-Danny

quoted

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org]
Sent: Wednesday, November 05, 2014 9:01 PM
To: Zhou, Danny
Cc: dev-VfR2kkLFssw@public.gmane.org; Fastabend, John R
Subject: Re: [dpdk-dev] bifurcated driver

Hi Danny,

2014-10-31 17:36, O'driscoll, Tim:

quoted

Bifurcated Driver (Danny.Zhou-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org)

Thanks for the presentation of bifurcated driver during the community

call.

quoted

quoted

quoted

I asked if you looked at ibverbs and you wanted a link to check.
The kernel module is here:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core

quoted

quoted

quoted

The userspace library:
    http://git.kernel.org/cgit/libs/infiniband/libibverbs.git

Extract from Kconfig:
"
config INFINIBAND_USER_ACCESS
    tristate "InfiniBand userspace access (verbs and CM)"
    select ANON_INODES
    ---help---
      Userspace InfiniBand access support.  This enables the
      kernel side of userspace verbs and the userspace
      communication manager (CM).  This allows userspace processes
      to set up connections and directly access InfiniBand
      hardware for fast-path operations.  You will also need
      libibverbs, libibcm and a hardware driver library from
      <http://www.openfabrics.org/git/>.
"

It seems to be close to the bifurcated driver needs.
Not sure if it can solve the security issues if there is no dedicated

MMU

quoted

quoted

quoted

in the NIC.

I feel we should sum up pros and cons of
    - igb_uio
    - uio_pci_generic
    - VFIO
    - ibverbs
    - bifurcated driver
I suggest to consider these criterias:
    - upstream status
    - usable with kernel netdev
    - usable in a vm
    - usable for ethernet
    - hardware requirements
    - security protection
    - performance

--
Thomas