L Liu Xiaodong #
Message details and related content
On Tue, Jun 15, 2021 at 10:13:21PM +0800, Xie Yongji wrote:
This series introduces a framework that makes it possible to implement
software-emulated vDPA devices in userspace. And to make it simple, the
emulated vDPA device's control path is handled in the kernel and only the
data path is implemented in the userspace.
Since the emuldated vDPA device's control path is handled in the kernel,
a message mechnism is introduced to make userspace be aware of the data
path related changes. Userspace can use read()/write() to receive/reply
the control messages.
In the data path, the core is mapping dma buffer into VDUSE daemon's
address space, which can be implemented in different ways depending on
the vdpa bus to which the vDPA device is attached.
In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with
bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma
buffer is reside in a userspace memory region which can be shared to the
VDUSE userspace processs via transferring the shmfd.
The details and our user case is shown below:
------------------------ ------------------------- ----------------------------------------------
| Container | | QEMU(VM) | | VDUSE daemon |
| --------- | | ------------------- | | ------------------------- ---------------- |
| |dev/vdx| | | |/dev/vhost-vdpa-x| | | | vDPA device emulation | | block driver | |
------------+----------- -----------+------------ -------------+----------------------+---------
| | | |
| | | |
------------+---------------------------+----------------------------+----------------------+---------
| | block device | | vhost device | | vduse driver | | TCP/IP | |
| -------+-------- --------+-------- -------+-------- -----+---- |
| | | | | |
| ----------+---------- ----------+----------- -------+------- | |
| | virtio-blk driver | | vhost-vdpa driver | | vdpa device | | |
| ----------+---------- ----------+----------- -------+------- | |
| | virtio bus | | | |
| --------+----+----------- | | | |
| | | | | |
| ----------+---------- | | | |Hi, Yongji Great work! your method is really wise that implements a software IOMMU so that data path gets processed by userspace application efficiently. Sorry, I've just realized your work and patches. I was working on a similar thing aiming to get vhost-user-blk device from SPDK vhost-target to be exported as local host kernel block device. It's diagram is like this: ----------------------------- ------------------------ | ----------------- | --------------------------------------- | <RunC Container> | <<<<<<<<| Shared-Memory |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | | --------- | v | ----------------- | | v | | |dev/vdx| | v | <virtio-local-agent> | | <Vhost-user Target> v | ------------+----------- v | ------------------------ | | --------------------------v------ | | v | |/dev/virtio-local-ctrl| | | | unix socket | |block driver | | | v ------------+---------------- --------+--------------------v--------- | v | | v ------------+----------------v--------------+----------------------------+--------------------v--------| | | block device | v | Misc device | | v | | -------+-------- v --------+------- | v | | | v | | v | | ----------+---------- v | | v | | | virtio-blk driver | v | | v | | ----------+---------- v | | v | | | virtio bus v | | v | | --------+---+------- v | | v | | | v | | v | | | v | | v | | ----------+---------- v ---------+----------- | v | | | virtio-blk device |--<----| virtio-local driver |----------------< v | | ----------+---------- ----------+----------- v | | ---------+--------| -------------------------------------------------------------------------------------| RNIC |--| PCIe |- ----+--- | NVMe | | --------