On Wed, Feb 15, 2023 at 08:51:27AM +0800, Ming Lei wrote:

On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote:

quoted

On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote:

quoted

On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote:

quoted

On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote:

quoted

On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote:

quoted

On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote:

quoted

Hello,

So far UBLK is only used for implementing virtual block device from
userspace, such as loop, nbd, qcow2, ...[1].

I won't be at LSF/MM so here are my thoughts:

Thanks for the thoughts, :-)

quoted

quoted

It could be useful for UBLK to cover real storage hardware too:

- for fast prototype or performance evaluation

- some network storages are attached to host, such as iscsi and nvme-tcp,
the current UBLK interface doesn't support such devices, since it needs
all LUNs/Namespaces to share host resources(such as tag)

Can you explain this in more detail? It seems like an iSCSI or
NVMe-over-TCP initiator could be implemented as a ublk server today.
What am I missing?

The current ublk can't do that yet, because the interface doesn't
support multiple ublk disks sharing single host, which is exactly
the case of scsi and nvme.

Can you give an example that shows exactly where a problem is hit?

I took a quick look at the ublk source code and didn't spot a place
where it prevents a single ublk server process from handling multiple
devices.

Regarding "host resources(such as tag)", can the ublk server deal with
that in userspace? The Linux block layer doesn't have the concept of a
"host", that would come in at the SCSI/NVMe level that's implemented in
userspace.

I don't understand yet...

blk_mq_tag_set is embedded into driver host structure, and referred by queue
via q->tag_set, both scsi and nvme allocates tag in host/queue wide,
that said all LUNs/NSs share host/queue tags, current every ublk
device is independent, and can't shard tags.

Does this actually prevent ublk servers with multiple ublk devices or is
it just sub-optimal?

It is former, ublk can't support multiple devices which share single host
because duplicated tag can be seen in host side, then io is failed.

The kernel sees two independent block devices so there is no issue
within the kernel.

Userspace can do its own hw tag allocation if there are shared storage
controller resources (e.g. NVMe CIDs) to avoid duplicating tags.

Have I missed something?

quoted

quoted

quoted

quoted

quoted

quoted

- SPDK has supported user space driver for real hardware

I think this could already be implemented today. There will be extra
memory copies because SPDK won't have access to the application's memory
pages.

Here I proposed zero copy, and current SPDK nvme-pci implementation haven't
such extra copy per my understanding.

quoted

quoted

So propose to extend UBLK for supporting real hardware device:

1) extend UBLK ABI interface to support disks attached to host, such
as SCSI Luns/NVME Namespaces

2) the followings are related with operating hardware from userspace,
so userspace driver has to be trusted, and root is required, and
can't support unprivileged UBLK device

Linux VFIO provides a safe userspace API for userspace device drivers.
That means memory and interrupts are isolated. Neither userspace nor the
hardware device can access memory or interrupts that the userspace
process is not allowed to access.

I think there are still limitations like all memory pages exposed to the
device need to be pinned. So effectively you might still need privileges
to get the mlock resource limits.

But overall I think what you're saying about root and unprivileged ublk
devices is not true. Hardware support should be developed with the goal
of supporting unprivileged userspace ublk servers.

Those unprivileged userspace ublk servers cannot claim any PCI device
they want. The user/admin will need to give them permission to open a
network card, SCSI HBA, etc.

It depends on implementation, please see

	https://spdk.io/doc/userspace.html

	The SPDK NVMe Driver, for instance, maps the BAR for the NVMe device and
	then follows along with the NVMe Specification to initialize the device,
	create queue pairs, and ultimately send I/O.

The above way needs userspace to operating hardware by the mapped BAR,
which can't be allowed for unprivileged user.

From https://spdk.io/doc/system_configuration.html:

  Running SPDK as non-privileged user

  One of the benefits of using the VFIO Linux kernel driver is the
  ability to perform DMA operations with peripheral devices as
  unprivileged user. The permissions to access particular devices still
  need to be granted by the system administrator, but only on a one-time
  basis. Note that this functionality is supported with DPDK starting
  from version 18.11.

This is what I had described in my previous reply.

My reference on spdk were mostly from spdk/nvme doc.
Just take quick look at spdk code, looks both vfio and direct
programming hardware are supported:

1) lib/nvme/nvme_vfio_user.c
const struct spdk_nvme_transport_ops vfio_ops {
	.qpair_submit_request = nvme_pcie_qpair_submit_request,

Ignore this, it's the userspace vfio-user UNIX domain socket protocol
support. It's not kernel VFIO and is unrelated to what we're discussing.
More info on vfio-user: https://spdk.io/news/2021/05/04/vfio-user/

Not sure, why does .qpair_submit_request point to
nvme_pcie_qpair_submit_request?

The lib/nvme/nvme_vfio_user.c code is for when SPDK connects to a
vfio-user NVMe PCI device. The vfio-user protocol support is not handled
by the regular DPDK/SPDK PCI driver APIs, so the lib/nvme/nvme_pcie.c
doesn't work with vfio-user devices.

However, a lot of the code can be shared with the regular NVMe PCI
driver and that's why .qpair_submit_request points to
nvme_pcie_qpair_submit_request instead of a special version for
vfio-user.

If the vfio-user protocol becomes more widely used for other devices
besides NVMe PCI, then I guess the DPDK/SPDK developers will figure out
a way to move the vfio-user code into the core PCI driver API so that a
single lib/nvme/nvme_pcie.c file works with all PCI APIs (kernel VFIO,
vfio-user, etc). The code was probably structured like this because it's
hard to make those changes and they wanted to get vfio-user NVMe PCI
working quickly.

quoted

quoted

2) lib/nvme/nvme_pcie.c
const struct spdk_nvme_transport_ops pcie_ops = {
	.qpair_submit_request = nvme_pcie_qpair_submit_request
		nvme_pcie_qpair_submit_tracker
			nvme_pcie_qpair_submit_tracker
				nvme_pcie_qpair_ring_sq_doorbell

but vfio dma isn't used in nvme_pcie_qpair_submit_request, and simply
write/read mmaped mmio.

I have only a small amount of SPDK code experienced, so this might be

Me too.

quoted

wrong, but I think the NVMe PCI driver code does not need to directly
call VFIO APIs. That is handled by DPDK/SPDK's EAL operating system
abstractions and device driver APIs.

DMA memory is mapped permanently so the device driver doesn't need to
perform individual map/unmap operations in the data path. NVMe PCI
request submission builds the NVMe command structures containing device
addresses (i.e. IOVAs when IOMMU is enabled).

If IOMMU isn't used, it is physical address of memory.

Then I guess you may understand why I said this way can't be done by
un-privileged user, cause driver is writing memory physical address to
device register directly.

But other driver can follow this approach if the way is accepted.

Okay, I understand now that you were thinking of non-IOMMU use cases.

Stefan