Re: [LSF/MM/BPF BoF]: extend UBLK to cover real storage hardware

On Fri, Feb 17, 2023 at 11:39:58AM -0500, Stefan Hajnoczi wrote:

On Fri, Feb 17, 2023 at 10:20:45AM +0800, Ming Lei wrote:

quoted

On Thu, Feb 16, 2023 at 12:21:32PM +0100, Andreas Hindborg wrote:

quoted

Ming Lei [off-list ref] writes:

quoted

On Thu, Feb 16, 2023 at 10:44:02AM +0100, Andreas Hindborg wrote:

quoted

Hi Ming,

Ming Lei [off-list ref] writes:

quoted

On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote:

quoted

On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote:

quoted

On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote:

quoted

On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote:

quoted

On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote:

quoted

On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote:

quoted

Hello,

So far UBLK is only used for implementing virtual block device from
userspace, such as loop, nbd, qcow2, ...[1].

I won't be at LSF/MM so here are my thoughts:

Thanks for the thoughts, :-)

quoted

quoted

It could be useful for UBLK to cover real storage hardware too:

- for fast prototype or performance evaluation

- some network storages are attached to host, such as iscsi and nvme-tcp,
the current UBLK interface doesn't support such devices, since it needs
all LUNs/Namespaces to share host resources(such as tag)

Can you explain this in more detail? It seems like an iSCSI or
NVMe-over-TCP initiator could be implemented as a ublk server today.
What am I missing?

The current ublk can't do that yet, because the interface doesn't
support multiple ublk disks sharing single host, which is exactly
the case of scsi and nvme.

Can you give an example that shows exactly where a problem is hit?

I took a quick look at the ublk source code and didn't spot a place
where it prevents a single ublk server process from handling multiple
devices.

Regarding "host resources(such as tag)", can the ublk server deal with
that in userspace? The Linux block layer doesn't have the concept of a
"host", that would come in at the SCSI/NVMe level that's implemented in
userspace.

I don't understand yet...

blk_mq_tag_set is embedded into driver host structure, and referred by queue
via q->tag_set, both scsi and nvme allocates tag in host/queue wide,
that said all LUNs/NSs share host/queue tags, current every ublk
device is independent, and can't shard tags.

Does this actually prevent ublk servers with multiple ublk devices or is
it just sub-optimal?

It is former, ublk can't support multiple devices which share single host
because duplicated tag can be seen in host side, then io is failed.

I have trouble following this discussion. Why can we not handle multiple
block devices in a single ublk user space process?

From this conversation it seems that the limiting factor is allocation
of the tag set of the virtual device in the kernel? But as far as I can
tell, the tag sets are allocated per virtual block device in
`ublk_ctrl_add_dev()`?

It seems to me that a single ublk user space process shuld be able to
connect to multiple storage devices (for instance nvme-of) and then
create a ublk device for each namespace, all from a single ublk process.

Could you elaborate on why this is not possible?

If the multiple storages devices are independent, the current ublk can
handle them just fine.

But if these storage devices(such as luns in iscsi, or NSs in nvme-tcp)
share single host, and use host-wide tagset, the current interface can't
work as expected, because tags is shared among all these devices. The
current ublk interface needs to be extended for covering this case.

Thanks for clarifying, that is very helpful.

Follow up question: What would the implications be if one tried to
expose (through ublk) each nvme namespace of an nvme-of controller with
an independent tag set?

https://lore.kernel.org/linux-block/877cwhrgul.fsf@metaspace.dk/T/#m57158db9f0108e529d8d62d1d56652c52e9e3e67 (local)

quoted

What are the benefits of sharing a tagset across
all namespaces of a controller?

The userspace implementation can be simplified a lot since generic
shared tag allocation isn't needed, meantime with good performance
(shared tags allocation in SMP is one hard problem)

In NVMe, tags are per Submission Queue. AFAIK there's no such thing as
shared tags across multiple SQs in NVMe. So userspace doesn't need an

In reality the max supported nr_queues of nvme is often much less than
nr_cpu_ids, for example, lots of nvme-pci devices just support at most
32 queues, I remembered that Azure nvme supports less(just 8 queues).
That is because queue isn't free in both software and hardware, which
implementation is often tradeoff between performance and cost.

Not mention, most of scsi devices are SQ in which tag allocations from
all CPUs are against single shared tagset.

So there is still per-queue tag allocations from different CPUs which aims
at same queue.

What we discussed are supposed to be generic solution, not something just
for ideal 1:1 mapping device, which isn't dominant in reality.

SMP tag allocator in the first place:
- Each ublk server thread has a separate io_uring context.
- Each ublk server thread has its own NVMe Submission Queue.
- Therefore it's trivial and cheap to allocate NVMe CIDs in userspace
  because there are no SMP concerns.

It isn't even trivial for 1:1 mapping, when any ublk server crashes
global tag will be leaked, and other ublk servers can't use the
leaked tag any more.

Not mention there are lots of SQ device(1:M), or nr_queues
is much less than nr_cpu_ids(N:M N < M). It is pretty easier to
see 1:M or N:M mapping for both nvme and scsi.

The issue isn't tag allocation, it's the fact that the kernel block
layer submits requests to userspace that don't fit into the NVMe
Submission Queue because multiple devices that appear independent from
the kernel perspective are sharing a single NVMe Submission Queue.
Userspace needs a basic I/O scheduler to ensure fairness across devices.
Round-robin for example.

We already have io scheduler for /dev/ublkbN. Also what I proposed is
just to align ublk device with the actual device definition, and so far
tags is the only shared resource in generic io code path.

There are no SMP concerns here either.

No, see above.

So I don't buy the argument that userspace would have to duplicate the
tag allocation code from Linux because that solves a different problem
that the ublk server doesn't have.

If the kernel is aware of tag sharing, then userspace doesn't have to do
(trivial) tag allocation or I/O scheduling. It can simply stuff ublk io

Again, it isn't trivial.

commands into NVMe queues without thinking, which wastes fewer CPU
cycles and is a little simpler.

tag allocation is pretty generic, which is supposed to be done in
kernel, then any userspace isn't supposed to duplicate the
not-trivial implementation.


Thanks, 
Ming