On Wed, Aug 31, 2022 at 02:31:12PM +0800, Ziyang Zhang wrote:

On 2022/8/30 23:23, Stefan Hajnoczi wrote:

quoted

On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:

quoted

+- UBLK_IO_NEED_GET_DATA
+  ublksrv pre-allocates IO buffer for each IO at default, any new project
+  should use this IO buffer to communicate with ublk driver. But existed
+  project may not work or be changed to in this way, so add this command
+  to provide chance for userspace to use its existed buffer for handling
+  IO.

I find it hard to understand this paragraph. It seems the
UBLK_IO_NEED_GET_DATA command allows userspace to set up something
related to IO buffers. What exactly does this command do?

Let me explain UBLK_IO_NEED_GET_DATA since it is designed by myself.

Without UBLK_IO_NEED_GET_DATA, ublk_drv will copy data from biovecs
into a pre-allocated buffer(addr is passed with the last COMMIT_AMD_FETCH ioucmd)
while processing a WRITE request. Please consider two cases:

(1)  if the backend(such as a dist-storage system using RPC) provides the data
     buffer, it has to provide the buffer IN ADVANCE(before sending the last
     COMMIT_AMD_FETCH) without knowing any knowledge of this incoming request.
     This makes existing backend very hard to adapt to ublk because they may
     want to know the data length or other attributes of the new request.

(2) If the backend does not provide the data buffer IN ADVANCE, ublksrv must
    pre-allocates data buffer. So a additional data copy from ublksrv to
    the backend(such as a RPC mempool) is unavoidable.

With UBLK_IO_NEED_GET_DATA, the WRITE request will be firstly issued to ublksrv
without data copy. Then, backend gets the request and it can allocate data
buffer and embed its addr inside a new ioucmd. After the kernel driver gets the
ioucmd, the data copy happens(from biovecs to backend's buffer). Finally,
the backend gets the request again with data to be written and it can truly
handle the request.

Thanks for the explanation. Maybe it can be included in the
documentation.

This reminds me of io_uring's IOSQE_BUFFER_SELECT where userspace
provides the kernel with a buffer pool and the kernel selects buffers.
It doesn't require an extra io_uring command roundtrip
(UBLK_IO_NEED_GET_DATA).

Did you already look at IOSQE_BUFFER_SELECT and decide a similar
approach won't work for your use case?

Stefan