Re: [PATCHv4 for-next 13/19] block/rnbd-clt: Support polling mode for IO latency optimization

On Mon, Apr 19, 2021 at 07:12:09AM +0200, Gioh Kim wrote:

On Sun, Apr 18, 2021 at 10:36 AM Leon Romanovsky [off-list ref] wrote:

quoted

On Wed, Apr 14, 2021 at 02:23:56PM +0200, Gioh Kim wrote:

quoted

From: Gioh Kim <redacted>

RNBD can make double-queues for irq-mode and poll-mode.
For example, on 4-CPU system 8 request-queues are created,
4 for irq-mode and 4 for poll-mode.
If the IO has HIPRI flag, the block-layer will call .poll function
of RNBD. Then IO is sent to the poll-mode queue.
Add optional nr_poll_queues argument for map_devices interface.

To support polling of RNBD, RTRS client creates connections
for both of irq-mode and direct-poll-mode.

For example, on 4-CPU system it could've create 5 connections:
con[0] => user message (softirq cq)
con[1:4] => softirq cq

After this patch, it can create 9 connections:
con[0] => user message (softirq cq)
con[1:4] => softirq cq
con[5:8] => DIRECT-POLL cq

<...>

quoted

quoted

+int rtrs_clt_rdma_cq_direct(struct rtrs_clt *clt, unsigned int index)
+{
+     int cnt;
+     struct rtrs_con *con;
+     struct rtrs_clt_sess *sess;
+     struct path_it it;
+
+     rcu_read_lock();
+     for (path_it_init(&it, clt);
+          (sess = it.next_path(&it)) && it.i < it.clt->paths_num; it.i++) {
+             if (unlikely(READ_ONCE(sess->state) != RTRS_CLT_CONNECTED))

We talked about useless likely/unlikely in your workloads.

Right, I've made a patch to remove all likely/unlikely
and will send with the next patch set.

This specific line is "brand new". We don't add code that will be
removed in next patch.

Thanks