Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

----- Original Message -----

From: "Laurence Oberman" <redacted>
To: "Bart Van Assche" <redacted>
Cc: "Doug Ledford" <redacted>, "Christoph Hellwig" <redacted>, "Sagi Grimberg" <redacted>,
"Steve Wise" [off-list ref], "Parav Pandit" [off-list ref], "Nicholas A. Bellinger"
[off-list ref], linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Sent: Tuesday, July 19, 2016 1:14:45 PM
Subject: Re: [PATCH v3 0/5] Reduce RDMA RW API SGE limit



----- Original Message -----

quoted

From: "Bart Van Assche" <redacted>
To: "Doug Ledford" <redacted>
Cc: "Christoph Hellwig" <redacted>, "Sagi Grimberg" <redacted>,
"Steve Wise" [off-list ref],
"Parav Pandit" [off-list ref], "Laurence Oberman"
[off-list ref], "Nicholas A. Bellinger"
[off-list ref], linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Sent: Tuesday, July 19, 2016 12:20:45 PM
Subject: [PATCH v3 0/5] Reduce RDMA RW API SGE limit

Hello Doug,

The five patches in this series modify the RDMA RW API slightly. This is
needed to avoid that the SRP and iSER target drivers submit RDMA
requests with an SGE list that exceeds the queue pair limits. The
ib_srpt changes in this series have been tested but the ib_isert changes
not yet.

The changes compared to v2 of this patch series are:
* For RDMA READs, limit SGE back to dev->attrs.max_sge_rd for iWARP.

Changes compared to v1:
* max_send_sge is now stored in struct ib_qp. This greatly simplifies
   this patch series.
* An unneeded initialization that I had added to rdma_rw_init_one_mr()
   has been left out again.
* Corrected "Fixes" tag in the patch description where needed.

The individual patches in this series are:

0001-IB-core-Make-rdma_rw_ctx_init-initialize-all-used-fi.patch
0002-IB-core-RDMA-RW-API-Do-not-exceed-QP-SGE-send-limit.patch
0003-IB-srpt-Limit-the-number-of-SG-elements-per-work-req.patch
0004-IB-srpt-Simplify-srpt_queue_response.patch
0005-IB-isert-Remove-an-unused-member-variable.patch

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Will pull and test these ASAP.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Replying to my own email:

Hello Bart

Testing SRP for these 5 patches here and I think the issues I am seeing may be changes in the latest upstream next kernel and not your code causing this.
'
I applied these to 4.7.0-rc7 and I am seeing some behavior I have not figured out yet.
I can find and map all LUNS via SRP but there are some differences in multipath.
I am still working on figuring some things out because I will likely have to apply these to an earlier base kernel to make sure.

I am not sure yet if something in the  4.7.0-rc7 base changed ALUA behavior, likely that is the case.
I have to go back and check.

On an older kernel with recent patches from you and Leon everything looks fine and is working fine.

4.7.0-rc1.bartleon+ 

360001ff0b035d000000000098d79000a dm-11 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 1:0:0:9  sdt  65:48  active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
  `- 2:0:0:9  sdv  65:80  active ready running
360001ff0b035d000000000078d770008 dm-9 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 2:0:0:7  sds  65:32  active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
  `- 1:0:0:7  sdp  8:240  active ready running


Looks fine here:

# time mkfs -t xfs -f  /dev/mapper/360001ff0b035d000000000078d770008 
meta-data=/dev/mapper/360001ff0b035d000000000078d770008 isize=256    agcount=33, agsize=243265536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=7784628224, imaxpct=5
         =                       sunit=4096   swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Finishes quickly, as usual.

real	0m1.764s
user	0m0.006s
sys	0m0.168s


[root@jumpclient ~]# sg_rtpg -d /dev/sds
Report target port groups:
  target port group id : 0x0 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x01
      0x02
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x03
      0x04
  target port group id : 0x2 , Pref=1
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x05
      0x06
  target port group id : 0x3 , Pref=1
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x07
      0x08


On this new kernel,4.7.0-rc7.bart5+

I see this behaviour:
ALUA is failing to map properly against the SFA14K,and we are coming up active/active and not active optimized/active non-optimized.

Creating an xfs on the LUN take a long time

[root@jumptest1 ~]# time mkfs -t xfs -f  /dev/mapper/360001ff0b035d000000000078d770008 
meta-data=/dev/mapper/360001ff0b035d000000000078d770008 isize=512    agcount=33, agsize=243265536 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=7784628224, imaxpct=5
         =                       sunit=4096   swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

real	10m34.395s
user	0m0.010s
sys	0m0.222s


[  321.500021] sd 2:0:0:29: alua: port group 01 state N non-preferred supports tolUsNA

360001ff0b035d000000000098d79000a dm-14 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=10 status=active
  |- 1:0:0:9  sdl  8:176  active ready running
  `- 2:0:0:9  sdba 67:64  active ready running
360001ff0b035d000000000078d770008 dm-12 DDN     ,SFA14K          
size=29T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=10 status=active
  |- 1:0:0:7  sdj  8:144  active ready running
  `- 2:0:0:7  sday 67:32  active ready running

Checking with sg_rtpg looks right however ???

Report target port groups:
  target port group id : 0x0 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x01
      0x02
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x03
      0x04
  target port group id : 0x2 , Pref=1
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x05
      0x06
  target port group id : 0x3 , Pref=1
    target port group asymmetric access state : 0x01 (active/non optimized)
    T_SUP : 0, O_SUP : 0, LBD_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x00 (no status available)
    vendor unique status : 0x00
    target port count : 02
    Relative target port ids:
      0x07
      0x08

Back when I have more.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html