Three SRP patches

Started by Bart Van Assche on Jan 26, 2021 · 8 messages from 3 authors · Last reply Feb 28, 2021

Series

3 patches, 3 files

+48 −117

Subsystems (1)

Maintainers (1)

STALE2084d

open the first message on its own page

  1. [PATCH blktests 1/3] tests/block/030: Make this test less noisy
    Related patches
  2. [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent
    Related patches
  3. [PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt()
    Related patches
Message details and related content
Hi Omar,

The three patches in this series improve the SRP tests. Please consider these
patches for inclusion in the upstream blktests repository.

Thanks,

Bart.

Bart Van Assche (3):
  tests/block/030: Make this test less noisy
  tests/srp/rc: Improve reliability of stop_lio_srpt()
  rdma: Use rdma link instead of /sys/class/infiniband/*/parent

 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/block/030            |   2 +-
 tests/srp/rc               |  52 ++++++-----------
 3 files changed, 48 insertions(+), 117 deletions(-)

[PATCH blktests 1/3] tests/block/030: Make this test less noisy

Message details and related content
Since test block/030 injects blk_mq_realloc_hw_ctxs() failures, it is
expected that writes into the 'submit_queues' attribute can fail. Send
the 'nproc: write error: Cannot allocate memory' failures to $FULL instead
of stderr. See also commit a668c61064f2 ("Add a test that triggers the
blk_mq_realloc_hw_ctxs() error path").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 tests/block/030 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/block/030 b/tests/block/030
index 84302c109c0c..d2e528697873 100755
--- a/tests/block/030
+++ b/tests/block/030
@@ -42,7 +42,7 @@ test() {
 	if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then
 		for ((i = 0; i < 100; i++)); do
 			echo 1 > $sq
-			nproc > $sq
+			{ nproc > $sq; } >>"$FULL" 2>&1
 		done
 	else
 		SKIP_REASON="Skipping test because $sq cannot be modified"

[PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

Message details and related content
The approach of verifying whether or not an RDMA interface is associated
with the rdma_rxe interface by looking up its parent device is deprecated
and will be removed soon from the Linux kernel. Hence this patch that uses
the rdma link command instead.

Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Yi Zhang <redacted>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/srp/rc               |   9 +--
 2 files changed, 32 insertions(+), 88 deletions(-)
diff --git a/common/multipath-over-rdma b/common/multipath-over-rdma
index 9d9d2b27af83..9e68189b56fd 100644
--- a/common/multipath-over-rdma
+++ b/common/multipath-over-rdma
@@ -57,6 +57,9 @@ _multipathd_version_ge() {
 }
 
 get_ipv4_addr() {
+	if [ ! -e "/sys/class/net/$1" ]; then
+		echo "get_ipv4_addr(): $1 is not a network interface" 1>&2
+	fi
 	ip -4 -o addr show dev "$1" |
 		sed -n 's/.*[[:blank:]]inet[[:blank:]]*\([^[:blank:]/]*\).*/\1/p'
 }
@@ -76,41 +79,11 @@ is_number() {
 	[ "$1" -eq "0$1" ] 2>/dev/null
 }
 
-# Check whether a device is an RDMA device. An example argument:
-# /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0
-is_rdma_device() {
-	local d i inode1 inode2
-
-	inode1=$(stat -c %i "$1")
-	# echo "inode1 = $inode1"
-	for i in /sys/class/infiniband/*; do
-		d=/sys/class/infiniband/"$(readlink "$i")"
-		d=$(dirname "$(dirname "$d")")
-		inode2=$(stat -c %i "$d")
-		# echo "inode2 = $inode2"
-		if [ "$inode1" = "$inode2" ]; then
-			return
-		fi
-	done
-	false
-}
-
 # Lists RDMA capable network interface names, e.g. ib0 ib1.
 rdma_network_interfaces() {
-	(
-		cd /sys/class/net &&
-			for i in *; do
-				[ -e "$i" ] || continue
-				# Skip IPoIB (ARPHRD_INFINIBAND) network
-				# interfaces.
-				[ "$(<"$i"/type)" = 32 ] && continue
-				[ -L "$i/device" ] || continue
-				d=$(readlink "$i/device" 2>/dev/null)
-				if [ -n "$d" ] && is_rdma_device "$i/$d"; then
-					echo "$i"
-				fi
-			done
-	)
+	rdma link show |
+		sed -n 's/^.*[[:blank:]]netdev[[:blank:]]\+\([^[:blank:]]*\)[[:blank:]]*/\1/p' |
+		sort -u
 }
 
 # Check whether any stacked block device holds block device $1. If so, echo
@@ -411,47 +384,36 @@ all_primary_gids() {
 	done
 }
 
-# Check whether or not an rdma_rxe instance has been associated with network
-# interface $1.
-has_rdma_rxe() {
-	local f
-
-	for f in /sys/class/infiniband/*/parent; do
-		if [ -e "$f" ] && [ "$(<"$f")" = "$1" ]; then
-			return 0
-		fi
-	done
-
-	return 1
+# Check whether or not an rdma_rxe or siw instance has been associated with
+# network interface $1.
+has_soft_rdma() {
+	rdma link | grep -q " netdev $1[[:blank:]]*\$"
 }
 
 # Load the rdma_rxe or siw kernel module and associate it with all network
 # interfaces.
 start_soft_rdma() {
+	local type
+
 	{
 	if [ -n "$use_siw" ]; then
 		modprobe siw || return $?
-		(
-			cd /sys/class/net &&
-				for i in *; do
-					[ -e "$i" ] || continue
-					[ -e "/sys/class/infiniband/${i}_siw" ] && continue
-					rdma link add "${i}_siw" type siw netdev "$i" ||
-						echo "Failed to bind the siw driver to $i"
-				done
-		)
+		type=siw
 	else
 		modprobe rdma_rxe || return $?
-		(
-			cd /sys/class/net &&
-				for i in *; do
-					if [ -e "$i" ] && ! has_rdma_rxe "$i"; then
-						echo "$i" > /sys/module/rdma_rxe/parameters/add ||
-							echo "Failed to bind the rdma_rxe driver to $i"
-					fi
-				done
-		)
+		type=rxe
 	fi
+	(
+		cd /sys/class/net &&
+			for i in *; do
+				[ -e "$i" ] || continue
+				[ "$i" = "lo" ] && continue
+				[ "$(<"$i/addr_len")" = 6 ] || continue
+				has_soft_rdma "$i" && continue
+				rdma link add "${i}_$type" type $type netdev "$i" ||
+				echo "Failed to bind the $type driver to $i"
+			done
+	)
 	} >>"$FULL"
 }
 
@@ -459,27 +421,16 @@ start_soft_rdma() {
 # unload the rdma_rxe kernel module.
 stop_soft_rdma() {
 	{
-	(
-		cd /sys/class/net &&
-			for i in *; do
-				if [ -e "$i" ] && has_rdma_rxe "$i"; then
-					{ echo "$i" > /sys/module/rdma_rxe/parameters/remove; } \
-						2>/dev/null
-				fi
-			done
-	)
+	rdma link |
+		sed -n 's,^link[[:blank:]]*\([^/]*\)/.* netdev .*,\1,p' |
+		while read -r i; do
+		      echo "$i ..."
+		      rdma link del "${i}" || echo "Failed to remove ${i}"
+		done
 	if ! unload_module rdma_rxe 10; then
 		echo "Unloading rdma_rxe failed"
 		return 1
 	fi
-	(
-		cd /sys/class/net &&
-			for i in *_siw; do
-				[ -e "$i" ] || continue
-				rdma link del "${i}" ||
-					echo "Failed to unbind the siw driver from ${i%_siw}"
-			done
-	)
 	if ! unload_module siw 10; then
 		echo "Unloading siw failed"
 		return 1
diff --git a/tests/srp/rc b/tests/srp/rc
index 700cd71ea155..07378fab2f2c 100755
--- a/tests/srp/rc
+++ b/tests/srp/rc
@@ -142,14 +142,7 @@ do_ib_cm_login() {
 }
 
 rdma_dev_to_net_dev() {
-	local b d rdma_dev=$1
-
-	b=/sys/class/infiniband/$rdma_dev/parent
-	if [ -e "$b" ]; then
-		echo "$(<"$b")"
-	else
-		echo "${rdma_dev%_siw}"
-	fi
+	rdma link show "$1/1" | sed 's/.* netdev //;s/[[:blank:]]*$//'
 }
 
 # Tell the SRP initiator to log in to an SRP target using the RDMA/CM.

[PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt()

Message details and related content
Remove the 'np' directory if it exists. Unload the iscsi_target_mod kernel
module if it has been loaded.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 tests/srp/rc | 43 +++++++++++++++----------------------------
 1 file changed, 15 insertions(+), 28 deletions(-)
diff --git a/tests/srp/rc b/tests/srp/rc
index 1f665a28db66..700cd71ea155 100755
--- a/tests/srp/rc
+++ b/tests/srp/rc
@@ -545,34 +545,21 @@ stop_lio_srpt() {
 		fi
 	done
 
-	if [ -e /sys/kernel/config/target/srpt ]; then
-		(
-			cd /sys/kernel/config/target/srpt && (
-				for d in */*/acls/*/*/lun*; do [ -L "$d" ] && rm "$d"; done
-				for d in */*/acls/*/lun*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*/acls/*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*/lun/lun*/*; do [ -L "$d" ] && rm "$d"; done
-				for d in */*/lun/lun*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*; do [ -e "$d/lun" ] && rmdir "$d"; done
-				for d in *; do [ -e "$d/fabric_statistics" ] && rmdir "$d"; done
-				true
-			) &&
-				cd .. &&
-				for ((i=0;i<10;i++)); do
-					rmdir srpt
-					[ -e srpt ] || break
-					sleep .1
-				done &&
-				[ ! -e srpt ] &&
-				unload_module ib_srpt 10
-		) || return $?
-	fi
-
-	rmdir /sys/kernel/config/target/core/*/* >&/dev/null
-	rmdir /sys/kernel/config/target/core/* >&/dev/null
-
-	for m in ib_srpt target_core_pscsi target_core_iblock target_core_file \
-			 target_core_stgt target_core_user target_core_mod
+	rmdir /sys/kernel/config/target/*/*/*/np/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/np >&/dev/null
+	rm -f /sys/kernel/config/target/*/*/*/acls/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/acls/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/acls/* >&/dev/null
+	rm -f /sys/kernel/config/target/*/*/*/lun/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/lun/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/* >&/dev/null
+
+	for m in ib_srpt iscsi_target_mod target_core_pscsi target_core_iblock \
+			 target_core_file target_core_stgt target_core_user \
+			 target_core_mod
 	do
 		unload_module $m 10 || return $?
 	done

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

Message details and related content
On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
The approach of verifying whether or not an RDMA interface is associated
with the rdma_rxe interface by looking up its parent device is deprecated
and will be removed soon from the Linux kernel. Hence this patch that uses
the rdma link command instead.

Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Yi Zhang <redacted>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/srp/rc               |   9 +--
 2 files changed, 32 insertions(+), 88 deletions(-)
I think we need to add _have_program rdma checks to srp and nvmeof-mp,
right? The first two patches look fine, I'll merge those.

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

Message details and related content
On 2/3/21 8:40 AM, Omar Sandoval wrote:
On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
quoted
The approach of verifying whether or not an RDMA interface is associated
with the rdma_rxe interface by looking up its parent device is deprecated
and will be removed soon from the Linux kernel. Hence this patch that uses
the rdma link command instead.

Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Yi Zhang <redacted>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/srp/rc               |   9 +--
 2 files changed, 32 insertions(+), 88 deletions(-)
I think we need to add _have_program rdma checks to srp and nvmeof-mp,
right? The first two patches look fine, I'll merge those.
Sure, I will add a _have_program rdma check and repost this patch.

Thanks,

Bart.

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

Message details and related content
Hello Bart

Currently blktests was broken on latest kernel, could you resend this updated patch. :)

Thanks
Yi

On 2/4/21 11:58 AM, Bart Van Assche wrote:
On 2/3/21 8:40 AM, Omar Sandoval wrote:
quoted
On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
quoted
The approach of verifying whether or not an RDMA interface is associated
with the rdma_rxe interface by looking up its parent device is deprecated
and will be removed soon from the Linux kernel. Hence this patch that uses
the rdma link command instead.

Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Yi Zhang <redacted>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
  common/multipath-over-rdma | 111 +++++++++++--------------------------
  tests/srp/rc               |   9 +--
  2 files changed, 32 insertions(+), 88 deletions(-)
I think we need to add _have_program rdma checks to srp and nvmeof-mp,
right? The first two patches look fine, I'll merge those.
Sure, I will add a _have_program rdma check and repost this patch.

Thanks,

Bart.
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help