[RFC iproute2 0/8] RDMA tool

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

----
This is initial phase to understand if user experience for this tool fits
RDMA and netdev communities exepectations. Also I would like to get feedback
if it is really worth to provide legacy sysfs for old kernels, or maybe I should
implement netlink from the beginning and abandon sysfs completely.
-----

Hi,

Please find below, the patch set with initial implementation of configuration
tool for RDMA subsystem, which will be supplementary tool to already existed
tools in netdev community (ip, devlink, ethtool, ..).

In opposite to netdev community, where standard tools exist to configure and
present different devices abilities, RDMA subsystem historically lacked it.

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

The development of tool was influenced by ip and devlink tools. This implies
to the object->command interface and naming convention.

In order to close object model, ensure reuse of existing code and make this
tool usable from day one, we decided to implement wrappers over legacy sysfs
prior to implementing netlink functionality. As a nice bonus, it will allow
to use this tool with old kernels too.

It is important to mention that any future extension will be required to be
done with netlink, so for already existing objects small conversion to netlink
will be unavoidable.

  # rdma -h
  Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
  where  OBJECT := { dev | link | ipoib | memory | stats | protocols | providers | monitor }
  OPTIONS := { -V[ersion] }

* DEV object equals to CA in IBTA specification and will provide
  a way to configure/present settings relevant to specific struct ib_device.

* LINK object represents port in IBTA specification and will give access to
  struct ib_port_immutable. From the day one, It prints netdev name of
  the corresponding IB port that makes ibdev2netdev script redundant.

* IPoIB object is supposed to be specific for IP-over-Infiniband upper
  layer protocol [3]. This ULP was mainly configured by combination
  of various sysfs knobs together with ethtool. Such situation adds
  challenges to add new and expose old configuration settings due to
  the mix between different subsystems.

* MEMORY object will be used to configure memory related settings,
  e.g. on-demand-paging (ODP), force-mr (force usage of MRs for
  RDMA READ/WRITE operation).

* STATS object is needed for everything related to statistics
  (per-PID, per-QP, per-device etc.). Despite the fact that RDMA
  devices provide extensive set of counters, the decision was to
  implement it in netlink directly, because there is a need to add
  filter mechanism to them, which doesn't exist now.

* PROTOCOLS object is going to be used for device special treatment
  of global to protocol settings (e.g. set device in RoCEv2 mode as
  a default, instead of RoCEv1, instead of configfs).

* PROVIDERS objects gives ability to get specific to the device
  information, like supported kABI objects [4].

* MONITOR object is needed to debug netlink communication and will
  follow standard functionality, which exists in ip and devlink tools.

There are number of ULPs which are not covered by this tool yet:
 * HFI-VNIC - I have no access to the HW and believe that Intel
   will add native object support for it.
 * Other storage related ULPs (iSER and SRP) were not introduced too,
   because they have special tools (scci-target-utils) to configure them.
   However it will be pretty straightforward to introduce new object,
   if there is demand for it.

At the initial stage, we implemented infrastructure to read legacy
sysfs entries (Patch #1), initial man pages (Patch #7) and provided
future object examples (Patch #2-6) to allow parallel development.

Following patches will focus on cleaning user interface, parsing other
relevant entries in similar fashion to the link capability mask (Patch #8)
and providing netlink interface.

These patches were tested with two following setups:
 * Setup A:
   - Two Mellanox ConnectX-4 devices (one port)
   - One Mellanox Connect-IB device (two ports)
 * Setup B:
   - One Mellanox ConnectX-4 device (one port)
   - One Mellanox ConnectX-3 Pro device (two ports)

Please consider the inclusion of the RDMA tool into iproute2 package,
so other participants will be able to speed up development.

[1] https://www.mail-archive.com/netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg148523.html
[2] http://www.medkio.com/talks/lpc_debug.pdf
[3] https://tools.ietf.org/html/rfc4392
[4] http://marc.info/?l=linux-rdma&m=149261526916544&w=2

TODO: Add json output

Cc: Stephen Hemminger <redacted>
Cc: Doug Ledford <redacted>
Cc: Jiri Pirko <jiri-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Ariel Almog <ariela-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Dennis Dalessandro <redacted>
Cc: Ram Amrani <ram.amrani-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>
Cc: Bart Van Assche <redacted>
Cc: Sagi Grimberg <redacted>
Cc: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Cc: Christoph Hellwig <redacted>
Cc: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Linux RDMA <redacted>
Cc: Linux Netdev <redacted>

Leon Romanovsky (8):
  rdma: Add basic infrastructure for RDMA tool
  rdma: Add dev object
  rdma: Add link object
  rdma: Add IPoIB object
  rdma: Add memory object
  rdma: add stubs for future objects
  man: rdma.8: Document objects and commands
  rdma: Add link capability parsing

 Makefile          |   2 +-
 man/man8/Makefile |   3 +-
 man/man8/rdma.8   | 109 +++++++++++++++++++
 rdma/.gitignore   |   1 +
 rdma/Makefile     |  15 +++
 rdma/dev.c        | 101 ++++++++++++++++++
 rdma/ipoib.c      |  54 ++++++++++
 rdma/link.c       | 160 ++++++++++++++++++++++++++++
 rdma/memory.c     |  30 ++++++
 rdma/monitor.c    |  22 ++++
 rdma/protocols.c  |  22 ++++
 rdma/providers.c  |  28 +++++
 rdma/rdma.c       | 104 ++++++++++++++++++
 rdma/rdma.h       |  93 ++++++++++++++++
 rdma/stats.c      |  22 ++++
 rdma/utils.c      | 313 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 16 files changed, 1077 insertions(+), 2 deletions(-)
 create mode 100644 man/man8/rdma.8
 create mode 100644 rdma/.gitignore
 create mode 100644 rdma/Makefile
 create mode 100644 rdma/dev.c
 create mode 100644 rdma/ipoib.c
 create mode 100644 rdma/link.c
 create mode 100644 rdma/memory.c
 create mode 100644 rdma/monitor.c
 create mode 100644 rdma/protocols.c
 create mode 100644 rdma/providers.c
 create mode 100644 rdma/rdma.c
 create mode 100644 rdma/rdma.h
 create mode 100644 rdma/stats.c
 create mode 100644 rdma/utils.c

--
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Leon Romanovsky <redacted>

Link object represents port of struct ib_device.

Supported commands are show, set and help.

Print all links for all devices:
 # rdma link
1/1: mlx5_0/1: ifname ib0 cap_mask 0x2651e848 lid 0x13 lid_mask_count 0 link_layer InfiniBand
	phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x2 sm_sl 0 state 4: ACTIVE
2/1: mlx5_1/1: ifname ib1 cap_mask 0x2651e848 lid 0xffff lid_mask_count 0 link_layer InfiniBand
	phys_state 3: Disabled rate 10 Gb/sec (4X) sm_lid 0x0 sm_sl 0 state 1: DOWN
3/1: mlx5_2/1: ifname ib2 cap_mask 0x26516848 lid 0x1a lid_mask_count 0 link_layer InfiniBand
	phys_state 5: LinkUp rate 56 Gb/sec (4X FDR) sm_lid 0x2 sm_sl 0 state 4: ACTIVE
3/2: mlx5_2/2: ifname ib3 cap_mask 0x26516848 lid 0xffff lid_mask_count 0 link_layer InfiniBand
	phys_state 3: Disabled rate 10 Gb/sec (4X) sm_lid 0x0 sm_sl 0 state 1: DOWN

Print all links for specific device:
 # rdma link show mlx5_2
3/1: mlx5_2/1: ifname ib2 cap_mask 0x26516848 lid 0x1a lid_mask_count 0 link_layer InfiniBand
	phys_state 5: LinkUp rate 56 Gb/sec (4X FDR) sm_lid 0x2 sm_sl 0 state 4: ACTIVE
3/2: mlx5_2/2: ifname ib3 cap_mask 0x26516848 lid 0xffff lid_mask_count 0 link_layer InfiniBand
	phys_state 3: Disabled rate 10 Gb/sec (4X) sm_lid 0x0 sm_sl 0 state 1: DOWN

Print specific link:
 # rdma link show mlx5_2/2
3/2: mlx5_2/2: ifname ib3 cap_mask 0x26516848 lid 0xffff lid_mask_count 0 link_layer InfiniBand
	phys_state 3: Disabled rate 10 Gb/sec (4X) sm_lid 0x0 sm_sl 0 state 1: DOWN

Set parameter;
 # rdma link set mlx5_2/2 type auto lb_unicast off

Signed-off-by: Leon Romanovsky <redacted>
---
 rdma/Makefile |   2 +-
 rdma/link.c   | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.c   |   3 +-
 rdma/rdma.h   |   1 +
 4 files changed, 116 insertions(+), 2 deletions(-)
 create mode 100644 rdma/link.c

diff --git a/rdma/Makefile b/rdma/Makefile
index 67e349b0..cf54ed36 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile

@@ -1,6 +1,6 @@
 include ../Config

-RDMA_OBJ = rdma.o utils.o dev.o
+RDMA_OBJ = rdma.o utils.o dev.o link.o
 TARGETS=rdma

 all:	$(TARGETS) $(LIBS)

diff --git a/rdma/link.c b/rdma/link.c
new file mode 100644
index 00000000..e86ff399
--- /dev/null
+++ b/rdma/link.c

@@ -0,0 +1,112 @@
+/*
+ * link.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static int link_help(struct rdma *rd)
+{
+	pr_out("Usage: %s link show [ DEV | DEV/PORT ]\n", rd->filename);
+	pr_out("       %s link set DEV/PORT { type { eth | ib | auto } |\n", rd->filename);
+	pr_out("                                lb_unicast { on | off } |\n");
+	pr_out("                                lb_multicast { on | off } }\n");
+	return 0;
+}
+
+static void dev_one_show(const struct dev_map *dev_map, uint32_t port_idx_first, uint32_t port_idx_last)
+{
+	char *nodes[] = { "cap_mask",
+			  "lid",
+			  "lid_mask_count",
+			  "link_layer",
+			  "phys_state",
+			  "rate",
+			  "sm_lid",
+			  "sm_sl",
+			  "state",
+       			  NULL };
+
+	struct port_map *port_map;
+	char data[4096];
+	int i, j;
+
+	for(j = port_idx_first ; j <= port_idx_last; j++) {
+		pr_out("%u/%u: %s/%u:", dev_map->idx, j, dev_map->dev_name, j);
+		list_for_each_entry(port_map, &dev_map->port_map_list, list)
+			if (j == port_map->idx)
+			       printf(" ifname %s", (port_map->ifname)?:"NONE");
+
+		for (i = 0 ; nodes[i] ; i++) {
+			if (rdma_sysfs_read_ib(dev_map->dev_name, j, nodes[i], data))
+				continue;
+
+			/* Split line before "phys_state" */
+			if (!strcmp(nodes[i], "phys_state"))
+				printf("\n\t");
+
+			pr_out(" %s %s", nodes[i], data);
+		}
+		pr_out("\n");
+	}
+}
+
+static int link_show(struct rdma *rd)
+{
+	struct dev_map *dev_map;
+
+	if (rd_no_arg(rd)) {
+		list_for_each_entry(dev_map, &rd->dev_map_list, list)
+			dev_one_show(dev_map, 1, dev_map->num_ports);
+	}
+	else {
+		uint32_t port_idx;
+		uint32_t num_ports;
+		dev_map = dev_map_lookup(rd, true);
+		port_idx = get_port_from_argv(rd);
+		if (!dev_map || port_idx > dev_map->num_ports) {
+			pr_err("Wrong device name\n");
+			return -EINVAL;
+		}
+		if (port_idx)
+			num_ports = port_idx;
+		else {
+			port_idx = 1;
+			num_ports = dev_map->num_ports;
+		}
+
+		dev_one_show(dev_map, port_idx, num_ports);
+	}
+	return 0;
+}
+
+static int link_set(struct rdma *rd)
+{
+	/* Not supported yet */
+	return 0;
+}
+
+int obj_link(struct rdma *rd)
+{
+	const struct rdma_obj objs[] = {
+		{ NULL, 	link_show },
+		{ "show",	link_show },
+		{ "list",	link_show },
+		{ "set",	link_set },
+		{ "help",	link_help },
+		{ 0 }
+	};
+
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	return rdma_exec_cmd(rd, objs, "link command");
+}

diff --git a/rdma/rdma.c b/rdma/rdma.c
index 7c537c5e..55cbf0e3 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c

@@ -17,7 +17,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { dev }\n"
+	       "where  OBJECT := { dev | link }\n"
 	       "       OPTIONS := { -V[ersion] }\n", name);
 }

@@ -32,6 +32,7 @@ static int rd_cmd(struct rdma *rd)
 	const struct rdma_obj objs[] = {
 		{ NULL,		obj_help },
 		{ "dev",	obj_dev },
+		{ "link",	obj_link },
 		{ "help",	obj_help },
 		{ 0 }
 	};

diff --git a/rdma/rdma.h b/rdma/rdma.h
index 2d81cd92..bdb77b5e 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h

@@ -63,6 +63,7 @@ struct rdma_obj {
  * Command interfaces
  */
 int obj_dev(struct rdma *rd);
+int obj_link(struct rdma *rd);

 /*
  * Parser interface

From: Leon Romanovsky <redacted>

Device (dev) object represents struct ib_device to user space.

The supported commands are show, set and help.

Print all devices:
 # rdma dev
1: mlx5_0: board_id MT_2190110032 fw_pages 261002 fw_ver 12.17.2046 hca_type MT4115 hw_rev 0
	node_desc hpchead HCA-1 node_guid e41d:2d03:0066:dee6 node_type 1: CA reg_pages 0
	sys_image_guid e41d:2d03:0066:dee6
2: mlx5_1: board_id MT_2190110032 fw_pages 250793 fw_ver 12.17.2046 hca_type MT4115 hw_rev 0
	node_desc hpchead HCA-2 node_guid e41d:2d03:0066:dee7 node_type 1: CA reg_pages 0
	sys_image_guid e41d:2d03:0066:dee6
3: mlx5_2: board_id MT_1210110019 fw_pages 68067 fw_ver 10.16.1020 hca_type MT4113 hw_rev 0
	node_desc hpchead HCA-3 node_guid 0002:c903:0016:75b0 node_type 1: CA reg_pages 0
	sys_image_guid 0002:c903:0016:75b0

Print specific device:
 # rdma dev show mlx5_1
2: mlx5_1: board_id MT_2190110032 fw_pages 250793 fw_ver 12.17.2046 hca_type MT4115 hw_rev 0
	node_desc hpchead HCA-2 node_guid e41d:2d03:0066:dee7 node_type 1: CA reg_pages 0
	sys_image_guid e41d:2d03:0066:dee6

Signed-off-by: Leon Romanovsky <redacted>
---
 rdma/Makefile |   2 +-
 rdma/dev.c    | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rdma/rdma.c   |   3 +-
 rdma/rdma.h   |   5 +++
 4 files changed, 109 insertions(+), 2 deletions(-)
 create mode 100644 rdma/dev.c

diff --git a/rdma/Makefile b/rdma/Makefile
index 65248b31..67e349b0 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile

@@ -1,6 +1,6 @@
 include ../Config

-RDMA_OBJ = rdma.o utils.o
+RDMA_OBJ = rdma.o utils.o dev.o
 TARGETS=rdma

 all:	$(TARGETS) $(LIBS)

diff --git a/rdma/dev.c b/rdma/dev.c
new file mode 100644
index 00000000..e6d71035
--- /dev/null
+++ b/rdma/dev.c

@@ -0,0 +1,101 @@
+/*
+ * dev.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static int dev_help(struct rdma *rd)
+{
+	pr_out("Usage: %s dev show [DEV]\n", rd->filename);
+	pr_out("       %s dev set DEV [ node_desc { DESCRIPTION } ]\n", rd->filename);
+	/* Add masking of device capabilities */
+	return 0;
+}
+
+static void dev_one_show(const struct dev_map *dev_map)
+{
+	char *nodes[] = { "board_id",
+			  "fw_pages",
+			  "fw_ver",
+			  "hca_type",
+			  "hw_rev",
+			  "node_desc",
+			  "node_guid",
+			  "node_type",
+			  "reg_pages",
+			  "sys_image_guid",
+			  /* hfi1 specific */
+			  "nctxts",
+			  "nfreectxts",
+			  "serial",
+			  "boardversion",
+			  "tempsense",
+			  NULL };
+
+	char data[4096];
+	int i;
+	pr_out("%u: %s:", dev_map->idx, dev_map->dev_name);
+	for (i = 0 ; nodes[i] ; i++) {
+		if (rdma_sysfs_read_ib(dev_map->dev_name, 0, nodes[i], data))
+			continue;
+
+		/* Split line before "node_desc" */
+		if (!strcmp(nodes[i], "node_desc") ||
+		    !strcmp(nodes[i], "sys_image_guid"))
+			printf("\n\t");
+
+		pr_out(" %s %s", nodes[i], data);
+	}
+	pr_out("\n");
+}
+
+static int dev_show(struct rdma *rd)
+{
+	struct dev_map *dev_map;
+
+	if (rd_no_arg(rd)) {
+		list_for_each_entry(dev_map, &rd->dev_map_list, list)
+			dev_one_show(dev_map);
+	}
+	else {
+		dev_map = dev_map_lookup(rd, false);
+		if (!dev_map) {
+			pr_err("Wrong device name\n");
+			return -ENOENT;
+		}
+		dev_one_show(dev_map);
+	}
+	return 0;
+}
+
+static int dev_set(struct rdma *rd)
+{
+	/* Not implemented yet */
+	return 0;
+}
+
+int obj_dev(struct rdma *rd)
+{
+	const struct rdma_obj objs[] = {
+		{ NULL,		dev_show },
+		{ "show",	dev_show },
+		{ "list",	dev_show },
+		{ "set",	dev_set },
+		{ "help",	dev_help },
+		{ 0 }
+	};
+
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	return rdma_exec_cmd(rd, objs, "dev command");
+}

diff --git a/rdma/rdma.c b/rdma/rdma.c
index bc7d1483..7c537c5e 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c

@@ -17,7 +17,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { }\n"
+	       "where  OBJECT := { dev }\n"
 	       "       OPTIONS := { -V[ersion] }\n", name);
 }

@@ -31,6 +31,7 @@ static int rd_cmd(struct rdma *rd)
 {
 	const struct rdma_obj objs[] = {
 		{ NULL,		obj_help },
+		{ "dev",	obj_dev },
 		{ "help",	obj_help },
 		{ 0 }
 	};

diff --git a/rdma/rdma.h b/rdma/rdma.h
index 156bb74c..2d81cd92 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h

@@ -60,6 +60,11 @@ struct rdma_obj {
 };

 /*
+ * Command interfaces
+ */
+int obj_dev(struct rdma *rd);
+
+/*
  * Parser interface
  */
 bool rd_no_arg(struct rdma *rd);

From: Leon Romanovsky <redacted>

Memory object gives to the user ability to manipulate over general
properties of memory for the specific devices. The memory properties
have broader usage than dev object can provide.

For example, on-demand-paging (ODP) configurations are mostly software related.

Signed-off-by: Leon Romanovsky <redacted>
---
 rdma/Makefile |  2 +-
 rdma/memory.c | 30 ++++++++++++++++++++++++++++++
 rdma/rdma.c   |  3 ++-
 rdma/rdma.h   |  1 +
 4 files changed, 34 insertions(+), 2 deletions(-)
 create mode 100644 rdma/memory.c

diff --git a/rdma/Makefile b/rdma/Makefile
index dd702b9f..5cf0d29f 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile

@@ -1,6 +1,6 @@
 include ../Config

-RDMA_OBJ = rdma.o utils.o dev.o link.o ipoib.o
+RDMA_OBJ = rdma.o utils.o dev.o link.o ipoib.o memory.o
 TARGETS=rdma

 all:	$(TARGETS) $(LIBS)

diff --git a/rdma/memory.c b/rdma/memory.c
new file mode 100644
index 00000000..68fd5dd3
--- /dev/null
+++ b/rdma/memory.c

@@ -0,0 +1,30 @@
+/*
+ * memory.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static void memory_help(char *filename)
+{
+	pr_out("Usage: %s memory show [ DEV ]\n", filename);
+	pr_out("       %s memory set DEV { odp { off | on } |\n", filename);
+	pr_out("       %s                  memic SIZE }\n", filename);
+}
+
+int obj_memory(struct rdma *rd)
+{
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	memory_help(rd->filename);
+	return 0;
+}

diff --git a/rdma/rdma.c b/rdma/rdma.c
index ffd70899..094d490d 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c

@@ -17,7 +17,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { dev | link | ipoib }\n"
+	       "where  OBJECT := { dev | link | ipoib | memory }\n"
 	       "       OPTIONS := { -V[ersion] }\n", name);
 }

@@ -34,6 +34,7 @@ static int rd_cmd(struct rdma *rd)
 		{ "dev",	obj_dev },
 		{ "link",	obj_link },
 		{ "ipoib",	obj_ipoib },
+		{ "memory",	obj_memory },
 		{ "help",	obj_help },
 		{ 0 }
 	};

diff --git a/rdma/rdma.h b/rdma/rdma.h
index 1fef4eb8..dcff066f 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h

@@ -65,6 +65,7 @@ struct rdma_obj {
 int obj_dev(struct rdma *rd);
 int obj_link(struct rdma *rd);
 int obj_ipoib(struct rdma *rd);
+int obj_memory(struct rdma *rd);

 /*
  * Parser interface

From: Leon Romanovsky <redacted>

The following objects (monitor, providers, stats and protocols) are not
implemented yet, however it is worth to place their stubs in the code.

This will serve as an initial starting point for other developers to
extend RDMA tool.

Signed-off-by: Leon Romanovsky <redacted>
---
 rdma/Makefile    |  2 +-
 rdma/monitor.c   | 22 ++++++++++++++++++++++
 rdma/protocols.c | 22 ++++++++++++++++++++++
 rdma/providers.c | 28 ++++++++++++++++++++++++++++
 rdma/rdma.c      |  6 +++++-
 rdma/rdma.h      |  4 ++++
 rdma/stats.c     | 22 ++++++++++++++++++++++
 7 files changed, 104 insertions(+), 2 deletions(-)
 create mode 100644 rdma/monitor.c
 create mode 100644 rdma/protocols.c
 create mode 100644 rdma/providers.c
 create mode 100644 rdma/stats.c

diff --git a/rdma/Makefile b/rdma/Makefile
index 5cf0d29f..eb71da68 100644
--- a/rdma/Makefile
+++ b/rdma/Makefile

@@ -1,6 +1,6 @@
 include ../Config

-RDMA_OBJ = rdma.o utils.o dev.o link.o ipoib.o memory.o
+RDMA_OBJ = rdma.o utils.o dev.o link.o ipoib.o memory.o stats.o protocols.o providers.o monitor.o
 TARGETS=rdma

 all:	$(TARGETS) $(LIBS)

diff --git a/rdma/monitor.c b/rdma/monitor.c
new file mode 100644
index 00000000..99d4b042
--- /dev/null
+++ b/rdma/monitor.c

@@ -0,0 +1,22 @@
+/*
+ * monitor.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+int obj_monitor(struct rdma *rd)
+{
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	return 0;
+}

diff --git a/rdma/protocols.c b/rdma/protocols.c
new file mode 100644
index 00000000..26de7d2b
--- /dev/null
+++ b/rdma/protocols.c

@@ -0,0 +1,22 @@
+/*
+ * protocols.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+int obj_protocols(struct rdma *rd)
+{
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	return 0;
+}

diff --git a/rdma/providers.c b/rdma/providers.c
new file mode 100644
index 00000000..8d516cca
--- /dev/null
+++ b/rdma/providers.c

@@ -0,0 +1,28 @@
+/*
+ * providers.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+static void providers_help(char *filename)
+{
+	pr_out("Usage: %s providers show [ DEV ]\n", filename);
+}
+
+int obj_providers(struct rdma *rd)
+{
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	providers_help(rd->filename);
+	return 0;
+}

diff --git a/rdma/rdma.c b/rdma/rdma.c
index 094d490d..a0a3ec81 100644
--- a/rdma/rdma.c
+++ b/rdma/rdma.c

@@ -17,7 +17,7 @@
 static void help(char *name)
 {
 	pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n"
-	       "where  OBJECT := { dev | link | ipoib | memory }\n"
+	       "where  OBJECT := { dev | link | ipoib | memory | stats | protocols | providers | monitor }\n"
 	       "       OPTIONS := { -V[ersion] }\n", name);
 }

@@ -35,6 +35,10 @@ static int rd_cmd(struct rdma *rd)
 		{ "link",	obj_link },
 		{ "ipoib",	obj_ipoib },
 		{ "memory",	obj_memory },
+		{ "stats",	obj_stats },
+		{ "providers",	obj_providers },
+		{ "protocols",	obj_protocols },
+		{ "monitor",	obj_monitor },
 		{ "help",	obj_help },
 		{ 0 }
 	};

diff --git a/rdma/rdma.h b/rdma/rdma.h
index dcff066f..11d940d7 100644
--- a/rdma/rdma.h
+++ b/rdma/rdma.h

@@ -66,6 +66,10 @@ int obj_dev(struct rdma *rd);
 int obj_link(struct rdma *rd);
 int obj_ipoib(struct rdma *rd);
 int obj_memory(struct rdma *rd);
+int obj_protocols(struct rdma *rd);
+int obj_stats(struct rdma *rd);
+int obj_providers(struct rdma *rd);
+int obj_monitor(struct rdma *rd);

 /*
  * Parser interface

diff --git a/rdma/stats.c b/rdma/stats.c
new file mode 100644
index 00000000..a557e59b
--- /dev/null
+++ b/rdma/stats.c

@@ -0,0 +1,22 @@
+/*
+ * stats.c	RDMA tool
+ *
+ *              This program is free software; you can redistribute it and/or
+ *              modify it under the terms of the GNU General Public License
+ *              as published by the Free Software Foundation; either version
+ *              2 of the License, or (at your option) any later version.
+ *
+ * Authors:     Leon Romanovsky <leonro@mellanox.com>
+ */
+
+#include "rdma.h"
+
+int obj_stats(struct rdma *rd)
+{
+	if (dev_map_init(rd)) {
+		pr_err("There are no RDMA devices\n");
+		return -ENOENT;
+	}
+
+	return 0;
+}

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Thanks,

Bart.

On Thu, May 04, 2017 at 06:10:54PM +0000, Bart Van Assche wrote:

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

quoted

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

We are planning to reuse the same infrastructure provided by iproute2,
like netlink parsing, access to distributions, same CLI and same standards.

Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
RDMA.

I do expect that iproute2 will be installed on every machine with any
type of connection, including IB and OPA.

So I think that it is enough to be part of that suite and don't invent
our own for one specific tool.

Thanks

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thu, May 04, 2017 at 08:10:54PM CEST, Bart.VanAssche@sandisk.com wrote:

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

quoted

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Bart, please realize that iproute2 is much more than "*IP routing* tool".
I understand you got confused by the name. Please see sources. Your comment
is totally pointless...

On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:

Thu, May 04, 2017 at 08:10:54PM CEST, Bart.VanAssche@sandisk.com wrote:

quoted

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

quoted

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Bart, please realize that iproute2 is much more than "*IP routing* tool".
I understand you got confused by the name. Please see sources. Your comment
is totally pointless...

I asked for a clarification that should have been in the cover letter but that
was missing from that cover letter. So I think that was the right thing to do
instead of pointless. BTW, can you explain why you are using an e-mail address
that is hiding that you are a Mellanox employee?

Bart.N�����r��y���b�X��ǧv�^�)޺{.n�+����{��ٚ�{ay�
ʇڙ�,j��f���h�����/oSc��ڳ9�u�����&jw��(�階�ݢj"���m�����z�ޖ���f���h���~�m�

Sat, May 06, 2017 at 04:40:24PM CEST, Bart.VanAssche@sandisk.com wrote:

On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:

quoted

Thu, May 04, 2017 at 08:10:54PM CEST, Bart.VanAssche@sandisk.com wrote:

quoted

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

quoted

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Bart, please realize that iproute2 is much more than "*IP routing* tool".
I understand you got confused by the name. Please see sources. Your comment
is totally pointless...

I asked for a clarification that should have been in the cover letter but that
was missing from that cover letter. So I think that was the right thing to do

I think that was just complete misunderstanding about what iproute2 is.

instead of pointless. BTW, can you explain why you are using an e-mail address
that is hiding that you are a Mellanox employee?

Who sais I have to do that? This is funny...

On Sat, May 06, 2017 at 02:40:24PM +0000, Bart Van Assche wrote:

On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:

quoted

Thu, May 04, 2017 at 08:10:54PM CEST, Bart.VanAssche@sandisk.com wrote:

quoted

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

quoted

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Bart, please realize that iproute2 is much more than "*IP routing* tool".
I understand you got confused by the name. Please see sources. Your comment
is totally pointless...

I asked for a clarification that should have been in the cover letter but that
was missing from that cover letter. So I think that was the right thing to do
instead of pointless. BTW, can you explain why you are using an e-mail address
that is hiding that you are a Mellanox employee?

For the same reason as I do. It is much easier to use outside servers
than Mellanox's IT infrastructure.

Right now, I'm speaking for myself, but for me, to properly answer with @mellanox.com,
I need to use very specific mail setup, while for any other addresses I can and use any
sane setup, including mobile application.

Bart.