[PATCH v2 0/2] Add cpu hotplug support for powerpc/perf/hv-24x7

This patchset add cpu hotplug support for hv_24x7 driver by adding
online/offline cpu hotplug function. It also add sysfs file
"cpumask" to expose current online cpu that can be used for
hv_24x7 event count.

Changelog:
v1 -> v2
- Changed function to pick active cpu incase of offline
  from "cpumask_any_but" to "cpumask_last", as
  cpumask_any_but function pick very next online cpu and incase where
  we are sequentially off-lining multiple cpus, "pmu_migrate_context"
  can add extra latency.
  - Suggested by: Gautham R Shenoy.

- Change documentation for cpumask and rather then hardcode the
  initialization for cpumask_attr_group, add loop to get very first
  NULL as suggested by Gautham R Shenoy.

Kajol Jain (2):
  powerpc/perf/hv-24x7: Add cpu hotplug support
  powerpc/hv-24x7: Add sysfs files inside hv-24x7 device to show cpumask

 .../sysfs-bus-event_source-devices-hv_24x7    |  7 ++
 arch/powerpc/perf/hv-24x7.c                   | 79 ++++++++++++++++++-
 include/linux/cpuhotplug.h                    |  1 +
 3 files changed, 86 insertions(+), 1 deletion(-)

-- 
2.18.2

Patch here adds cpu hotplug functions to hv_24x7 pmu.
A new cpuhp_state "CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE" enum
is added.

The online function update the cpumask only if its NULL.
As the primary intention for adding hotplug support
is to desiginate a CPU to make HCALL to collect the
count data.

The offline function test and clear corresponding cpu in a cpumask
and update cpumask to any other active cpu.

Signed-off-by: Kajol Jain <redacted>
---
 arch/powerpc/perf/hv-24x7.c | 45 +++++++++++++++++++++++++++++++++++++
 include/linux/cpuhotplug.h  |  1 +
 2 files changed, 46 insertions(+)

diff --git a/arch/powerpc/perf/hv-24x7.c b/arch/powerpc/perf/hv-24x7.c
index db213eb7cb02..ce4739e2b407 100644
--- a/arch/powerpc/perf/hv-24x7.c
+++ b/arch/powerpc/perf/hv-24x7.c

@@ -31,6 +31,8 @@ static int interface_version;
 /* Whether we have to aggregate result data for some domains. */
 static bool aggregate_result_elements;
 
+static cpumask_t hv_24x7_cpumask;
+
 static bool domain_is_valid(unsigned domain)
 {
 	switch (domain) {

@@ -1641,6 +1643,44 @@ static struct pmu h_24x7_pmu = {
 	.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
 };
 
+static int ppc_hv_24x7_cpu_online(unsigned int cpu)
+{
+	/* Make this CPU the designated target for counter collection */
+	if (cpumask_empty(&hv_24x7_cpumask))
+		cpumask_set_cpu(cpu, &hv_24x7_cpumask);
+
+	return 0;
+}
+
+static int ppc_hv_24x7_cpu_offline(unsigned int cpu)
+{
+	int target = -1;
+
+	/* Check if exiting cpu is used for collecting 24x7 events */
+	if (!cpumask_test_and_clear_cpu(cpu, &hv_24x7_cpumask))
+		return 0;
+
+	/* Find a new cpu to collect 24x7 events */
+	target = cpumask_last(cpu_active_mask);
+
+	if (target < 0 || target >= nr_cpu_ids)
+		return -1;
+
+	/* Migrate 24x7 events to the new target */
+	cpumask_set_cpu(target, &hv_24x7_cpumask);
+	perf_pmu_migrate_context(&h_24x7_pmu, cpu, target);
+
+	return 0;
+}
+
+static int hv_24x7_cpu_hotplug_init(void)
+{
+	return cpuhp_setup_state(CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,
+			  "perf/powerpc/hv_24x7:online",
+			  ppc_hv_24x7_cpu_online,
+			  ppc_hv_24x7_cpu_offline);
+}
+
 static int hv_24x7_init(void)
 {
 	int r;

@@ -1685,6 +1725,11 @@ static int hv_24x7_init(void)
 	if (r)
 		return r;
 
+	/* init cpuhotplug */
+	r = hv_24x7_cpu_hotplug_init();
+	if (r)
+		pr_err("hv_24x7: CPU hotplug init failed\n");
+
 	r = perf_pmu_register(&h_24x7_pmu, h_24x7_pmu.name, -1);
 	if (r)
 		return r;

diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 191772d4a4d7..a2710e654b64 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h

@@ -181,6 +181,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_POWERPC_CORE_IMC_ONLINE,
 	CPUHP_AP_PERF_POWERPC_THREAD_IMC_ONLINE,
 	CPUHP_AP_PERF_POWERPC_TRACE_IMC_ONLINE,
+	CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,
 	CPUHP_AP_WATCHDOG_ONLINE,
 	CPUHP_AP_WORKQUEUE_ONLINE,
 	CPUHP_AP_RCUTREE_ONLINE,

-- 
2.18.2

Patch here adds a cpumask attr to hv_24x7 pmu along with ABI documentation.

command:# cat /sys/devices/hv_24x7/cpumask
0

Signed-off-by: Kajol Jain <redacted>
---
 .../sysfs-bus-event_source-devices-hv_24x7    |  7 ++++
 arch/powerpc/perf/hv-24x7.c                   | 36 +++++++++++++++++--
 2 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7 b/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7
index e8698afcd952..f9dd3755b049 100644
--- a/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7
+++ b/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7

@@ -43,6 +43,13 @@ Description:	read only
 		This sysfs interface exposes the number of cores per chip
 		present in the system.
 
+What:		/sys/devices/hv_24x7/cpumask
+Date:		June 2020
+Contact:	Linux on PowerPC Developer List <linuxppc-dev@lists.ozlabs.org>
+Description:	read only
+		This sysfs file exposes the cpumask which is designated to make
+		HCALLs to retrive hv-24x7 pmu event counter data.
+
 What:		/sys/bus/event_source/devices/hv_24x7/event_descs/<event-name>
 Date:		February 2014
 Contact:	Linux on PowerPC Developer List <linuxppc-dev@lists.ozlabs.org>

diff --git a/arch/powerpc/perf/hv-24x7.c b/arch/powerpc/perf/hv-24x7.c
index ce4739e2b407..3c699612d29f 100644
--- a/arch/powerpc/perf/hv-24x7.c
+++ b/arch/powerpc/perf/hv-24x7.c

@@ -448,6 +448,12 @@ static ssize_t device_show_string(struct device *dev,
 	return sprintf(buf, "%s\n", (char *)d->var);
 }
 
+static ssize_t cpumask_get_attr(struct device *dev,
+				struct device_attribute *attr, char *buf)
+{
+	return cpumap_print_to_pagebuf(true, buf, &hv_24x7_cpumask);
+}
+
 static ssize_t sockets_show(struct device *dev,
 			    struct device_attribute *attr, char *buf)
 {

@@ -1116,6 +1122,17 @@ static DEVICE_ATTR_RO(sockets);
 static DEVICE_ATTR_RO(chipspersocket);
 static DEVICE_ATTR_RO(coresperchip);
 
+static DEVICE_ATTR(cpumask, S_IRUGO, cpumask_get_attr, NULL);
+
+static struct attribute *cpumask_attrs[] = {
+	&dev_attr_cpumask.attr,
+	NULL,
+};
+
+static struct attribute_group cpumask_attr_group = {
+	.attrs = cpumask_attrs,
+};
+
 static struct bin_attribute *if_bin_attrs[] = {
 	&bin_attr_catalog,
 	NULL,

@@ -1143,6 +1160,11 @@ static const struct attribute_group *attr_groups[] = {
 	&event_desc_group,
 	&event_long_desc_group,
 	&if_group,
+	/*
+	 * This NULL is a placeholder for the cpumask attr which will update
+	 * onlyif cpuhotplug registration is successful
+	 */
+	NULL,
 	NULL,
 };
 

@@ -1683,7 +1705,7 @@ static int hv_24x7_cpu_hotplug_init(void)
 
 static int hv_24x7_init(void)
 {
-	int r;
+	int r, i = -1;
 	unsigned long hret;
 	struct hv_perf_caps caps;
 

@@ -1727,8 +1749,18 @@ static int hv_24x7_init(void)
 
 	/* init cpuhotplug */
 	r = hv_24x7_cpu_hotplug_init();
-	if (r)
+	if (r) {
 		pr_err("hv_24x7: CPU hotplug init failed\n");
+	} else {
+		/*
+		 * Cpu hotplug init is successful, add the
+		 * cpumask file as part of pmu attr group and
+		 * assign it to very first NULL location.
+		 */
+		while (attr_groups[++i])
+			/* nothing */;
+		attr_groups[i] = &cpumask_attr_group;
+	}
 
 	r = perf_pmu_register(&h_24x7_pmu, h_24x7_pmu.name, -1);
 	if (r)

-- 
2.18.2

On Wed, Jun 24, 2020 at 03:47:53PM +0530, Kajol Jain wrote:

Patch here adds cpu hotplug functions to hv_24x7 pmu.
A new cpuhp_state "CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE" enum
is added.

The online function update the cpumask only if its NULL.

      ^^^^^^^^^^^^^^^^^^^^^^                     ^^^^^^^^^
	online callback function updates	 it is empty.
						 

As the primary intention for adding hotplug support
is to desiginate a CPU to make HCALL to collect the

        ^^^^^^^^^^
	designate
	

count data.

The offline function test and clear corresponding cpu in a cpumask
and update cpumask to any other active cpu.

Signed-off-by: Kajol Jain <redacted>

Otherwise, looks good to me.

Reviewed-by: Gautham R. Shenoy <redacted>

On 6/24/20 4:06 PM, Gautham R Shenoy wrote:

On Wed, Jun 24, 2020 at 03:47:53PM +0530, Kajol Jain wrote:

quoted

Patch here adds cpu hotplug functions to hv_24x7 pmu.
A new cpuhp_state "CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE" enum
is added.

The online function update the cpumask only if its NULL.

      ^^^^^^^^^^^^^^^^^^^^^^                     ^^^^^^^^^
	online callback function updates	 it is empty.
						 

quoted

As the primary intention for adding hotplug support
is to desiginate a CPU to make HCALL to collect the

        ^^^^^^^^^^
	designate
	

Sorry My bad. Thanks for reviewing the patch. Will correct these
mistakes.

Thanks,
Kajol Jain

quoted

count data.

The offline function test and clear corresponding cpu in a cpumask
and update cpumask to any other active cpu.

Signed-off-by: Kajol Jain <redacted>

Otherwise, looks good to me.

Reviewed-by: Gautham R. Shenoy <redacted>

quoted

---
 arch/powerpc/perf/hv-24x7.c | 45 +++++++++++++++++++++++++++++++++++++
 include/linux/cpuhotplug.h  |  1 +
 2 files changed, 46 insertions(+)

diff --git a/arch/powerpc/perf/hv-24x7.c b/arch/powerpc/perf/hv-24x7.c
index db213eb7cb02..ce4739e2b407 100644
--- a/arch/powerpc/perf/hv-24x7.c
+++ b/arch/powerpc/perf/hv-24x7.c

@@ -31,6 +31,8 @@ static int interface_version;
 /* Whether we have to aggregate result data for some domains. */
 static bool aggregate_result_elements;

+static cpumask_t hv_24x7_cpumask;
+
 static bool domain_is_valid(unsigned domain)
 {
 	switch (domain) {

@@ -1641,6 +1643,44 @@ static struct pmu h_24x7_pmu = {
 	.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
 };

+static int ppc_hv_24x7_cpu_online(unsigned int cpu)
+{
+	/* Make this CPU the designated target for counter collection */
+	if (cpumask_empty(&hv_24x7_cpumask))
+		cpumask_set_cpu(cpu, &hv_24x7_cpumask);
+
+	return 0;
+}
+
+static int ppc_hv_24x7_cpu_offline(unsigned int cpu)
+{
+	int target = -1;
+
+	/* Check if exiting cpu is used for collecting 24x7 events */
+	if (!cpumask_test_and_clear_cpu(cpu, &hv_24x7_cpumask))
+		return 0;
+
+	/* Find a new cpu to collect 24x7 events */
+	target = cpumask_last(cpu_active_mask);
+
+	if (target < 0 || target >= nr_cpu_ids)
+		return -1;
+
+	/* Migrate 24x7 events to the new target */
+	cpumask_set_cpu(target, &hv_24x7_cpumask);
+	perf_pmu_migrate_context(&h_24x7_pmu, cpu, target);
+
+	return 0;
+}
+
+static int hv_24x7_cpu_hotplug_init(void)
+{
+	return cpuhp_setup_state(CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,
+			  "perf/powerpc/hv_24x7:online",
+			  ppc_hv_24x7_cpu_online,
+			  ppc_hv_24x7_cpu_offline);
+}
+
 static int hv_24x7_init(void)
 {
 	int r;

@@ -1685,6 +1725,11 @@ static int hv_24x7_init(void)
 	if (r)
 		return r;

+	/* init cpuhotplug */
+	r = hv_24x7_cpu_hotplug_init();
+	if (r)
+		pr_err("hv_24x7: CPU hotplug init failed\n");
+
 	r = perf_pmu_register(&h_24x7_pmu, h_24x7_pmu.name, -1);
 	if (r)
 		return r;

diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 191772d4a4d7..a2710e654b64 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h

@@ -181,6 +181,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_POWERPC_CORE_IMC_ONLINE,
 	CPUHP_AP_PERF_POWERPC_THREAD_IMC_ONLINE,
 	CPUHP_AP_PERF_POWERPC_TRACE_IMC_ONLINE,
+	CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,
 	CPUHP_AP_WATCHDOG_ONLINE,
 	CPUHP_AP_WORKQUEUE_ONLINE,
 	CPUHP_AP_RCUTREE_ONLINE,

-- 
2.18.2

Hi Kajol,

On Wed, Jun 24, 2020 at 03:47:54PM +0530, Kajol Jain wrote:

Patch here adds a cpumask attr to hv_24x7 pmu along with ABI documentation.

command:# cat /sys/devices/hv_24x7/cpumask
0

Since this sysfs interface is read-only, and the user cannot change
the CPU which will be making the HCALLs to obtain the 24x7 counts,
does the user even need to know if currently CPU X is the one which is
going to make HCALLs to retrive the 24x7 counts ? Does it help in any
kind of trouble-shooting ?

It would have made sense if the interface was read-write, since a user
can set this to a CPU which is not running user applications. This
would help in minimising jitter on those active CPUs running the user
applications.

On 6/24/20 4:26 PM, Gautham R Shenoy wrote:

Hi Kajol,

On Wed, Jun 24, 2020 at 03:47:54PM +0530, Kajol Jain wrote:

quoted

Patch here adds a cpumask attr to hv_24x7 pmu along with ABI documentation.

command:# cat /sys/devices/hv_24x7/cpumask
0

Since this sysfs interface is read-only, and the user cannot change
the CPU which will be making the HCALLs to obtain the 24x7 counts,
does the user even need to know if currently CPU X is the one which is
going to make HCALLs to retrive the 24x7 counts ? Does it help in any
kind of trouble-shooting ?

Primary use to expose the cpumask is for the perf tool.
Which has the capability to parse the driver sysfs folder
and understand the cpumask file. Having cpumask
file will reduce the number of perf commandline
parameters (will avoid "-C" option in the perf tool
command line). I can also notify the user which is
the current cpu used to retrieve the counter data.

It would have made sense if the interface was read-write, since a user
can set this to a CPU which is not running user applications. This
would help in minimising jitter on those active CPUs running the user
applications.

With cpumask backed by hotplug
notifiers, enabling user write access to it will
complicate the code with more additional check.
CPU will come to play only if the user request for
counter data. If not, then there will be no HCALLs made
using the CPU.

Maddy

quoted

Signed-off-by: Kajol Jain <redacted>
---
  .../sysfs-bus-event_source-devices-hv_24x7    |  7 ++++
  arch/powerpc/perf/hv-24x7.c                   | 36 +++++++++++++++++--
  2 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7 b/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7
index e8698afcd952..f9dd3755b049 100644
--- a/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7
+++ b/Documentation/ABI/testing/sysfs-bus-event_source-devices-hv_24x7

@@ -43,6 +43,13 @@ Description:	read only
  		This sysfs interface exposes the number of cores per chip
  		present in the system.

+What:		/sys/devices/hv_24x7/cpumask
+Date:		June 2020
+Contact:	Linux on PowerPC Developer List <linuxppc-dev@lists.ozlabs.org>
+Description:	read only
+		This sysfs file exposes the cpumask which is designated to make
+		HCALLs to retrive hv-24x7 pmu event counter data.
+
  What:		/sys/bus/event_source/devices/hv_24x7/event_descs/<event-name>
  Date:		February 2014
  Contact:	Linux on PowerPC Developer List <linuxppc-dev@lists.ozlabs.org>

diff --git a/arch/powerpc/perf/hv-24x7.c b/arch/powerpc/perf/hv-24x7.c
index ce4739e2b407..3c699612d29f 100644
--- a/arch/powerpc/perf/hv-24x7.c
+++ b/arch/powerpc/perf/hv-24x7.c

@@ -448,6 +448,12 @@ static ssize_t device_show_string(struct device *dev,
  	return sprintf(buf, "%s\n", (char *)d->var);
  }

+static ssize_t cpumask_get_attr(struct device *dev,
+				struct device_attribute *attr, char *buf)
+{
+	return cpumap_print_to_pagebuf(true, buf, &hv_24x7_cpumask);
+}
+
  static ssize_t sockets_show(struct device *dev,
  			    struct device_attribute *attr, char *buf)
  {

@@ -1116,6 +1122,17 @@ static DEVICE_ATTR_RO(sockets);
  static DEVICE_ATTR_RO(chipspersocket);
  static DEVICE_ATTR_RO(coresperchip);

+static DEVICE_ATTR(cpumask, S_IRUGO, cpumask_get_attr, NULL);
+
+static struct attribute *cpumask_attrs[] = {
+	&dev_attr_cpumask.attr,
+	NULL,
+};
+
+static struct attribute_group cpumask_attr_group = {
+	.attrs = cpumask_attrs,
+};
+
  static struct bin_attribute *if_bin_attrs[] = {
  	&bin_attr_catalog,
  	NULL,

@@ -1143,6 +1160,11 @@ static const struct attribute_group *attr_groups[] = {
  	&event_desc_group,
  	&event_long_desc_group,
  	&if_group,
+	/*
+	 * This NULL is a placeholder for the cpumask attr which will update
+	 * onlyif cpuhotplug registration is successful
+	 */
+	NULL,
  	NULL,
  };

@@ -1683,7 +1705,7 @@ static int hv_24x7_cpu_hotplug_init(void)

  static int hv_24x7_init(void)
  {
-	int r;
+	int r, i = -1;
  	unsigned long hret;
  	struct hv_perf_caps caps;

@@ -1727,8 +1749,18 @@ static int hv_24x7_init(void)

  	/* init cpuhotplug */
  	r = hv_24x7_cpu_hotplug_init();
-	if (r)
+	if (r) {
  		pr_err("hv_24x7: CPU hotplug init failed\n");
+	} else {
+		/*
+		 * Cpu hotplug init is successful, add the
+		 * cpumask file as part of pmu attr group and
+		 * assign it to very first NULL location.
+		 */
+		while (attr_groups[++i])
+			/* nothing */;
+		attr_groups[i] = &cpumask_attr_group;
+	}

  	r = perf_pmu_register(&h_24x7_pmu, h_24x7_pmu.name, -1);
  	if (r)

-- 
2.18.2

On Wed, Jun 24, 2020 at 05:58:31PM +0530, Madhavan Srinivasan wrote:

On 6/24/20 4:26 PM, Gautham R Shenoy wrote:

quoted

Hi Kajol,

On Wed, Jun 24, 2020 at 03:47:54PM +0530, Kajol Jain wrote:

quoted

Patch here adds a cpumask attr to hv_24x7 pmu along with ABI documentation.

command:# cat /sys/devices/hv_24x7/cpumask
0

Since this sysfs interface is read-only, and the user cannot change
the CPU which will be making the HCALLs to obtain the 24x7 counts,
does the user even need to know if currently CPU X is the one which is
going to make HCALLs to retrive the 24x7 counts ? Does it help in any
kind of trouble-shooting ?

Primary use to expose the cpumask is for the perf tool.
Which has the capability to parse the driver sysfs folder
and understand the cpumask file. Having cpumask
file will reduce the number of perf commandline
parameters (will avoid "-C" option in the perf tool
command line). I can also notify the user which is
the current cpu used to retrieve the counter data.

Fair enough. Can we include this in the patch description ?

quoted

It would have made sense if the interface was read-write, since a user
can set this to a CPU which is not running user applications. This
would help in minimising jitter on those active CPUs running the user
applications.

With cpumask backed by hotplug
notifiers, enabling user write access to it will
complicate the code with more additional check.
CPU will come to play only if the user request for
counter data. If not, then there will be no HCALLs made
using the CPU.

Well, I was wondering if you could make the interface writable because
I couldn't think of the use of a read-only interface. With the
perf-use case you have provided, I guess it makes sense. I am ok with
it being a read-only interface.

Maddy

--
Thanks and Regards
gautham.

On 6/26/20 1:15 PM, Gautham R Shenoy wrote:

On Wed, Jun 24, 2020 at 05:58:31PM +0530, Madhavan Srinivasan wrote:

quoted

On 6/24/20 4:26 PM, Gautham R Shenoy wrote:

quoted

Hi Kajol,

On Wed, Jun 24, 2020 at 03:47:54PM +0530, Kajol Jain wrote:

quoted

Patch here adds a cpumask attr to hv_24x7 pmu along with ABI documentation.

command:# cat /sys/devices/hv_24x7/cpumask
0

Since this sysfs interface is read-only, and the user cannot change
the CPU which will be making the HCALLs to obtain the 24x7 counts,
does the user even need to know if currently CPU X is the one which is
going to make HCALLs to retrive the 24x7 counts ? Does it help in any
kind of trouble-shooting ?

Primary use to expose the cpumask is for the perf tool.
Which has the capability to parse the driver sysfs folder
and understand the cpumask file. Having cpumask
file will reduce the number of perf commandline
parameters (will avoid "-C" option in the perf tool
command line). I can also notify the user which is
the current cpu used to retrieve the counter data.

Fair enough. Can we include this in the patch description ?

Sure will update in next version of patchset.

Thanks,
Kajol Jain

quoted

quoted

It would have made sense if the interface was read-write, since a user
can set this to a CPU which is not running user applications. This
would help in minimising jitter on those active CPUs running the user
applications.

With cpumask backed by hotplug
notifiers, enabling user write access to it will
complicate the code with more additional check.
CPU will come to play only if the user request for
counter data. If not, then there will be no HCALLs made
using the CPU.

Well, I was wondering if you could make the interface writable because
I couldn't think of the use of a read-only interface. With the
perf-use case you have provided, I guess it makes sense. I am ok with
it being a read-only interface.

quoted

Maddy

--
Thanks and Regards
gautham.