Re: [PATCH 4/7] thermal: intel: hfi: Handle CPU hotplug events

On Tue, Nov 30, 2021 at 2:22 PM Ricardo Neri
[off-list ref] wrote:

On Wed, Nov 24, 2021 at 03:48:49PM +0100, Rafael J. Wysocki wrote:

quoted

On Sat, Nov 6, 2021 at 2:34 AM Ricardo Neri
[off-list ref] wrote:

[cut]

quoted

quoted

+/**
+ * intel_hfi_offline() - Disable HFI on @cpu
+ * @cpu:       CPU in which the HFI will be disabled
+ *
+ * Remove @cpu from those covered by its HFI instance.
+ *
+ * On some processors, hardware remembers previous programming settings even
+ * after being reprogrammed. Thus, keep HFI enabled even if all CPUs in the
+ * die/package of @cpu are offline. See note in intel_hfi_online().
+ */
+void intel_hfi_offline(unsigned int cpu)
+{
+       struct cpumask *die_cpumask = topology_core_cpumask(cpu);
+       struct hfi_cpu_info *info = &per_cpu(hfi_cpu_info, cpu);
+       struct hfi_instance *hfi_instance;
+
+       if (!boot_cpu_has(X86_FEATURE_INTEL_HFI))
+               return;
+
+       hfi_instance = info->hfi_instance;
+       if (!hfi_instance)
+               return;
+
+       if (!hfi_instance->initialized)
+               return;
+
+       mutex_lock(&hfi_lock);
+
+       /*
+        * We were using the core cpumask of @cpu to track CPUs in the same
+        * die/package. Now it is going offline and we need to find another
+        * CPU we can use.
+        */
+       if (die_cpumask == hfi_instance->cpus) {
+               int new_cpu;
+
+               new_cpu = cpumask_any_but(hfi_instance->cpus, cpu);
+               if (new_cpu >= nr_cpu_ids)
+                       /* All other CPUs in the package are offline. */
+                       hfi_instance->cpus = NULL;
+               else
+                       hfi_instance->cpus = topology_core_cpumask(new_cpu);

Hmmm.  Is topology_core_cpumask() updated when CPUs go offline and online?

Yes. A CPU going offline is cleared from its siblings' cpumask [1] and its own [2]
in remove_siblinginfo() via cpu_disable_common(). A CPU going online is set
in its siblings' cpumask and its own in set_cpu_sibling_map() [3].

OK, so it is necessary to ensure that intel_hfi_offline() will always
run after remove_siblinginfo() so it sees the updated mask.  How do we
ensure that?

[1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n1592
[2]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n1617
[3]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n657