On Fri, 10 Oct 2025 14:50:57 +0100,
Thierry Reding [off-list ref] wrote:

On Thu, Oct 09, 2025 at 07:11:20PM +0100, Marc Zyngier wrote:

quoted

On Thu, 09 Oct 2025 18:04:58 +0100,
Marc Zyngier [off-list ref] wrote:

quoted

On Thu, 09 Oct 2025 17:05:15 +0100,
Thierry Reding [off-list ref] wrote:

quoted

[1  <text/plain; us-ascii (quoted-printable)>]
On Thu, Oct 09, 2025 at 03:30:56PM +0100, Marc Zyngier wrote:

quoted

Hi Thierry,

On Thu, 09 Oct 2025 12:38:55 +0100,
Thierry Reding [off-list ref] wrote:

quoted

Which brings me to the actual question: what is the right way to solve
this? I had, maybe naively, assumed that the default CPU affinity, which
includes all available CPUs, would be sufficient to have interrupts
balanced across all of those CPUs, but that doesn't appear to be the
case. At least not with the GIC (v3) driver which selects one CPU (CPU 0
in this particular case) from the affinity mask to set the "effective
affinity", which then dictates where IRQs are handled and where the
corresponding IRQ thread function is run.

There's a (GIC-specific) answer to that, and that's the "1 of N"
distribution model. The problem is that it is a massive headache (it
completely breaks with per-CPU context).

Heh, that started out as a very promising first paragraph but turned
ugly very quickly... =)

quoted

We could try and hack this in somehow, but defining a reasonable API
is complicated. The set of CPUs receiving 1:N interrupts is a *global*
set, which means you cannot have one interrupt targeting CPUs 0-1, and
another targeting CPUs 2-3. You can only have a single set for all 1:N
interrupts. How would you define such a set in a platform agnostic
manner so that a random driver could use this? I definitely don't want
to have a GIC-specific API.

I see. I've been thinking that maybe the only way to solve this is using
some sort of policy. A very simple policy might be: use CPU 0 as the
"default" interrupt (much like it is now) because like you said there
might be assumptions built-in that break when the interrupt is scheduled
elsewhere. But then let individual drivers opt into the 1:N set, which
would perhaps span all available CPUs but the first one. From an API PoV
this would just be a flag that's passed to request_irq() (or one of its
derivatives).

The $10k question is how do you pick the victim CPUs? I can't see how
to do it in a reasonable way unless we decide that interrupts that
have an affinity matching cpu_possible_mask are 1:N. And then we're
left with wondering what to do about CPU hotplug.

For fun and giggles, here's the result of a 5 minute hack. It enables
1:N distribution on SPIs that have an "all cpus" affinity. It works on
one machine, doesn't on another -- no idea why yet. YMMV.

This is of course conditioned on your favourite HW supporting the 1:N
feature, and it is likely that things will catch fire quickly. It will
probably make your overall interrupt latency *worse*, but maybe less
variable. Let me know.

You might be onto something here. Mind you, I've only done very limited
testing, but the system does boot and the QSPI related timeouts are gone
completely.

Hey, progress.

Here's some snippets from the boot log that might be interesting:

[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GIC: enabling workaround for GICv3: NVIDIA erratum T241-FABRIC-4
[    0.000000] GIC: enabling workaround for GICv3: ARM64 erratum 2941627
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 320 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, 1:N
[    0.000000] GICv3: CPU0: found redistributor 20000 region 0:0x0000000022100000
[...]
[    0.000000] GICv3: using LPI property table @0x0000000101500000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000101540000
[...]

There's a bunch of ITS info that I dropped, as well as the same
redistributor and LPI property table block for each of the 288 CPUs.

/proc/interrupts is much too big to paste here, but it looks like the
QSPI interrupts now end up evenly distributed across the first 72 CPUs
in this system. Not sure why 72, but possibly because this is a 4 NUMA
node system with 72 CPUs each, so the CPU mask might've been restricted
to just the first node.

It could well be that your firmware sets GICR_CTLR.DPG1NS on the 3
other nodes, and the patch I gave you doesn't try to change that.
Check with [1], which does the right thing on that front (it fixed a
similar problem on my slightly more modest 12 CPU machine).

On the face of it this looks quite promising. Where do we go from here?

For a start, you really should consider sending me one of these
machines. I have plans for it ;-)

Any areas that we need to test more exhaustively to see if this breaks?

CPU hotplug is the main area of concern, and I'm pretty sure it breaks
this distribution mechanism (or the other way around). Another thing
is that if firmware isn't aware that 1:N interrupts can (or should)
wake-up a CPU from sleep, bad things will happen. Given that nobody
uses 1:N, you can bet that any bit of privileged SW (TF-A,
hypervisors) is likely to be buggy (I've already spotted bugs in KVM
around this).

The other concern is the shape of the API we would expose to drivers,
because I'm not sure we want this sort of "scatter-gun" approach for
all SPIs, and I don't know how that translates to other architectures.

Thomas should probably weight in here.

Thanks,

	M.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=irq/gicv3-1ofN&id=5856e2eb479fc41ea60e76440f768079a1a21a36

-- 
Without deviation from the norm, progress is not possible.