Re: [PATCH v2 0/8] Cache coherency management subsystem

Jonathan Cameron wrote:

On Wed, 25 Jun 2025 18:03:43 +0100
Jonathan Cameron [off-list ref] wrote:

quoted

On Wed, 25 Jun 2025 11:31:52 +0200
Peter Zijlstra [off-list ref] wrote:

quoted

On Wed, Jun 25, 2025 at 02:12:39AM -0700, H. Peter Anvin wrote:  

quoted

On June 25, 2025 1:52:04 AM PDT, Peter Zijlstra [off-list ref] wrote:    

quoted

On Tue, Jun 24, 2025 at 04:47:56PM +0100, Jonathan Cameron wrote:
   

quoted

On x86 there is the much loved WBINVD instruction that causes a write back
and invalidate of all caches in the system. It is expensive but it is    

Expensive is not the only problem. It actively interferes with things
like Cache-Allocation-Technology (RDT-CAT for the intel folks). Doing
WBINVD utterly destroys the cache subsystem for everybody on the
machine.
   

quoted

necessary in a few corner cases.     

Don't we have things like CLFLUSH/CLFLUSHOPT/CLWB exactly so that we can
avoid doing dumb things like WBINVD ?!?
   

quoted

These are cases where the contents of
Physical Memory may change without any writes from the host. Whilst there
are a few reasons this might happen, the one I care about here is when
we are adding or removing mappings on CXL. So typically going from
there being actual memory at a host Physical Address to nothing there
(reads as zero, writes dropped) or visa-versa.     

quoted

The
thing that makes it very hard to handle with CPU flushes is that the
instructions are normally VA based and not guaranteed to reach beyond
the Point of Coherence or similar. You might be able to (ab)use
various flush operations intended to ensure persistence memory but
in general they don't work either.    

Urgh so this. Dan, Dave, are we getting new instructions to deal with
this? I'm really not keen on having WBINVD in active use.
   

WBINVD is the nuclear weapon to use when you have lost all notion of
where the problematic data can be, and amounts to a full reset of the
cache system. 

WBINVD can block interrupts for many *milliseconds*, system wide, and
so is really only useful for once-per-boot type events, like MTRR
initialization.    

Right this... But that CXL thing sounds like that's semi 'regular' to
the point that providing some infrastructure around it makes sense. This
should not be.  

I'm fully on board with the WBINVD issues (and hope for something new for
the X86 world). However, this particular infrastructure (for those systems
that can do so) is about pushing the problem and information to where it
can be handled in a lot less disruptive fashion. It can take 'a while' but
we are flushing only cache entries in the requested PA range. Other than
some potential excess snoop traffic if the coherency tracking isn't precise,
there should be limited affect on the rest of the system.

So, for the systems I particularly care about, the CXL case isn't that bad.

Just for giggles, if you want some horror stories the (dropped) ARM PSCI
spec provides for approaches that require synchronization of calls across
all CPUs.

"CPU Rendezvous" in the attributes of CLEAN_INV_MEMREGION requires all
CPUs to make a call within an impdef (discoverable) timeout.
https://developer.arm.com/documentation/den0022/falp1/?lang=en

I gather no one actually needs that on 'real' systems - that is systems
where we actually need to do these flushes! The ACPI 'RFC' doesn't support
that delight.

Seems I introduced some confusion.  Let me try summarizing:

1. x86 has a potential feature gap. From a CXL ecosystem point of view I'd
   like to see that gap closed. (Inappropriate for me to make any proposals
   on how to do it on that architecture).

I disagree this is an x86 feature gap. This is CXL exporting complexity
to Linux. Linux is better served in the long term by CXL cleaning up
that problem than Linux deploying more software mitigations.

2. This patch set has nothing to do with x86 (beyond modifying a function
   signature). The hardware it is targeting avoids many of the issues around
   WBINVD. The solution is not specific to ARM64, though the implementation
   I care about is on an ARM64 implementation.

Right now, on x86 we have a functionally correct solution, this patch set
adds infrastructure and 2 implementations to provide similar for other
architectures.

Theoretically there could be a threshold at which a CLFLUSHOPT loop is a
better option, but I would rather it be the case* that software CXL
cache management is stop-gap for early generation CXL platforms.

* personal kernel developer opinion, not necessarily opinion of $employer