Re: Candidate Linux ABI for Intel AMX and hypothetical new related features

* Dave Hansen:

On 5/21/21 7:44 AM, Florian Weimer wrote:

quoted

* Dave Hansen via Libc-alpha:

quoted

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Why can't userspace look at XCR0 to make the decision?

The thing we're trying to avoid is a #NM exception from XFD (the new
first-use detection feature) that occurs on the first use of AMX.
XCR0 will have XCR0[AMX]=1, even if XFD is "armed" and ready to
generate the #NM.

I see.  So essentially the hardware wants to offer transparent
initialize-on-use, but Linux does not seem to want to implement it this
way.

Is there still a chance to bring the hardware and Linux into alignment?

Thanks,
Florian

On Fri, May 21, 2021 at 12:19 PM Florian Weimer [off-list ref] wrote:

I see.  So essentially the hardware wants to offer transparent
initialize-on-use, but Linux does not seem to want to implement it this
way.

That is a reasonable summary.

Is there still a chance to bring the hardware and Linux into alignment?

The hardware was done some time ago, so this is a Linux decision.

The current trajectory is that for user space to use TMUL it must
1. query CPUID to see if the instructions exist
2. query xgetbv(XCR0) to see if the OS supports the state
3. Tell Linux that this thread wants to use the state.

The original proposal required just #1 and #2.
It is clear that Linux can not support that, and so #3 is being added.

Of course, if #2 is false, then Linux will return failure for #3,
so technically you could skip that check and just make this new syscall.
Probably user-space will still need to query CPUID for the instructions,
since there will be a many-to-one mapping of instructions to state.

Len Brown, Intel Open Source Technology Center

On 5/21/21 9:19 AM, Florian Weimer wrote:

quoted

On 5/21/21 7:44 AM, Florian Weimer wrote:

quoted

* Dave Hansen via Libc-alpha:

quoted

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Why can't userspace look at XCR0 to make the decision?

The thing we're trying to avoid is a #NM exception from XFD (the new
first-use detection feature) that occurs on the first use of AMX.
XCR0 will have XCR0[AMX]=1, even if XFD is "armed" and ready to
generate the #NM.

I see.  So essentially the hardware wants to offer transparent
initialize-on-use, but Linux does not seem to want to implement it this
way.

I don't quite see it that way.  The hardware wants to offer the OS a
guarantee that it will know *BEFORE* an application tried to establish
specific register state.

An OS could implement relatively transparent XSAVE backing resizing with
it, like the earlier AMX patches did.

Or, the OS could use it to implement a nice, immediate thwack if the app
misbehaves and violates the ABI, like we're moving toward now.

Is there still a chance to bring the hardware and Linux into alignment?

I think they're aligned just fine.  XFD might be a bit overblown as a
feature for how Linux will use it, but other OSes might get some mileage
out of it.

On Fri, May 21, 2021, at 9:19 AM, Florian Weimer wrote:

* Dave Hansen:

quoted

On 5/21/21 7:44 AM, Florian Weimer wrote:

quoted

* Dave Hansen via Libc-alpha:

quoted

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Why can't userspace look at XCR0 to make the decision?

The thing we're trying to avoid is a #NM exception from XFD (the new
first-use detection feature) that occurs on the first use of AMX.
XCR0 will have XCR0[AMX]=1, even if XFD is "armed" and ready to
generate the #NM.

I see.  So essentially the hardware wants to offer transparent
initialize-on-use, but Linux does not seem to want to implement it this
way.

Is there still a chance to bring the hardware and Linux into alignment?

arch_prctl(SET_XSTATE_INIT_ON_FIRST_USE, TILE_STUFF);?

As long as this is allowed to fail, I don’t have a huge problem with it.

I think several things here are regrettable:

1. Legacy XSTATE code might assume that XCR0 is a constant.

2. Intel virt really doesn’t like us context switching XCR0, although we might say that this is Intel’s fault and therefore Intel’s problem. AMD hardware doesn’t appear to have this issue.

3. AMX bring tangled up in XSTATE is unfortunate.  The whole XSTATE mechanism is less than amazing.

IMO the best we can make of this whole situation is to make XCR0 dynamic, but the legacy compatibility issues are potentially problematic.

Thanks,
Florian

On Fri, May 21 2021 at 18:19, Florian Weimer wrote:

* Dave Hansen:

quoted

On 5/21/21 7:44 AM, Florian Weimer wrote:

quoted

Why can't userspace look at XCR0 to make the decision?

The thing we're trying to avoid is a #NM exception from XFD (the new
first-use detection feature) that occurs on the first use of AMX.
XCR0 will have XCR0[AMX]=1, even if XFD is "armed" and ready to
generate the #NM.

I see.  So essentially the hardware wants to offer transparent
initialize-on-use, but Linux does not seem to want to implement it this
way.

The hardware offers an exception which can be used to implement that,
but the hardware does not dictate that usage.

If we'd go that way we lost any control over that resource and I can
demonstrate with AVX512 today what kind of consequences that has with
mixed criticality realtime workloads.

The only solution we have today is to disable AVX512 completely, which
sucks because restricted usage can be benefitial for some of the
computations.

The problem is that the approach of user space in general seems to be
blindly_select_max(AVX). I've seen that in quite some places.

With AMX (and the stuff coming next) we have the chance to do proper
resource control and it would be outright stupid not to take that
opportunity.

Thanks,

        tglx

On Fri, May 21 2021 at 09:31, Andy Lutomirski wrote:

arch_prctl(SET_XSTATE_INIT_ON_FIRST_USE, TILE_STUFF);?

As long as this is allowed to fail, I don’t have a huge problem with
it.

I'm fine with that. It's still controlled by the OS and can return
-EPERM.

If allowed then the application would also accept to be insta killed if
that #NM allocation fails. Any bug report vs. that will be ignored.

I think several things here are regrettable:

1. Legacy XSTATE code might assume that XCR0 is a constant.

2. Intel virt really doesn’t like us context switching XCR0, although
we might say that this is Intel’s fault and therefore Intel’s
problem. AMD hardware doesn’t appear to have this issue.

3. AMX bring tangled up in XSTATE is unfortunate.  The whole XSTATE
mechanism is less than amazing.

IMO the best we can make of this whole situation is to make XCR0
dynamic, but the legacy compatibility issues are potentially
problematic.

Why? The bit can be enabled and #NM catches the violation of the ABI
contract if the application did not request usage. No XCR0 fiddling on
context switch required.

Thanks,

        tglx

On Fri, May 21, 2021, at 12:10 PM, Thomas Gleixner wrote:

On Fri, May 21 2021 at 09:31, Andy Lutomirski wrote:

quoted

arch_prctl(SET_XSTATE_INIT_ON_FIRST_USE, TILE_STUFF);?

As long as this is allowed to fail, I don’t have a huge problem with
it.

I'm fine with that. It's still controlled by the OS and can return
-EPERM.

If allowed then the application would also accept to be insta killed if
that #NM allocation fails. Any bug report vs. that will be ignored.

quoted

I think several things here are regrettable:

1. Legacy XSTATE code might assume that XCR0 is a constant.

2. Intel virt really doesn’t like us context switching XCR0, although
we might say that this is Intel’s fault and therefore Intel’s
problem. AMD hardware doesn’t appear to have this issue.

3. AMX bring tangled up in XSTATE is unfortunate.  The whole XSTATE
mechanism is less than amazing.

IMO the best we can make of this whole situation is to make XCR0
dynamic, but the legacy compatibility issues are potentially
problematic.

Why? The bit can be enabled and #NM catches the violation of the ABI
contract if the application did not request usage. No XCR0 fiddling on
context switch required.

Thanks,

        tglx

XFD does nothing about signals.  It also doesn’t help give applications a non-Linux-specific way to ask if AMX is available. The SDM says that one can read XCR0.  Sure, we can use it, but cross platform libraries seem likely to get it wrong.

On Fri, May 21 2021 at 13:07, Andy Lutomirski wrote:

On Fri, May 21, 2021, at 12:10 PM, Thomas Gleixner wrote:

quoted

Why? The bit can be enabled and #NM catches the violation of the ABI
contract if the application did not request usage. No XCR0 fiddling on
context switch required.

XFD does nothing about signals.

It's a matter of what's implemented in #NM. XFD just arms #NM

It also doesn’t help give applications a non-Linux-specific way to ask
if AMX is available. The SDM says that one can read XCR0.  Sure, we
can use it, but cross platform libraries seem likely to get it wrong.

Well, that's the inevitable consequence of Intel declaring that
everything needs to be exposed unconditionally for the very wrong
reasons.

Thanks,

        tglx

On Fri, May 21, 2021 at 3:10 PM Thomas Gleixner [off-list ref] wrote:

On Fri, May 21 2021 at 09:31, Andy Lutomirski wrote:

quoted

arch_prctl(SET_XSTATE_INIT_ON_FIRST_USE, TILE_STUFF);?

As long as this is allowed to fail, I don’t have a huge problem with
it.

I'm fine with that. It's still controlled by the OS and can return
-EPERM.

If allowed then the application would also accept to be insta killed if
that #NM allocation fails. Any bug report vs. that will be ignored.

Regarding pre-allocation vs on-demand allocation, consider two scenarios:

1. Synchronous.  At process or thread start up time, prctl()
synchronously allocates 8K context switch buffers.  Return code is 0
-- good go go!  10 seconds later the program decides to create
additional threads.  Woops. vmalloc failed, and the process
synchronously dies.  bug filed.

2. On demand.  Same scenario, except vmalloc failure upon creation of
those additional threads sends a SIGSEGV at the instruction where AMX
is touched.  bug filed.

Why ignore the 2nd bug and not ignore the 1st bug?

My concern about synchronous allocation is that it will be very easy
to abuse.  programs and threads can ask for buffers they will never
use.  With on-demand allocation, we allocate buffers only if they are
actually needed.

Len Brown, Intel Open Source Technology Center

On Fri, May 21 2021 at 18:07, Len Brown wrote:

On Fri, May 21, 2021 at 3:10 PM Thomas Gleixner [off-list ref] wrote:
Regarding pre-allocation vs on-demand allocation, consider two scenarios:

1. Synchronous.  At process or thread start up time, prctl()
synchronously allocates 8K context switch buffers.  Return code is 0
-- good go go!  10 seconds later the program decides to create
additional threads.  Woops. vmalloc failed, and the process
synchronously dies.  bug filed.

No. pthread_create() will fail with -ENOMEM. A return value of
-ENOMEM is not a bug. 

If the application fails to check the error code then it's not the
kernels problem and not a kernel bug either.

2. On demand.  Same scenario, except vmalloc failure upon creation of
those additional threads sends a SIGSEGV at the instruction where AMX
is touched.  bug filed.

Why ignore the 2nd bug and not ignore the 1st bug?

See above.

My concern about synchronous allocation is that it will be very easy
to abuse.  programs and threads can ask for buffers they will never
use.  With on-demand allocation, we allocate buffers only if they are
actually needed.

Programs ask for memory in various ways. The buffer is not any different
than any other memory allocation of the application/thread. It's
accounted for and when the limits are reached the allocation fails.

But it fails in a way which can be acted upon at the application level
and not in a way where the kernel has no other choice than killing the
whole process.

So where is the problem? 

Thanks,

        tglx

On 5/21/21 3:07 PM, Len Brown wrote:

My concern about synchronous allocation is that it will be very easy
to abuse.  programs and threads can ask for buffers they will never
use.  With on-demand allocation, we allocate buffers only if they are
actually needed.

If someone wants to abuse the on-demand allocation, they will simply
write a single bit to an AMX register.  That does *NOT* mean they will
actually execute an instruction that actually uses AMX to do something
meaningful.

In the face of abuse, I think the two approaches are very similar.

On Fri, May 21, 2021 at 7:06 PM Dave Hansen [off-list ref] wrote:

On 5/21/21 3:07 PM, Len Brown wrote:

quoted

My concern about synchronous allocation is that it will be very easy
to abuse.  programs and threads can ask for buffers they will never
use.  With on-demand allocation, we allocate buffers only if they are
actually needed.

If someone wants to abuse the on-demand allocation, they will simply
write a single bit to an AMX register.  That does *NOT* mean they will
actually execute an instruction that actually uses AMX to do something
meaningful.

In the face of abuse, I think the two approaches are very similar.

I didn't mean "abuse" in terms of malicious resource hogging.
I meant "abuse" in terms of unnecessarily using resources out of laziness.

With this proposed API, we seem to be combining two requirements, and
I wonder if we should be treating them independently.

Requirement 1: "Fine grained control".  We want the kernel to be able
to prohibit a program from using AMX.  The foundation for this is a
system call that the kernel can say "No".  It may deny access for
whatever reason it wants, including inability to allocate a buffer, or
some TBD administer-invoked hook in the system call, say membership or
lack of membership of the process in an empowered cgroup.

Requirement 2: Ability to synchronously fail upon buffer allocation.
I agree that pthread_create() returning an error code is more friendly
way to kill a program rather than a SIGSEGV when touching AMX state
for the first time.  But the reality is, that program is almost
certainly going to exit either way.

So the 1st question is if the system call requesting permission should
be on a per-process basis, or a per-task basis.

A. per-task.  If we do it this way, then we will likely wind up
mandating a GET at the start of every routine in every library that
touches AMX, and potentially also a PUT.  This is because the library
has no idea what thread called it.  The plus is that this will address
the "used once and sits on a buffer for the rest of the process
lifetime' scenario.  The minus is that high performance users will be
executing thousands of unnecessary system calls that have zero value.

B. per-process.  If we do it this way, then the run time linker can do
a single system call on behalf of the entire process, and there is no
need to sprinkle system calls throughout the library.  Presumably the
startup code would query CPUID, query XCR0, query this system call,
and set a global variable to access by all threads going forward.  The
plus is that permission makes more sense on a process basis than on a
task basis.  Why would the kernel give one thread in a process
permission, and not another thread -- and if that happened, would a
process actually be able to figure out what to do?  If we do
per-process, I don't see that the PUT call would be useful, and I
would skip it.

Neither A or B has an advantage in the situation where a thread is

30 more lines