Re: Candidate Linux ABI for Intel AMX and hypothetical new related features

Len,

On Sun, May 02 2021 at 11:27, Len Brown wrote:

Here is how it works:

1. The kernel boots and sees the feature in CPUID.

2. If the kernel supports that feature, it sets XCR0[feature].

    For some features, there may be a bunch of kernel support,
    while simple features may require only state save/restore.

2a.  If the kernel doesn't support the feature, XCR0[feature] remains cleared.

3. user-space sees the feature in CPUID

4. user-space sees for the feature via xgetbv[XCR0]

5. If the feature is enabled in XCR0, the user happily uses it.

    For AMX, Linux implements "transparent first use"
    so that it doesn't have to allocate 8KB context switch
    buffers for tasks that don't actually use AMX.
    It does this by arming XFD for all tasks, and taking a #NM
    to allocate a context switch buffer only for those tasks
    that actually execute AMX instructions.

I thought more about this and it's absolutely the wrong way to go for
several reasons.

AMX (or whatever comes next) is nothing else than a device and it
just should be treated as such. The fact that it is not exposed
via a driver and a device node does not matter at all.

Not doing so requires this awkward buffer allocation issue via #NM with
all it's downsides; it's just wrong to force the kernel to manage
resources of a user space task without being able to return a proper
error code. 

It also prevents fine grained control over access to this
functionality. As AMX is clearly a shared resource which is not per HT
thread (maybe not even per core) and it has impact on power/frequency it
is important to be able to restrict access on a per process/cgroup
scope.

Having a proper interface (syscall, prctl) which user space can use to
ask for permission and allocation of the necessary buffer(s) is clearly
avoiding the downsides and provides the necessary mechanisms for proper
control and failure handling.

It's not the end of the world if something which wants to utilize this
has do issue a syscall during detection. It does not matter whether
that's a library or just the application code itself.

That's a one off operation and every involved entity can cache the
result in TLS.

AVX512 has already proven that XSTATE management is fragile and error
prone, so we really have to stop this instead of creating yet another
half baken solution.

Thanks,

        tglx

Having a proper interface (syscall, prctl) which user space can use to
ask for permission and allocation of the necessary buffer(s) is clearly
avoiding the downsides and provides the necessary mechanisms for proper
control and failure handling.

this would need to be a "get / put" interface, so a refcount; that way things nest nicely.
For API symmetry I'd want to have the put there, even if we may decide to be infinitely lazy
in cleaning up the state.

it also would want it to take an arguement that's a bitmask, so that this can be applied
to future state as well.

Eh actually I'd start with also adding AVX512 to this. Even though for obvious compat reasons
that one is on by default (so at process start we might need to start with a count of 1)
it's interesting to fold that into this same framework.
(and who knows, dropping AVX512 state if you don't need it might improve context switches)

Syscalls are relatively cheap (and I can imagine the C library doing a TLS cache of the count
if it becomes an issue) so can be done on a relatively finegrained level.

I've worked on OpenBLAS before, and that library basically has a global initialization function
that ends up getting called on the first big math op (it may spawn threads as well etc) but which
"stays around" for consecutive math functions; a get/put model would work quite well for such math
library (since it's based on BLAS like almost all such math libraries, I expect this to be the common
pattern)

Hi Thomas,

On Mon, May 17, 2021 at 5:45 AM Thomas Gleixner [off-list ref] wrote:

AMX (or whatever comes next) is nothing else than a device and it
just should be treated as such. The fact that it is not exposed
via a driver and a device node does not matter at all.

TMM registers are part of the CPU architectural state.
If TMM registers exist for one logical CPU, they exist for all CPUs --
including HT siblings.
(Intel supports only homogeneous ISA)

Ditto for the instructions that access and operate on TMM registers.

One can reasonably predict, that like Intel has done for all other registers,
there will be future instructions added to the ISA to operate on TMM registers,
including in combination with non-TMM registers that are also part
of the architectural state.

It is an unfortunate word choice that some documentation calls the
TMUL instruction
an "accelerator".  It isn't.  It is part of the ISA, like any other instruction.

I agree that a device interface may make sense for real accelerators
that don't run x86 instructions, I don't see long term viability for attempting
to carve a sub-set of x86 instructions into a device, particularly when
the set of instructions will continue to evolve.

Not doing so requires this awkward buffer allocation issue via #NM with
all it's downsides; it's just wrong to force the kernel to manage
resources of a user space task without being able to return a proper
error code.

The hardware #NM support for fault on first use is a feature to allow the OS
to optimize space so that pages do not have to be dedicated to back registers
unless/until they are actually used.

There is absolutely no requirement that a particular
OS take advantage of that feature.  If you think that this optimization is
awkward, we can easily delete/disable it and simply statically allocate buffers
for all threads at initialization time.  Though you'll have to convince me
why the word "awkward" applies, rather than "elegant".

Regarding error return for allocation failures.

I'm not familiar with the use-case where vmalloc would be likely to fail today,
and I'd be interested if anybody can detail that use-case.
But even if there is none today, I grate that Linux could evolve to make vmalloc
fail in the future, and so an interface to reqeust pre-allocation of buffers
is reasonable insurance.  Chang has implemented this prctl in v5
of the TMUL patch series.

It also prevents fine grained control over access to this
functionality. As AMX is clearly a shared resource which is not per HT
thread (maybe not even per core) and it has impact on power/frequency it
is important to be able to restrict access on a per process/cgroup
scope.

AMX is analogous to the multiplier used by AVX-512.
The architectural state must exist on every CPU, including HT siblings.
Today, the HT siblings share the same execution unit,
and I have no reason to expect that will change.

I thought we already addressed the FUD surrounding power/frequency.
As with every kind of instruction -- those that use
more power will leave less power for their peers, and there is a mechanism
to track that power budget.  I acknowledge that the mechanism was overly
conservative and slow to recover in initial AVX-512 systems, and that issue
persists even with the latest publically available hardware today.
I acknowledge that you do not trust that Intel has addressed this
(for both AVX-512 and AMX) in the first hardware that supports AMX.

Having a proper interface (syscall, prctl) which user space can use to
ask for permission and allocation of the necessary buffer(s) is clearly
avoiding the downsides and provides the necessary mechanisms for proper
control and failure handling.

It's not the end of the world if something which wants to utilize this
has do issue a syscall during detection. It does not matter whether
that's a library or just the application code itself.

That's a one off operation and every involved entity can cache the
result in TLS.

AVX512 has already proven that XSTATE management is fragile and error
prone, so we really have to stop this instead of creating yet another
half baked solution.

We fixed the glibc ABI issue.  It is available now and production
release is this summer.
Yes, it should have been addressed when AVX-512 was deployed.

thanks
Len Brown, Intel Open Source Technology Center

Len,

On Thu, May 20 2021 at 11:35, Len Brown wrote:

On Mon, May 17, 2021 at 5:45 AM Thomas Gleixner [off-list ref] wrote:

quoted

AMX (or whatever comes next) is nothing else than a device and it
just should be treated as such. The fact that it is not exposed
via a driver and a device node does not matter at all.

TMM registers are part of the CPU architectural state.
If TMM registers exist for one logical CPU, they exist for all CPUs --
including HT siblings. (Intel supports only homogeneous ISA)

Ditto for the instructions that access and operate on TMM registers.

One can reasonably predict, that like Intel has done for all other registers,
there will be future instructions added to the ISA to operate on TMM registers,
including in combination with non-TMM registers that are also part
of the architectural state.

It is an unfortunate word choice that some documentation calls the
TMUL instruction an "accelerator".  It isn't. It is part of the ISA,
like any other instruction.

of course I know that it is an instruction and the register state is
part of the per CPU architectural state.

Though there is a fundamental difference between per logical CPU
architectural state and per logical CPU resources and you know that as
well as I do.

IOW, that does not change the fact that AMX is a shared resource. That's
true for AVX and that's also true for the architectural RNG, which is
also accessed like "any other instruction". We've seen how well that
works.

That's the whole point. Because it's a shared resource with causes
contention and also has side effects vs. power/thermal and state size
this _is_ different from 'any other instruction'.

I agree that a device interface may make sense for real accelerators
that don't run x86 instructions, I don't see long term viability for attempting
to carve a sub-set of x86 instructions into a device, particularly when
the set of instructions will continue to evolve.

Nobody asked for a device interface for AMX. All I asked for is a
_mandatory_ "request usage" interface, e.g. prctl.

Just for the record:

 Your like "any other instruction" argument is a nothing else than a
 strawman.

 There exist instructions today which need OS assistance, e.g. the SGX
 related instructions, the upcoming TDX related ones, ENQCMD & al.

Please tell me _why_ they are so different. They are part of the ISA and
still are subject to fine grained (OS) control.

quoted

Not doing so requires this awkward buffer allocation issue via #NM with
all it's downsides; it's just wrong to force the kernel to manage
resources of a user space task without being able to return a proper
error code.

The hardware #NM support for fault on first use is a feature to allow the OS
to optimize space so that pages do not have to be dedicated to back registers
unless/until they are actually used.

There is absolutely no requirement that a particular
OS take advantage of that feature.  If you think that this optimization is
awkward, we can easily delete/disable it and simply statically allocate buffers
for all threads at initialization time.  Though you'll have to convince me
why the word "awkward" applies, rather than "elegant".

It's not elegant. It's a hack to avoid rethinking the approach to this
kind of features.

But I have to admit that it's a cute hack and it even can be utilized
for a access-request based solution.

Regarding error return for allocation failures.

I'm not familiar with the use-case where vmalloc would be likely to fail today,
and I'd be interested if anybody can detail that use-case.

It does not matter whether it's likely or not. Unlikely simply does not
exist at cloud-scale.

But even if there is none today, I grate that Linux could evolve to make vmalloc
fail in the future, and so an interface to reqeust pre-allocation of buffers
is reasonable insurance.  Chang has implemented this prctl in v5
of the TMUL patch series.

No, it's not a reasonable insurance, simply because it's not mandatory.

quoted

It also prevents fine grained control over access to this
functionality. As AMX is clearly a shared resource which is not per HT
thread (maybe not even per core) and it has impact on power/frequency it
is important to be able to restrict access on a per process/cgroup
scope.

AMX is analogous to the multiplier used by AVX-512.
The architectural state must exist on every CPU, including HT siblings.
Today, the HT siblings share the same execution unit,
and I have no reason to expect that will change.

I'm well aware that HT siblings share the same execution unit for
AVX.

Though AMX is if I remember the discussions two years ago correctly
shared by more than the HT siblings which makes things worse.

I thought we already addressed the FUD surrounding power/frequency.

What's FUD here?

  The fact that AMX is a shared resource which has contention issues?

  The fact that AMX usage has an influence on power/frequency?

If that's FUD by now, then your documentation needs an update.

As with every kind of instruction -- those that use
more power will leave less power for their peers, and there is a mechanism
to track that power budget.  I acknowledge that the mechanism was overly
conservative and slow to recover in initial AVX-512 systems, and that issue
persists even with the latest publically available hardware today.
I acknowledge that you do not trust that Intel has addressed this
(for both AVX-512 and AMX) in the first hardware that supports AMX.

It does not matter whether I trust Intel or not to get this right. It
does neither matter whether there is a mechanism to track the budget or
not.

What matters is that the proposed #NM automatism simply prevents fine
grained access control for a _shared_ resource which has implications on
power and frequency and performance in general due to the fact that it's
shared and causes contention.

And because the #NM hack allows the world and its dog to use AMX any
unpriviledged user can utilize it. See the idea to use it for grep...

You might want to talk to the people in your company who care about
real-time and functional safety whether they think it's a good idea to
allow unrestricted access to functionality which has an influence on the
overall system behaviour with no other knob than to turn it off
completely. Turn it off completely is not an option simply because there
are valid use cases even in that area.

quoted

Having a proper interface (syscall, prctl) which user space can use to
ask for permission and allocation of the necessary buffer(s) is clearly
avoiding the downsides and provides the necessary mechanisms for proper
control and failure handling.

It's not the end of the world if something which wants to utilize this
has do issue a syscall during detection. It does not matter whether
that's a library or just the application code itself.

That's a one off operation and every involved entity can cache the
result in TLS.

AVX512 has already proven that XSTATE management is fragile and error
prone, so we really have to stop this instead of creating yet another
half baked solution.

We fixed the glibc ABI issue.  It is available now and production
release is this summer.

That does not answer my questions at all.

Yes, it should have been addressed when AVX-512 was deployed.

Correct. And in hindsight we should have insisted to have fine grained
control over that back then, but that's water under the bridge.

AMX and what's coming next is not.

Thanks,

        tglx

On 5/20/21 1:54 PM, Thomas Gleixner wrote:

quoted

Regarding error return for allocation failures.

I'm not familiar with the use-case where vmalloc would be likely to fail today,
and I'd be interested if anybody can detail that use-case.

It does not matter whether it's likely or not. Unlikely simply does not
exist at cloud-scale.

Len, I may have led you astray in some of our discussions on this topic.
 Here are the cold hard facts:

 * vmalloc() can fail (the memory.kmem cgroup limit is probably the most
   likely place to be exposed to this)
 * vmalloc() failure in a fault (like #NM) will result in SIGSEGV
 * vmalloc() failure in a syscall can be handled with -ENOMEM

In some of our discussions, I told you that reasonably-sized vmalloc()s
don't practically fail and that we shouldn't be concerned with failure
for our vmalloc()-in-#NM use-case.  In other words, I'm OK with crashing
apps at the point that vmalloc() is failing.

However, Thomas was pretty clear that he's not OK with that.  To
paraphrase: if we can avoid expanding the scope of where memory
allocation failures result in SIGSEGV, we should do it.

While I don't *entirely* agree that it's worth it, I can respect
Thomas's opinion here.  It leads me in the direction of wanting to
drive dynamic xstate vmalloc()s from an explicit syscall ABI.

My apologies if I sent the AMX support on an unproductive tangent here.

On Thu, May 20, 2021 at 4:54 PM Thomas Gleixner [off-list ref] wrote:

Thomas,

quoted

AMX is analogous to the multiplier used by AVX-512.
The architectural state must exist on every CPU, including HT siblings.
Today, the HT siblings share the same execution unit,
and I have no reason to expect that will change.

I'm well aware that HT siblings share the same execution unit for
AVX.

Though AMX is if I remember the discussions two years ago correctly
shared by more than the HT siblings which makes things worse.

I regret that we were unable to get together in the last year to have
an updated discussion.  I think if we had, then we would have saved
a lot of mis-understanding and a lot of email!

So let me emphasize here:

There is one TMUL execution unit per core.
It is shared by the HT siblings within that core.

So the comparison to the AVX-512 multiplier is a good one.

Len Brown, Intel Open Source Technology Center

Len,

On Thu, May 20 2021 at 17:22, Len Brown wrote:

On Thu, May 20, 2021 at 4:54 PM Thomas Gleixner [off-list ref] wrote:

quoted

quoted

AMX is analogous to the multiplier used by AVX-512.
The architectural state must exist on every CPU, including HT siblings.
Today, the HT siblings share the same execution unit,
and I have no reason to expect that will change.

I'm well aware that HT siblings share the same execution unit for
AVX.

Though AMX is if I remember the discussions two years ago correctly
shared by more than the HT siblings which makes things worse.

I regret that we were unable to get together in the last year to have
an updated discussion.  I think if we had, then we would have saved
a lot of mis-understanding and a lot of email!

So let me emphasize here:

There is one TMUL execution unit per core.
It is shared by the HT siblings within that core.

So the comparison to the AVX-512 multiplier is a good one.

Fine, but that does not at all change the facts that:

  1) It's shared between logical CPUs

  2) It has effects on power/thermal and therefore effects which reach
     outside of the core scope

  3) Your appproach of making it unconditionlly available via the
     proposed #NM prevents the OS and subsequently the system admin /
     system designer to implement fine grained control over that
     resource.

     And no, an opt-in approach by providing a non-mandatory
     preallocation prctl does not solve that problem.

Thanks,

        tglx

On Thu, May 20, 2021 at 5:13 PM Dave Hansen [off-list ref] wrote:

quoted

quoted

Regarding error return for allocation failures.

...

 * vmalloc() can fail (the memory.kmem cgroup limit is probably the most
   likely place to be exposed to this)
 * vmalloc() failure in a fault (like #NM) will result in SIGSEGV
 * vmalloc() failure in a syscall can be handled with -ENOMEM

Thanks for clarifying this, Dave.

We added the explicit-allocate to v5,
which should be on the list by tomorrow.

So the questions are:
1. who calls it -- a call/thread or process?  the application?  a
library -- which library?
2. is it optional, or mandatory?
3. if it is mandatory, what is the best way to enforce it?
4. should we have a "release" system call too?

1. Every thread needs a context switch buffer.  Does every thread make
the system call?  It seems sort of awkward for a library to always
make a system call before doing a TMUL.  It would be functionally
harmless, but it would add latency to an otherwise low-latency
operation.  If some central library does it, and caches that it has
done it before, then it would be ugly, but at least it would remove an
unnecessary user/kernel transition.

2. If it is optional, then v5 is code complete -- because it allows
you to allocate either explicitly via prtcl, or transparently via #NM.

3. If it is mandatory, then we should re-purpose the XFD mechanism:
app starts with XFD armed, by default
if app touches AMX before prctl, it takes a signal (and dies).
When app calls prctl, allocate buffer disarm XFD for that app (exactly
what #NM trap does today).

4. I don't see a justification for a release concept, but it is
possible -- though sort of sticky with possible nested calls from
combinations of apps and libraries.  If that were sorted out by a
central library, then the actual system call on the last release per
thread would re-arm XFD to prevent access until the next explicit
request.  Unclear if it is important that the kernel actually do the
free -- some things might run faster if we keep it around...

Len Brown, Intel Open Source Technology Center

On Thu, May 20, 2021 at 5:41 PM Thomas Gleixner [off-list ref] wrote:

Len,

On Thu, May 20 2021 at 17:22, Len Brown wrote:

quoted

On Thu, May 20, 2021 at 4:54 PM Thomas Gleixner [off-list ref] wrote:

quoted

quoted

AMX is analogous to the multiplier used by AVX-512.
The architectural state must exist on every CPU, including HT siblings.
Today, the HT siblings share the same execution unit,
and I have no reason to expect that will change.

I'm well aware that HT siblings share the same execution unit for
AVX.

Though AMX is if I remember the discussions two years ago correctly
shared by more than the HT siblings which makes things worse.

I regret that we were unable to get together in the last year to have
an updated discussion.  I think if we had, then we would have saved
a lot of mis-understanding and a lot of email!

So let me emphasize here:

There is one TMUL execution unit per core.
It is shared by the HT siblings within that core.

So the comparison to the AVX-512 multiplier is a good one.

Fine, but that does not at all change the facts that:

  1) It's shared between logical CPUs

  2) It has effects on power/thermal and therefore effects which reach
     outside of the core scope

FWIW, this is true of *every* instruction in the CPU.
Indeed, even when the CPU is executing *no* instructions at all,
the C-state chosen by that CPU has power/thermal impacts on its peers.

Granted, high performance instructions such as AVX-512 and TMUL
are the most extreme case.

  3) Your approach of making it unconditionally available via the
     proposed #NM prevents the OS and subsequently the system admin /
     system designer to implement fine grained control over that
     resource.

     And no, an opt-in approach by providing a non-mandatory
     preallocation prctl does not solve that problem.

I'm perfectly fine with making the explicit allocation (aka opt-in) mandatory,
and enforcing it.

Len Brown, Intel Open Source Technology Center

On 5/20/21 2:41 PM, Len Brown wrote:

So the questions are:
1. who calls it -- a call/thread or process?  the application?  a
library -- which library?
2. is it optional, or mandatory?
3. if it is mandatory, what is the best way to enforce it?
4. should we have a "release" system call too?

1. Every thread needs a context switch buffer.  Does every thread make
the system call?  It seems sort of awkward for a library to always
make a system call before doing a TMUL.  It would be functionally
harmless, but it would add latency to an otherwise low-latency
operation.  If some central library does it, and caches that it has
done it before, then it would be ugly, but at least it would remove an
unnecessary user/kernel transition.

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

2. If it is optional, then v5 is code complete -- because it allows
you to allocate either explicitly via prtcl, or transparently via #NM.

It needs to be mandatory.  If it's not, then nobody will use it, and
they'll suffer the dreaded SIGSEGV-on-vmalloc()-failure and start filing
bug reports.

3. If it is mandatory, then we should re-purpose the XFD mechanism:
app starts with XFD armed, by default
if app touches AMX before prctl, it takes a signal (and dies).
When app calls prctl, allocate buffer disarm XFD for that app (exactly
what #NM trap does today).

Yes, that sounds like a good use of XFD.

4. I don't see a justification for a release concept, but it is
possible -- though sort of sticky with possible nested calls from
combinations of apps and libraries.  If that were sorted out by a
central library, then the actual system call on the last release per
thread would re-arm XFD to prevent access until the next explicit
request.  Unclear if it is important that the kernel actually do the
free -- some things might run faster if we keep it around...

I think would be more of a get/put model rather than an allocate/free model.

The "put" could effectively be a noop for now.  But, if we don't put
this in the ABI up front, we can't add it later.  That means that we
could never add a lazy-free, even if we wanted to.

Len,

On Thu, May 20 2021 at 17:49, Len Brown wrote:

On Thu, May 20, 2021 at 5:41 PM Thomas Gleixner [off-list ref] wrote:

quoted

  2) It has effects on power/thermal and therefore effects which reach
     outside of the core scope

FWIW, this is true of *every* instruction in the CPU.
Indeed, even when the CPU is executing *no* instructions at all,
the C-state chosen by that CPU has power/thermal impacts on its peers.

Granted, high performance instructions such as AVX-512 and TMUL
are the most extreme case.

Right and we have to draw the line somewhere.

quoted

  3) Your approach of making it unconditionally available via the
     proposed #NM prevents the OS and subsequently the system admin /
     system designer to implement fine grained control over that
     resource.

     And no, an opt-in approach by providing a non-mandatory
     preallocation prctl does not solve that problem.

I'm perfectly fine with making the explicit allocation (aka opt-in) mandatory,
and enforcing it.

Great!

Thanks,

        tglx

Dave, Len,

On Thu, May 20 2021 at 15:53, Dave Hansen wrote:

On 5/20/21 2:41 PM, Len Brown wrote:

quoted

So the questions are:
1. who calls it -- a call/thread or process?  the application?  a
library -- which library?
2. is it optional, or mandatory?
3. if it is mandatory, what is the best way to enforce it?
4. should we have a "release" system call too?

1. Every thread needs a context switch buffer.  Does every thread make
the system call?  It seems sort of awkward for a library to always
make a system call before doing a TMUL.  It would be functionally
harmless, but it would add latency to an otherwise low-latency
operation.  If some central library does it, and caches that it has
done it before, then it would be ugly, but at least it would remove an
unnecessary user/kernel transition.

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Correct.

quoted

2. If it is optional, then v5 is code complete -- because it allows
you to allocate either explicitly via prtcl, or transparently via #NM.

It needs to be mandatory.  If it's not, then nobody will use it, and
they'll suffer the dreaded SIGSEGV-on-vmalloc()-failure and start filing
bug reports.

Yes. Plus mandatory allows to do access control. IOW the prctl() can
return EPERM.

quoted

3. If it is mandatory, then we should re-purpose the XFD mechanism:
app starts with XFD armed, by default
if app touches AMX before prctl, it takes a signal (and dies).

Yes.

quoted

When app calls prctl, allocate buffer disarm XFD for that app (exactly
what #NM trap does today).

Yes, that sounds like a good use of XFD.

Agreed.

quoted

4. I don't see a justification for a release concept, but it is
possible -- though sort of sticky with possible nested calls from
combinations of apps and libraries.  If that were sorted out by a
central library, then the actual system call on the last release per
thread would re-arm XFD to prevent access until the next explicit
request.  Unclear if it is important that the kernel actually do the
free -- some things might run faster if we keep it around...

I think would be more of a get/put model rather than an allocate/free model.

The "put" could effectively be a noop for now.

Yes.

But, if we don't put this in the ABI up front, we can't add it later.
That means that we could never add a lazy-free, even if we wanted to.

As I said somewhere in that thread, something like:

       prctl(PR_QUERY_XSTATE_FEATURES,....
       prctl(PR_ENABLE_XSTATE_FEATURES,....
       prctl(PR_DISABLE_XSTATE_FEATURES,....

To make this work you need refcounting and the last put (DISABLE) drops
the buffer and re-arms XFD. But of course an application/library can do
the put late if it knows that it's going to use it over and over.

Thanks,

        tglx

* Dave Hansen via Libc-alpha:

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Why can't userspace look at XCR0 to make the decision?

And we added an interface for querying x86 CPU features to glibc 2.33
which is completely incompatible with this because it assumes that CPU
features do not change during the lifetime of a process. 8-(

Thanks,
Florian

On Fri, May 21, 2021 at 04:44:58PM +0200, Florian Weimer wrote:

And we added an interface for querying x86 CPU features to glibc 2.33
which is completely incompatible with this because it assumes that CPU
features do not change during the lifetime of a process. 8-(

How many x86 kernel maintainers signed off on that patch?

On 5/21/21 7:44 AM, Florian Weimer wrote:

* Dave Hansen via Libc-alpha:

quoted

Our system calls are *REALLY* fast.  We can even do a vsyscall for this
if we want to get the overhead down near zero.  Userspace can also cache
the "I did the prctl()" state in thread-local storage if it wants to
avoid the syscall.

Why can't userspace look at XCR0 to make the decision?

The thing we're trying to avoid is a #NM exception from XFD (the new
first-use detection feature) that occurs on the first use of AMX.  XCR0
will have XCR0[AMX]=1, even if XFD is "armed" and ready to generate the #NM.