Re: [PATCH v4 1/5] getcpu_cache system call: cache CPU number of running thread

On Wed, Feb 24 2016, Mathieu Desnoyers [off-list ref] wrote:

       Typically, a library or application will keep the cpu  number
       cache  in  a  thread-local  storage variable, or other memory
       areas belonging to each thread. It is recommended to  perform
       a  volatile  read of the cpu number cache to prevent the com‐
       piler from doing load tearing. An alternative approach is  to
       read  the  cpu  number cache from inline assembly in a single
       instruction.

       Each thread is responsible for registering its own cpu number
       cache.   Only  one  cpu  cache  address can be registered per
       thread.

       The symbol  __getcpu_cache_tls  is  recommended  to  be  used
       across  libraries  and  applications  wishing  to  register a
       thread-local getcpu_cache. The  attribute  "weak"  is  recom‐
       mended  when  declaring this variable in libraries.  Applica‐
       tions can choose to define their own version of  this  symbol
       without the weak attribute as a performance improvement.

       In  a  typical usage scenario, the thread registering the cpu
       number cache will be performing reads from that cache. It  is
       however  also allowed to read the cpu number cache from other
       threads. The cpu number cache updates performed by the kernel
       provide single-copy atomicity semantics, which guarantee that
       other threads performing single-copy atomic reads of the  cpu
       number cache will always observe a consistent value.

       Memory registered as cpu number cache should never be deallo‐
       cated before the thread which registered it  exits:  specifi‐
       cally, it should not be freed, and the library containing the
       registered thread-local storage should not be dlclose'd.

Maybe spell out the consequence if this is violated - since the SIGSEGV
only happens on migration, it may take a while to strike.

Random thoughts: The current implementation ensures that getcpu_cache is
"idempotent" from within a single thread - once set, it can never get
unset nor set to some other pointer. I think that can be useful, since
it means a library can reliably use the TLS variable itself (initialized
with some negative number) as an indicator of whether
getcpu_cache(GETCPU_CACHE_SET) has been called. So if a single test on a
fast path where the library would need to load __getcpu_cache_tls anyway
is acceptable, it can avoid requiring some library init function to be
called in each thread - which can sometimes be hard to arrange. Is this
something we want to guarantee - that is, will we never implement
GETCPU_CACHE_UNSET or a "force" flag to _SET? Either way, I think we
should spend a few words on it to avoid the current behaviour becoming
accidental ABI.

In another thread:

However, there are other use-cases for having a fast mechanism for
reading the current CPU number, besides restartable sequences.  For
instance, it can be used by glibc to implement a faster sched_getcpu.

Will glibc do that? It may be a little contentious for glibc to claim a
unique resource such as task_struct::cpu_cache for itself, even if
everybody is supposed to use the same symbol. Hm, maybe one could say
that if an application does define the symbol __getcpu_cache_tls (which
is techically in the implementation namespace), that gives glibc (and
any other library) license to do getcpu_cache(SET, &&__getcpu_cache_tls)
(pseudo-code, of course). If a library initializes its own weak version
with -2 it can check whether the application defined
__getcpu_cache_tls. Ok, I'm probably overthinking this...

Rasmus