On Wed, Feb 24 2016, Mathieu Desnoyers [off-list ref] wrote:
Typically, a library or application will keep the cpu number
cache in a thread-local storage variable, or other memory
areas belonging to each thread. It is recommended to perform
a volatile read of the cpu number cache to prevent the com‐
piler from doing load tearing. An alternative approach is to
read the cpu number cache from inline assembly in a single
instruction.
Each thread is responsible for registering its own cpu number
cache. Only one cpu cache address can be registered per
thread.
The symbol __getcpu_cache_tls is recommended to be used
across libraries and applications wishing to register a
thread-local getcpu_cache. The attribute "weak" is recom‐
mended when declaring this variable in libraries. Applica‐
tions can choose to define their own version of this symbol
without the weak attribute as a performance improvement.
In a typical usage scenario, the thread registering the cpu
number cache will be performing reads from that cache. It is
however also allowed to read the cpu number cache from other
threads. The cpu number cache updates performed by the kernel
provide single-copy atomicity semantics, which guarantee that
other threads performing single-copy atomic reads of the cpu
number cache will always observe a consistent value.
Memory registered as cpu number cache should never be deallo‐
cated before the thread which registered it exits: specifi‐
cally, it should not be freed, and the library containing the
registered thread-local storage should not be dlclose'd.
Maybe spell out the consequence if this is violated - since the SIGSEGV
only happens on migration, it may take a while to strike.
Random thoughts: The current implementation ensures that getcpu_cache is
"idempotent" from within a single thread - once set, it can never get
unset nor set to some other pointer. I think that can be useful, since
it means a library can reliably use the TLS variable itself (initialized
with some negative number) as an indicator of whether
getcpu_cache(GETCPU_CACHE_SET) has been called. So if a single test on a
fast path where the library would need to load __getcpu_cache_tls anyway
is acceptable, it can avoid requiring some library init function to be
called in each thread - which can sometimes be hard to arrange. Is this
something we want to guarantee - that is, will we never implement
GETCPU_CACHE_UNSET or a "force" flag to _SET? Either way, I think we
should spend a few words on it to avoid the current behaviour becoming
accidental ABI.
In another thread:
However, there are other use-cases for having a fast mechanism for
reading the current CPU number, besides restartable sequences. For
instance, it can be used by glibc to implement a faster sched_getcpu.
Will glibc do that? It may be a little contentious for glibc to claim a
unique resource such as task_struct::cpu_cache for itself, even if
everybody is supposed to use the same symbol. Hm, maybe one could say
that if an application does define the symbol __getcpu_cache_tls (which
is techically in the implementation namespace), that gives glibc (and
any other library) license to do getcpu_cache(SET, &&__getcpu_cache_tls)
(pseudo-code, of course). If a library initializes its own weak version
with -2 it can check whether the application defined
__getcpu_cache_tls. Ok, I'm probably overthinking this...
Rasmus