I've been studying Jann Horn's biased locking example:
Re: [PATCH 0/4 POC] Allow executing code and syscalls in another address space
<https://lore.kernel.org/linux-api/CAG48ez02UDn_yeLuLF4c=kX0=h2Qq8Fdb0cer1yN8atbXSNjkQ@mail.gmail.com/>
It uses MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ as part of the biased lock
revocation.
How does the this code know that the process has called
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ? Could it fall back to
MEMBARRIER_CMD_GLOBAL instead? Why is it that MEMBARRIER_CMD_GLOBAL
does not require registration (the broader/more expensive barrier), but
the more restricted versions do?
Or put differently, why wouldn't we request
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ unconditionally at
process start in glibc, once we start biased locking in a few places?
Thanks,
Florian
----- On Dec 13, 2021, at 1:47 PM, Florian Weimer fweimer@redhat.com wrote:
I've been studying Jann Horn's biased locking example:
Re: [PATCH 0/4 POC] Allow executing code and syscalls in another address space
<https://lore.kernel.org/linux-api/CAG48ez02UDn_yeLuLF4c=kX0=h2Qq8Fdb0cer1yN8atbXSNjkQ@mail.gmail.com/>
It uses MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ as part of the biased lock
revocation.
How does the this code know that the process has called
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ?
I won't speak for this code snippet in particular, but in general
issuing MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ from a thread which
belongs to a process which has not performed
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ will result in
membarrier returning -EPERM. If the kernel is built without CONFIG_RSEQ
support, it will return -EINVAL:
membarrier_private_expedited():
} else if (flags == MEMBARRIER_FLAG_RSEQ) {
if (!IS_ENABLED(CONFIG_RSEQ))
return -EINVAL;
if (!(atomic_read(&mm->membarrier_state) &
MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY))
return -EPERM;
If you want to create code which optionally depends on availability
of EXPEDITED_RSEQ membarrier, I suspect you will want to perform
registration from a library constructor, and keep track of registration
success/failure in a static variable within the library.
Could it fall back to
MEMBARRIER_CMD_GLOBAL instead?
No. CMD_GLOBAL does not issue the required rseq fence used by the
algorithm discussed. Also, CMD_GLOBAL has quite a few other shortcomings:
it takes a while to execute, and is incompatible with nohz_full kernels.
Why is it that MEMBARRIER_CMD_GLOBAL
does not require registration (the broader/more expensive barrier), but
the more restricted versions do?
The more restricted versions (which require explicit registration) have a
close integration with the Linux scheduler, and in some cases require
additional code to be executed when scheduling between threads which
belong to different processes, for instance the for "SYNC_CORE" membarrier,
which is useful for JITs:
static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm)
{
if (current->mm != mm)
return;
if (likely(!(atomic_read(&mm->membarrier_state) &
MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE)))
return;
sync_core_before_usermode();
}
Also, for the "global-expedited" commands, these can generate IPIs which will
interrupt the flow of threads running on behalf of a registered process.
Therefore, in order to make sure we do not add delays to real-time sensitive
applications, we made this registration "opt-in".
In order to make sure the programming model is the same for expedited
private/global plain/sync-core/rseq membarrier commands, we require that
each process perform a registration beforehand.
Or put differently, why wouldn't we request
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ unconditionally at
process start in glibc, once we start biased locking in a few places?
The registration of membarrier expedited can be either performed immediately
when the process starts, or later, possibly when there are other threads
running concurrently. Note however that the registration scheme has been
optimized for the scenario where it is called when a single thread is
running in the process (see sync_runqueues_membarrier_state()). Otherwise
we need to use the more heavyweight synchronize_rcu(). So my advice would
be to perform the membarrier expedited registration while the process
is still single-threaded if possible, rather than postpone this and
do it entirely lazily on first use, which may happen while other
threads are already running.
Thanks,
Mathieu
Thanks,
Florian
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
On Mon, Dec 13, 2021 at 7:48 PM Florian Weimer [off-list ref] wrote:
I've been studying Jann Horn's biased locking example:
Re: [PATCH 0/4 POC] Allow executing code and syscalls in another address space
<https://lore.kernel.org/linux-api/CAG48ez02UDn_yeLuLF4c=kX0=h2Qq8Fdb0cer1yN8atbXSNjkQ@mail.gmail.com/>
It uses MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ as part of the biased lock
revocation.
How does the this code know that the process has called
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ? Could it fall back to
MEMBARRIER_CMD_GLOBAL instead?
AFAIK no - MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ specifically
forces targeted processes to go through an RSEQ preemption. That only
happens when this special membarrier command is used and when an
actual task switch happens; other membarrier flavors don't guarantee
that.
Also, MEMBARRIER_CMD_GLOBAL can take really long in terms of wall
clock time - it's basically just synchronize_rcu(), and as the
documentation at
https://www.kernel.org/doc/html/latest/RCU/Design/Requirements/Requirements.html
says:
"The synchronize_rcu() grace-period-wait primitive is optimized for
throughput. It may therefore incur several milliseconds of latency in
addition to the duration of the longest RCU read-side critical
section."
You can see that synchronize_rcu() indeed takes quite long in terms of
wall clock time (but not in terms of CPU time - as the documentation
says, it's optimized for throughput in a parallel context) with a
simple test program:
jannh@laptop:~/test/rcu$ cat rcu_membarrier.c
#define _GNU_SOURCE
#include <stdio.h>
#include <linux/membarrier.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <time.h>
#include <err.h>
int main(void) {
for (int i=0; i<20; i++) {
struct timespec ts1;
if (clock_gettime(CLOCK_MONOTONIC, &ts1))
err(1, "time");
if (syscall(__NR_membarrier, MEMBARRIER_CMD_GLOBAL, 0, 0))
err(1, "membarrier");
struct timespec ts2;
if (clock_gettime(CLOCK_MONOTONIC, &ts2))
err(1, "time");
unsigned long delta_ns = (ts2.tv_nsec - ts1.tv_nsec) +
(1000UL*1000*1000) * (ts2.tv_sec - ts1.tv_sec);
printf("MEMBARRIER_CMD_GLOBAL took %lu nanoseconds\n", delta_ns);
}
}
jannh@laptop:~/test/rcu$ gcc -o rcu_membarrier rcu_membarrier.c -Wall
jannh@laptop:~/test/rcu$ time ./rcu_membarrier
MEMBARRIER_CMD_GLOBAL took 17155142 nanoseconds
MEMBARRIER_CMD_GLOBAL took 19207001 nanoseconds
MEMBARRIER_CMD_GLOBAL took 16087350 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15963711 nanoseconds
MEMBARRIER_CMD_GLOBAL took 16336149 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15931331 nanoseconds
MEMBARRIER_CMD_GLOBAL took 16020315 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15873814 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15945667 nanoseconds
MEMBARRIER_CMD_GLOBAL took 23815452 nanoseconds
MEMBARRIER_CMD_GLOBAL took 23626444 nanoseconds
MEMBARRIER_CMD_GLOBAL took 19911435 nanoseconds
MEMBARRIER_CMD_GLOBAL took 23967343 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15943147 nanoseconds
MEMBARRIER_CMD_GLOBAL took 23914809 nanoseconds
MEMBARRIER_CMD_GLOBAL took 32498986 nanoseconds
MEMBARRIER_CMD_GLOBAL took 19450932 nanoseconds
MEMBARRIER_CMD_GLOBAL took 16281308 nanoseconds
MEMBARRIER_CMD_GLOBAL took 24045168 nanoseconds
MEMBARRIER_CMD_GLOBAL took 15406698 nanoseconds
real 0m0.458s
user 0m0.058s
sys 0m0.031s
jannh@laptop:~/test/rcu$
Every invocation of MEMBARRIER_CMD_GLOBAL on my laptop took >10 ms.
----- On Dec 13, 2021, at 1:47 PM, Florian Weimer fweimer@redhat.com wrote:
I've been studying Jann Horn's biased locking example:
Re: [PATCH 0/4 POC] Allow executing code and syscalls in another address space
<https://lore.kernel.org/linux-api/CAG48ez02UDn_yeLuLF4c=kX0=h2Qq8Fdb0cer1yN8atbXSNjkQ@mail.gmail.com/>
It uses MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ as part of the biased lock
revocation.
By the way, there might be something good to salvage from this prototype I did a
while back:
https://github.com/compudj/rseq-test/blob/adapt-lock/test-rseq-adaptative-lock.c
The neat trick there is to use a combination of the Zero Flag and rbx==0/1 to detect
whether the rseq critical section was aborted before or after successful execution
of the CAS. This allows the rseq c.s. to cover an entire loop, which contains a CAS
instruction, without requiring that the critical section ends with a "commit"
instruction.
Some characteristics of this prototype:
- Don't busy-wait in user-space if the lock owner belongs to the same CPU where the
waiter executes. Immediately use futex.
- Adaptative busy-wait delay (per-lock).
- If busy-spinning is preempted, it jumps to abort immediately when resumed. Therefore,
the loop count for adaptative busy-spinning is very precise.
Of course, much more work would be needed, but I suspect a few ideas there can be
useful.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com