Re: [PATCH v4 1/1] powerpc: enable dynamic preemption

Hi Shrikanth,

Thanks for the quick response and for looping in Paul.

This is true only if user selected PREEMPT_DYNAMIC option i think.

Yes, exactly. The issue is that CONFIG_PREEMPT_DYNAMIC=y has been in
the Fedora/RHEL kernel config since kernel 5.16 (2021). It was
silently ignored on ppc64le until your 6.16 patch added
HAVE_PREEMPT_DYNAMIC_KEY, so this is the first time PREEMPT_RCU
actually takes effect for all Fedora/RHEL ppc64le users.

Does your preemption mode remain the same in two cases?

Verified. On 6.15-rc6 (before PREEMPT_DYNAMIC takes effect):

  CONFIG_PREEMPT_VOLUNTARY=y
  (no CONFIG_PREEMPT_DYNAMIC, no /sys/kernel/debug/sched/preempt)

On both 7.1 and 7.2-rc4 (with PREEMPT_DYNAMIC active):

  CONFIG_PREEMPT_LAZY=y
  CONFIG_PREEMPT_DYNAMIC=y
  CONFIG_PREEMPT_RCU=y
  cat /sys/kernel/debug/sched/preempt: "full (lazy)"

So the runtime preemption mode is the same on 7.1 and 7.2. The
difference between 6.15 and 7.1+ is that the preemption model changed
from voluntary (static) to full/lazy (dynamic), which is what pulls
in PREEMPT_RCU.

Weak memory model would need barriers irrespective of
HAVE_PREEMPT_DYNAMIC_CALL or HAVE_PREEMPT_DYNAMIC_KEY.
Static key too is expected to minimal cost.

You're right, I should clarify -- the overhead is not in the static
key mechanism itself. The cost comes from __rcu_read_lock() and
__rcu_read_unlock() which are called when CONFIG_PREEMPT_RCU=y.
These need real memory barriers (lwsync/isync) on ppc64le regardless
of whether the dynamic mechanism uses keys or calls.

The real question is: why does enabling PREEMPT_RCU cost ~33% on
ppc64le but only ~3% on x86_64? The answer is that x86_64's TSO
memory model makes the barriers in __rcu_read_lock/__rcu_read_unlock
essentially free, while ppc64le's weak ordering requires explicit
lwsync/isync instructions, which are expensive when called thousands
of times per second in the SELinux AVC hot path.

I also have new data from SELinux isolation testing that helps
quantify this. On kernel 7.1 (which already has PREEMPT_RCU=y):

SELinux mode    kill bogo-ops/sec    vs enforcing
------------    -----------------    ------------
Enforcing       69,107               baseline
Permissive      70,248               +1.6%
Disabled        93,566               +35.4%

Permissive ~ enforcing confirms the overhead is not in SELinux policy
evaluation. Disabling SELinux removes the rcu_read_lock/unlock call
sites in the AVC path and recovers most of the performance -- but
still leaves a ~13% gap vs 6.12 (no PREEMPT_RCU), which is the base
cost of PREEMPT_RCU in the non-SELinux parts of the kill() path.

So the core issue is: on ppc64le, CONFIG_PREEMPT_RCU makes
rcu_read_lock/unlock significantly more expensive, and the kill()
syscall path hits them very heavily through SELinux AVC lookups.

Thanks,
Jirka

On Mon, Jul 27, 2026 at 6:19 AM Shrikanth Hegde [off-list ref] wrote:

+cc paul for any further/RCU insights.

On 7/27/26 12:03 AM, Jirka Hladky wrote:

quoted

   Hi Shrikanth, Christophe,

Hi Jirka, thanks for the report.

quoted

  I'm seeing a significant performance regression on ppc64le after this
  patch landed in 6.16, caused by CONFIG_PREEMPT_RCU becoming active
  once HAVE_PREEMPT_DYNAMIC_KEY is selected.

This is true only if user selected PREEMPT_DYNAMIC option i think.

config PREEMPT_RCU
         bool
         default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC)
         select TREE_RCU

quoted

  Benchmark: stress-ng kill stressor (tight kill() syscall loop),
  single thread, POWER10 LPAR (8 vCPUs, 1 core SMT-8).

  Bisected across Fedora ELN kernel builds on ppc64le:

    kernel               CONFIG_PREEMPT_RCU   kill bogo-ops/sec
  ---  6.15-rc6 (eln148)    no                   103,207
    6.16     (eln150)          yes                   70,281    (-32%)
    6.18     (eln154)          yes                   72,552    (-30%)

Does your preemption mode remain the same in two cases?

quoted

  For comparison, x86_64 (AMD EPYC 9355P) with the same config change
  shows only a 2.8% regression:

    6.12 x86_64          37,436
    7.2  x86_64           36,392    (-2.8%)

  perf report shows the overhead comes from rcu_read_lock/unlock in the
  SELinux AVC path (check_kill_permission -> security_task_kill ->
  selinux_task_kill -> avc_has_perm -> avc_lookup):

    Function               6.15 (no PREEMPT_RCU)   6.16 (PREEMPT_RCU)
  ---  avc_lookup                          15.23%              24.79%
    __rcu_read_lock                      ~0%                 4.52%
    __rcu_read_unlock                    ~0%                 4.17%
    selinux_task_kill                    6.23%               7.35%
    audit_signal_info*                   0.94%               3.59%

  On x86_64, rcu_read_lock/unlock are cheap thanks to static calls
  (HAVE_PREEMPT_DYNAMIC_CALL). On ppc64le with the KEY-based
  implementation, the weak memory model requires real barriers
  (lwsync/isync) making each RCU read-side critical section
  significantly more expensive.

Weak memory model would need barriers irrespective of HAVE_PREEMPT_DYNAMIC_CALL
or HAVE_PREEMPT_DYNAMIC_KEY. That's my assumption. I will look
more into it. Also i don't know much about PREEMPT_RCU. So might take a while.

quoted

  This aligns with Christophe's earlier review comment that
  HAVE_PREEMPT_DYNAMIC_CALL should be more performant. Would
  implementing static calls for ppc64 be feasible to close this gap?

Static key too is expected to minimal cost. There maybe more into this.

quoted

  Test details:
  - Machine: IBM POWER10 (pvr 0080 0200), pHyp virtualization
  - stress-ng 0.21.03, gcc 14.3.1, glibc 2.39
  - Tuned profile: virtual-guest
  - SELinux: enforcing (permissive recovers only ~7%)

  Happy to run additional tests if needed.

-- 
-Jirka