Re: [RFC] tracing: Try user copies with page faults disabled first

On Wed, 15 Jul 2026 08:54:54 -0700
Usama Arif [off-list ref] wrote:

trace_user_fault_read() is called with preemption disabled to copy user
memory into a per-cpu scratch buffer. The existing implementation enables
preemption around the copy because faulting user memory can sleep. That
opens a window where another task can run on the same CPU and clobber the
per-cpu buffer, so the copy is wrapped in a retry loop: sample
nr_context_switches_cpu(), do the preempt-enabled copy, and retry if the
counter changed. If this fails to complete 100 times, the function gives up
with a warning.

nr_context_switches_cpu() reads rq->nr_switches. That counter increments
for every context switch on the CPU, not only for switches to tasks that
use this tracing scratch buffer. On a heavily loaded system, unrelated
scheduler activity can move the counter during every preempt-enabled copy
attempt, exhaust the retry guard, and trigger the warning.

This is showing up across the Meta fleet around 100 times a day since the
kernel began upgrading to 7.1, mostly on arm servers:

  Error: Too many tries to read user space
  WARNING: kernel/trace/trace.c:6244 at trace_user_fault_read+0x284/0x2c8, CPU#28: Collection-18/677527
  CPU: 28 UID: 0 PID: 677527 Comm: Collection-18 Kdump: loaded Not tainted 7.1.0-.... #1 PREEMPTLAZY
  Hardware name: Quanta Java Island MP 29F0EMA08CH/Java Island, BIOS F0EJ3A16 03/12/2026
  Call trace:
   trace_user_fault_read+0x284/0x2c8 (P)
   syscall_get_data+0x144/0x2c0
   perf_syscall_enter+0xc0/0x2d8
   syscall_trace_enter+0x1a0/0x270
   do_el0_svc+0x54/0xb8
   el0_svc+0x44/0x268
   el0t_64_sync_handler+0x7c/0x120
   el0t_64_sync+0x17c/0x180
  ---[ end trace 0000000000000000 ]---

The retry loop is only needed when preemption must be enabled for the user
copy. If the user pages are already resident, the copy can complete without
fault handling that sleeps, and preemption can stay disabled throughout.

Add a fast path that first tries the copy with page faults disabled. For
the plain copy_from_user case, use __copy_from_user_inatomic(). If the
probe faults, the architecture exception-table fixup returns a non-zero
not-copied count and trace_user_fault_read() falls back to the existing
preempt-enabled slow path.

Custom copy callbacks need the same behavior. Update the syscall argument
copy callbacks to report a non-zero return only when the pagefault-disabled
probe faults. With page faults enabled, keep their previous behavior:
record the syscall event and omit only the individual user argument that
still cannot be copied.

The slow path remains in place for nonresident pages and permanent copy
failures. nr_context_switches_cpu() still overcounts, but the retry loop is
now avoided for the common resident-page case that does not need fault
handling.

This was reported also under memory load.

I have a patch that will only do a retry if another *user* task schedules
in, and will not be bothered by kernel tasks (which may be scheduled in due
to the copy from user to begin with).

Can you see if that works too?

  https://lore.kernel.org/all/20260710083357.49e05ff6@gandalf.local.home/ (local)

I'm also thinking it may be backwards to try with page_fault disabled
first. Because if the text is not present in memory (which is the case for
a lot of syscalls), it will fail there and then go into the "slow path" to
try with enabling preemption. If you bug hits then, it could cause the
warning to show up.

If you first try with preemption disabled, it will likely pull the page
into memory, so if it does schedule, doing it with pagefault_disabled() a
second time is more likely to succeed (the page has been pulled in via the
first attempt). That is, the first attempt pulls in the page but fails due
to being scheduled out, the second attempt with pagefault_disabled() may be
more likely to succeed. Thus, on option is to simply alternate between
re-enabling preemption, and calling with page_fault() disabled.

-- Steve

Reported-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
This warning is very likely occuring when the fleetwide profiler runs
and something else seems to load the server. (As we have perf_syscall_enter()
in the stack and I see the profiler process active when the warning prints).
It can occur several days after boot, so its a bit difficult to verify
if the warning will go away with this patch deployed.
If the patch looks good, we can deploy it in the fleet and report back.
---