On Wed, 8 Jul 2026 21:37:53 +0900
Jeongho Choi [off-list ref] wrote:

Hello,

We are seeing a reproducible kernel panic related to the tracing code
when it fails to read user-space memory.

The issue was originally reported through the Android/Google Issue
Tracker, and we were advised to report it to the upstream trace mailing
list because the affected code is upstream.

Environment:

Architecture: arm64
Kernel: Linux 6.18.21
Base: Android Common Kernel (android17-6.18)
Affected area: kernel/trace/

The relevant error/panic log is:
[48916.569148] [9:           lmkd:  536] Error: Too many tries to read
user space
[48916.569156] [9:           lmkd:  536] WARNING: CPU: 9 PID: 536 at
kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360
[48916.569443] [9:           lmkd:  536] CPU: 9 UID: 1069 PID: 536 Comm:
lmkd Tainted: G           OE       6.18.21-android17-5-ga1a8e8cab9ec-4k
#1 PREEMPT  25372cd4750dcac3c0fe86b57d47c665f97a6046

[48916.569450] [9:           lmkd:  536] pstate: 63402005 (nZCv daif
+PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[48916.569452] [9:           lmkd:  536] pc :
trace_user_fault_read+0x334/0x360
[48916.569454] [9:           lmkd:  536] lr :
trace_user_fault_read+0x330/0x360
[48916.569456] [9:           lmkd:  536] sp : ffffffc096993cc0
[48916.569457] [9:           lmkd:  536] x29: ffffffc096993cd0 x28:
0000000000000065 x27: ffffff8817e45200
[48916.569461] [9:           lmkd:  536] x26: 0000000000000000 x25:
000000011616c082 x24: 0000000000000000
[48916.569463] [9:           lmkd:  536] x23: 0000000000000009 x22:
000000000000002b x21: ffffff880597f740
[48916.569466] [9:           lmkd:  536] x20: 0000007fc42dd940 x19:
ffffff8817e45770 x18: ffffffd5947ce240
[48916.569468] [9:           lmkd:  536] x17: 2073656972742079 x16:
6e616d206f6f5420 x15: 3a726f727245205d
[48916.569471] [9:           lmkd:  536] x14: 36333520203a646b x13:
6563617073207265 x12: 0000000000000001
[48916.569473] [9:           lmkd:  536] x11: 6f74207365697274 x10:
0000000000000001 x9 : 4bd9ad4516d75100
[48916.569476] [9:           lmkd:  536] x8 : 4bd9ad4516d75100 x7 :
205d383431393635 x6 : 2e36313938345b0a
[48916.569479] [9:           lmkd:  536] x5 : ffffffc080fa5998 x4 :
ffffffd591707202 x3 : 0001360a00000000
[48916.569481] [9:           lmkd:  536] x2 : ffffffc096993af4 x1 :
00000000000000c0 x0 : 0000000000000028
[48916.569484] [9:           lmkd:  536] Call trace:
[48916.569486] [9:           lmkd:  536]
trace_user_fault_read+0x334/0x360 (P)
[48916.569488] [9:           lmkd:  536]  tracing_mark_write+0x84/0x174
[48916.569491] [9:           lmkd:  536]  __arm64_sys_write+0x2a0/0x5c0
[48916.569494] [9:           lmkd:  536]  invoke_syscall+0x58/0xe4
[48916.569498] [9:           lmkd:  536]  do_el0_svc+0x48/0xdc
[48916.569500] [9:           lmkd:  536]  el0_svc+0x3c/0x98
[48916.569503] [9:           lmkd:  536]
el0t_64_sync_handler+0x20/0x130
[48916.569505] [9:           lmkd:  536]  el0t_64_sync+0x1c4/0x1c8
[48916.569508] [9:           lmkd:  536] Kernel panic - not syncing:
kernel: panic_on_warn set ...


The code at the WARN location mentioned in the log above is as follows.

7374                 if (WARN_ONCE(trys++ > 100, "Error: Too many
  tries to read user space"))
7375                         return NULL;


Our current analysis is as follows:

In the Gmail process, during a low memory situation, LMKD writes strings
to /sys/kernel/tracing/trace_marker for systrace recording. At the same
time, it broadcasts a sigkill due to low memory, which is causing the
LMKD trace marker operation to stall.

Hm, in my view, this warning indicates that the circuit breaker has
triggered correctly, so that is not a bug. Under the heavy memory
pressure and low-memory situation, the page can be reclaimed soon
after it is copied.

However, this seems a bit strange that we only checks the CPU-wide context
switching in the loop. Instead, can we introduce a per-cpu sequence counter
to per-cpu buffer, and check it? 

Could you try this ?

From f76d8e4400a5961725d17899f4290c9334987e2b Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <mhiramat@kernel.org>
Date: Fri, 10 Jul 2026 12:11:00 +0900
Subject: [PATCH] tracing: Use per-CPU sequence counter in
 trace_user_fault_read

trace_user_fault_read() copies trace data from user space to the
per-CPU trace buffer. When preemption is enabled during the copy, it
checks if any context switches occurred on the current CPU via
nr_context_switches_cpu() to detect whether the buffer may have been
corrupted by another trace writer.

However, under heavy memory pressure, copying from user space can trigger
page faults (e.g., for swapped-out BSS or anonymous pages) that block and
cause a context switch. Because nr_context_switches_cpu() detects any
context switch (even unrelated ones), it mistakenly assumes the buffer was
corrupted. This leads to repeated retries (up to the 100-try limit), which
causes a WARN_ONCE backtrace and returns -EFAULT to user space, even if
no other task ever accessed the trace buffer.

To mitigate this issue, replace the CPU-wide context switch check with
a dedicated per-CPU sequence counter in struct trace_user_buf.

Since only other tasks invoking trace_user_fault_read() on the same CPU
will increment this counter, unrelated context switches (including those
from page fault sleep) will no longer trigger retries.

Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
 kernel/trace/trace.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 1bc27c0ad029..46cec63f5798 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c

@@ -5989,6 +5989,7 @@ static ssize_t write_marker_to_buffer(struct trace_array *tr, const char *buf,
 
 struct trace_user_buf {
 	char		*buf;
+	unsigned int	sequence;
 };
 
 static DEFINE_MUTEX(trace_user_buffer_mutex);

@@ -6031,7 +6032,10 @@ static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t si
 
 	/* Clear each buffer in case of error */
 	for_each_possible_cpu(cpu) {
-		per_cpu_ptr(tinfo->tbuf, cpu)->buf = NULL;
+		struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+
+		tbuf->buf = NULL;
+		tbuf->sequence = 0;
 	}
 
 	for_each_possible_cpu(cpu) {

@@ -6196,8 +6200,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			     trace_user_buf_copy copy_func, void *data)
 {
 	int cpu = smp_processor_id();
-	char *buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
-	unsigned int cnt;
+	struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+	char *buffer = tbuf->buf;
+	unsigned int seq;
 	int trys = 0;
 	int ret;
 

@@ -6211,10 +6216,10 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		return NULL;
 
 	/*
-	 * This acts similar to a seqcount. The per CPU context switches are
+	 * This acts similar to a seqcount. The per CPU sequence counters are
 	 * recorded, migration is disabled and preemption is enabled. The
 	 * read of the user space memory is copied into the per CPU buffer.
-	 * Preemption is disabled again, and if the per CPU context switches count
+	 * Preemption is disabled again, and if the per CPU sequence count
 	 * is still the same, it means the buffer has not been corrupted.
 	 * If the count is different, it is assumed the buffer is corrupted
 	 * and reading must be tried again.

@@ -6235,7 +6240,8 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			preempt_enable_notrace();
 			preempt_disable_notrace();
 			cpu = smp_processor_id();
-			buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf;
+			tbuf = per_cpu_ptr(tinfo->tbuf, cpu);
+			buffer = tbuf->buf;
 		}
 
 		/*

@@ -6250,8 +6256,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 		if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space"))
 			return NULL;
 
-		/* Read the current CPU context switch counter */
-		cnt = nr_context_switches_cpu(cpu);
+		/* Increment the per-CPU buffer sequence counter */
+		tbuf->sequence++;
+		seq = tbuf->sequence;
 
 		/*
 		 * Preemption is going to be enabled, but this task must

@@ -6282,12 +6289,12 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo,
 			return NULL;
 
 		/*
-		 * Preemption is disabled again, now check the per CPU context
-		 * switch counter. If it doesn't match, then another user space
+		 * Preemption is disabled again, now check the per CPU sequence
+		 * counter. If it doesn't match, then another user space
 		 * process may have schedule in and corrupted our buffer. In that
 		 * case the copying must be retried.
 		 */
-	} while (nr_context_switches_cpu(cpu) != cnt);
+	} while (tbuf->sequence != seq);
 
 	return buffer;
 }

-- 
2.43.0



-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>