Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()

On 10/9/2025 3:42 PM, Breno Leitao wrote:

Hello,

I am seeing a crash in some production host in function
mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
that doesn't seem to be addressable. I am seeing this on 6.13, but,
looking at the upstream code, the function is the same.

Unfortunately I am not able to reproduce this on upstream kernel easily.
Host is running ConnectX-7.

Here is the quick stack of the problem:

	Unable to handle kernel paging request at virtual address 00000000213afe58

	#0  string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
	#1  string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
	#2  vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
	#3  snprintf (lib/vsprintf.c:2983:6)

Looking further, I found this code:

         snprintf(tmp, sizeof(tmp), str_frmt->string,
                  str_frmt->params[0],
                  str_frmt->params[1],
                  str_frmt->params[2],
                  str_frmt->params[3],
                  str_frmt->params[4],
                  str_frmt->params[5],
                  str_frmt->params[6]);


and the str_frmt has the following content:

	*(struct tracer_string_format *)0xffff100026547260 = {
	.string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
	.params = (int [7]){ 557514328, 3 },
	.num_of_params = (int)2,
	.last_param_num = (int)2,
	.event_id = (u8)3,
	.tmsn = (u32)5201,
	.hlist = (struct hlist_node){
		.next = (struct hlist_node *)0xffff0009f63ce078,
		.pprev = (struct hlist_node **)0xffff0004123ec8d8,
	},
	.list = (struct list_head){
		.next = (struct list_head *)0xdead000000000100,
		.prev = (struct list_head *)0xdead000000000122,
	},
	.timestamp = (u32)22,
	.lost = (bool)0,
	}


My understanding that we are printf %s with params[0], which is 557514328 (aka
0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
is invalid, and crash.

Is this a known issue?

Not a known issue, not expected, thanks for reporting.
We will send patch to protect from such crash.
Please send FW version it was detected on.

Thanks,
Moshe.

Thanks
--breno