Re: mlx5: CX7: fw_tracer: crash at mlx5_tracer_print_trace()

On 15/10/2025 12:13, Moshe Shemesh wrote:

On 10/9/2025 3:42 PM, Breno Leitao wrote:

quoted

Hello,

I am seeing a crash in some production host in function
mlx5_tracer_print_trace() that sprintf a string (%s) pointing to value
that doesn't seem to be addressable. I am seeing this on 6.13, but,
looking at the upstream code, the function is the same.

Unfortunately I am not able to reproduce this on upstream kernel easily.
Host is running ConnectX-7.

Here is the quick stack of the problem:

    Unable to handle kernel paging request at virtual address 00000000213afe58

    #0  string_nocheck(buf=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe59, len=0) (lib/vsprintf.c:646:12)
    #1  string(end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], s=0x213afe58) (lib/vsprintf.c:728:9)
    #2  vsnprintf(buf=0xffff8002a11af8e0[vmap stack: 1315725 (kworker/u576:1) +0xf8e0], fmt=0xffff10006cd4950a, end=0xffff8002a11afae0[vmap stack: 1315725 (kworker/u576:1) +0xfae0], str=0xffff8002a11af909[vmap stack: 1315725 (kworker/u576:1) +0xf909], old_fmt=0xffff10006cd49508) (lib/vsprintf.c:2848:10)
    #3  snprintf (lib/vsprintf.c:2983:6)

Looking further, I found this code:

         snprintf(tmp, sizeof(tmp), str_frmt->string,
                  str_frmt->params[0],
                  str_frmt->params[1],
                  str_frmt->params[2],
                  str_frmt->params[3],
                  str_frmt->params[4],
                  str_frmt->params[5],
                  str_frmt->params[6]);


and the str_frmt has the following content:

    *(struct tracer_string_format *)0xffff100026547260 = {
    .string = (char *)0xffff10006cd494df = "PCA 9655E init, failed to verify command %s, failed %d",
    .params = (int [7]){ 557514328, 3 },
    .num_of_params = (int)2,
    .last_param_num = (int)2,
    .event_id = (u8)3,
    .tmsn = (u32)5201,
    .hlist = (struct hlist_node){
        .next = (struct hlist_node *)0xffff0009f63ce078,
        .pprev = (struct hlist_node **)0xffff0004123ec8d8,
    },
    .list = (struct list_head){
        .next = (struct list_head *)0xdead000000000100,
        .prev = (struct list_head *)0xdead000000000122,
    },
    .timestamp = (u32)22,
    .lost = (bool)0,
    }


My understanding that we are printf %s with params[0], which is 557514328 (aka
0x213afe58). So, sprintf is trying to access the content of 0x213afe58, which
is invalid, and crash.

Is this a known issue?

Not a known issue, not expected, thanks for reporting.
We will send patch to protect from such crash.
Please send FW version it was detected on.

Hello!

I work with Breno and just following up on his behalf while he is away. Just wanted to check
if there was an update on the patch?

Thanks
Usama

Thanks,
Moshe.

quoted

Thanks
--breno