On 11/21/2016 04:26 PM, He Zhe wrote:

On 11/19/2016 12:35 AM, Steven Rostedt wrote:

quoted

On Fri, 18 Nov 2016 14:23:24 +0800
He Zhe [off-list ref] wrote:

quoted

Hi,

I'm using 4.8.6-rt5 and find the following issue on nxp-t4xxx board/qemuppc/qemux86-64.
When executing "echo function_graph > /sys/kernel/debug/tracing/current_tracer" the system hangs.
I traced the kernel to where it dies but don't know how to fix. Please help. Thank you.

(gdb) b ftrace_startup

...

Run till exit from #0 ftrace_startup (ops=0xc0cf1704 <graph_ops>, command=8) at /PATH/TO/kernel/trace/ftrace.c:2653
0xc00eb768 in register_ftrace_graph (retfunc=0xc010186c <trace_graph_return>, entryfunc=0xc01014c4 <trace_graph_entry>)
    at /PATH/TO/kernel-source/kernel/trace/ftrace.c:5908
5908 ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET);
Value returned is $1 = 0
(gdb) l
5903 */
5904 __ftrace_graph_entry = entryfunc;
5905 ftrace_graph_entry = ftrace_graph_entry_test;
5906 update_function_graph_func();
5907
5908 ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET);
5909 out:
5910 mutex_unlock(&ftrace_lock);
5911 return ret;
5912 }
(gdb) n
5910 mutex_unlock(&ftrace_lock);
(gdb) n

hangs...

Can't you get a backtrace from this? It's running under qemu right? So
it should be able to break it and see where it is at.

Note, my PowerPC box has just died, so I'm no longer able to work on
it, and don't have time to set up a qemu ppc image at the moment.

Thanks you.
So far we've found it happens on nxp-t4020(ppc) nxp-ls1021a(arma7) intel-x86 qemuppc qemux86-64.
Here is my backtrace and the config is attached.

(gdb) l
5905            ftrace_graph_entry = ftrace_graph_entry_test;
5906            update_function_graph_func();
5907
5908            ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET);
5909    out:
5910            mutex_unlock(&ftrace_lock);   <--- dies here
5911            return ret;
5912    }
5913
5914    void unregister_ftrace_graph(void)
(gdb) bt
#0  register_ftrace_graph (retfunc=0xc0101548 <trace_graph_return>, entryfunc=0xc01011a0 <trace_graph_entry>)
    at /PATH/kernel/trace/ftrace.c:5910
#1  0xc0100e3c in graph_trace_init (tr=<optimized out>) at /PATH/kernel/trace/trace_functions_graph.c:461
#2  0xc00f46cc in tracer_init (tr=<optimized out>, t=<optimized out>) at /PATH/kernel/trace/trace.c:4588
#3  tracing_set_tracer (tr=0xc0cdd970 <global_trace>, buf=0xca347e18 "function_graph")
    at /PATH/kernel/trace/trace.c:4849
#4  0xc00f4868 in tracing_set_trace_write (filp=<optimized out>, ubuf=<optimized out>, cnt=3392437791, ppos=0xca347f18)
    at /PATH/kernel/trace/trace.c:4887
#5  0xc01b77c0 in __vfs_write (file=<optimized out>, p=<optimized out>, count=<optimized out>, pos=<optimized out>)
    at /PATH/fs/read_write.c:510
#6  0xc01b88ec in vfs_write (file=0xcd9e2000, buf=0x1011ff80 "function_graph\n", count=15, pos=0xca347f18)
    at /PATH/fs/read_write.c:560
#7  0xc01b9968 in SYSC_write (count=<optimized out>, buf=<optimized out>, fd=<optimized out>)
    at /PATH/fs/read_write.c:607
#8  SyS_write (fd=<optimized out>, buf=269614976, count=15) at /PATH/fs/read_write.c:599
#9  0xc00127e4 in syscall_dotrace_cont () at /PATH/arch/powerpc/kernel/entry_32.S:353

And I've tried on "a7fac75 Linux 4.8.1" and the latest linux-4.8.y-rt-rebase branch.They have the same issue.

Zhe

Zhe

quoted

-- Steve