Re: tprobe event tracing error

Hi,
(Cc: linux-kernel-trace ML for sharing this knowledge)

On Mon, 26 Feb 2024 16:36:29 +0300
Максим Морсков [off-list ref] wrote:

Hello, dear Masami.
I am researching Linux event tracing subsystem in part of tprobes,
and found interesting behavior in kernel version 6.6:
 
echo  't:my_fchmodat sys_enter_fchmodat' | sudo tee ‘/sys/kernel/tracing/dynamic_events’
bash: line 1: echo: write error: Invalid argument

Yeah, I understand that you are confused by this behavior, but it is
actually expected behavior. syscalls:* events looks like trace events
based on tracepoint, but those are software generated trace event.

You can find raw_syscalls:* trace events, that is based on the tracepoint,
and other syscalls:* are based on that raw_syscalls:* trace points.
(IOW, those are a kind of pre-compiled dynamic events)

e.g.

/sys/kernel/tracing # echo "t sys_enter \$arg*" >> dynamic_events 
/sys/kernel/tracing # cat dynamic_events 
t:tracepoints/sys_enter sys_enter regs=regs id=id

/sys/kernel/tracing # echo "t sys_enter_open \$arg*" >> dynamic_events 
sh: write error: Invalid argument
/sys/kernel/tracing # cat error_log 
[  227.981347] trace_fprobe: error: Tracepoint is not found
  Command: t sys_enter_open $arg*
             ^

So, tprobe can not find the hard-coded tracepoints for those dynamically
generated syscall trace events. But raw_syscall sys_enter/sys_exit are OK.

Thank you,

 
sudo cat /sys/kernel/tracing/error_log
trace_fprobe: error: Tracepoint is not found 
 Command: t:my_fchmodat sys_enter_fchmodat
                                                    ^
 
But such tracepoint ( sys_enter_fchmodat ) exists in system:
sudo perf list | grep chmod   
 syscalls:sys_enter_ chmod                           [Tracepoint event] 
 syscalls:sys_enter_f chmod                          [Tracepoint event] 
 syscalls:sys_enter_f chmod at                        [Tracepoint event] 
 syscalls:sys_enter_f chmod at2                       [Tracepoint event] 
 syscalls:sys_exit_ chmod                            [Tracepoint event] 
 syscalls:sys_exit_f chmod                           [Tracepoint event] 
 syscalls:sys_exit_f chmod at                         [Tracepoint event] 
 syscalls:sys_exit_f chmod at2                        [Tracepoint event]
 
sudo ls -lha /sys/kernel/tracing/events/syscalls/ | grep chmod
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_ chmod
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod at 
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_enter_f chmod at2 
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_ chmod
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod at 
drwxr-xr-x 1 root root 0 фев 26 16:07 sys_exit_f chmod at2
 
My kernel config in the attachment.
I noticed that you are developing event tracing tprobe subsystem.
May be you could explain such behavior?
Thank you in advance!
 
--
Best regards
Maksim Morskov

-- 
Masami Hiramatsu (Google) [off-list ref]