Re: [PATCH v13 00/20] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Date: 2024-09-12 01:09:34
Also in:
bpf, lkml
On Wed, 11 Sep 2024 20:27:27 +0200 Jiri Olsa [off-list ref] wrote:
On Sun, Aug 18, 2024 at 09:47:53PM +0900, Masami Hiramatsu (Google) wrote:quoted
Hi, Here is the 13th version of the series to re-implement the fprobe on function-graph tracer. The previous version is; https://lore.kernel.org/all/172000134410.63468.13742222887213469474.stgit@devnote2/ (local) This version is based on v6.11-rc3. In this version, I added a bugfix as [1/20], which should go to urgent branch, and dropped the performance improvement patch which was introduced in v12 because I found that does not work with new kernel. Overview -------- This series rewrites the fprobe on this function-graph. The purposes of this change are; 1) Remove dependency of the rethook from fprobe so that we can reduce the return hook code and shadow stack. 2) Make 'ftrace_regs' the common trace interface for the function boundary. 1) Currently we have 2(or 3) different function return hook codes, the function-graph tracer and rethook (and legacy kretprobe). But since this is redundant and needs double maintenance cost, I would like to unify those. From the user's viewpoint, function- graph tracer is very useful to grasp the execution path. For this purpose, it is hard to use the rethook in the function-graph tracer, but the opposite is possible. (Strictly speaking, kretprobe can not use it because it requires 'pt_regs' for historical reasons.) 2) Now the fprobe provides the 'pt_regs' for its handler, but that is wrong for the function entry and exit. Moreover, depending on the architecture, there is no way to accurately reproduce 'pt_regs' outside of interrupt or exception handlers. This means fprobe should not use 'pt_regs' because it does not use such exceptions. (Conversely, kprobe should use 'pt_regs' because it is an abstract interface of the software breakpoint exception.) This series changes fprobe to use function-graph tracer for tracing function entry and exit, instead of mixture of ftrace and rethook. Unlike the rethook which is a per-task list of system-wide allocated nodes, the function graph's ret_stack is a per-task shadow stack. Thus it does not need to set 'nr_maxactive' (which is the number of pre-allocated nodes). Also the handlers will get the 'ftrace_regs' instead of 'pt_regs'. Since eBPF mulit_kprobe/multi_kretprobe events still use 'pt_regs' as their register interface, this changes it to convert 'ftrace_regs' to 'pt_regs'. Of course this conversion makes an incomplete 'pt_regs', so users must access only registers for function parameters or return value. Design ------ Instead of using ftrace's function entry hook directly, the new fprobe is built on top of the function-graph's entry and return callbacks with 'ftrace_regs'. Since the fprobe requires access to 'ftrace_regs', the architecture must support CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS and CONFIG_HAVE_FTRACE_GRAPH_FUNC, which enables to call function-graph entry callback with 'ftrace_regs', and also CONFIG_HAVE_FUNCTION_GRAPH_FREGS, which passes the ftrace_regs to return_to_handler. All fprobes share a single function-graph ops (means shares a common ftrace filter) similar to the kprobe-on-ftrace. This needs another layer to find corresponding fprobe in the common function-graph callbacks, but has much better scalability, since the number of registered function-graph ops is limited. In the entry callback, the fprobe runs its entry_handler and saves the address of 'fprobe' on the function-graph's shadow stack as data. The return callback decodes the data to get the 'fprobe' address, and runs the exit_handler. The fprobe introduces two hash-tables, one is for entry callback which searches fprobes related to the given function address passed by entry callback. The other is for a return callback which checks if the given 'fprobe' data structure pointer is still valid. Note that it is possible to unregister fprobe before the return callback runs. Thus the address validation must be done before using it in the return callback. Download -------- This series can be applied against the ftrace/for-next branch in linux-trace tree. This series can also be found below branch. https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/fprobe-on-fgraph Thank you, ---hi, I ran the kprobe_multi bench and I'm seeing lower throughput numbers with new fprobe implementation base (bpf-next/master) root@amd:/home/jolsa/bpf-next/tools/testing/selftests/bpf# ./bench -w2 -d5 -a trig-kprobe-multi Setting up benchmark 'trig-kprobe-multi'... Benchmark 'trig-kprobe-multi' started. Iter 0 ( 70.969us): hits 8.530M/s ( 8.530M/prod), drops 0.000M/s, total operations 8.530M/s Iter 1 (-24.985us): hits 8.541M/s ( 8.541M/prod), drops 0.000M/s, total operations 8.541M/s Iter 2 ( -2.976us): hits 8.537M/s ( 8.537M/prod), drops 0.000M/s, total operations 8.537M/s Iter 3 ( -1.502us): hits 8.540M/s ( 8.540M/prod), drops 0.000M/s, total operations 8.540M/s Iter 4 ( 0.392us): hits 8.540M/s ( 8.540M/prod), drops 0.000M/s, total operations 8.540M/s Iter 5 ( 1.308us): hits 8.545M/s ( 8.545M/prod), drops 0.000M/s, total operations 8.545M/s Iter 6 ( 2.783us): hits 8.530M/s ( 8.530M/prod), drops 0.000M/s, total operations 8.530M/s Iter 7 ( -3.775us): hits 8.535M/s ( 8.535M/prod), drops 0.000M/s, total operations 8.535M/s Summary: hits 8.538 ± 0.005M/s ( 8.538M/prod), drops 0.000 ± 0.000M/s, total operations 8.538 ± 0.005M/s your patchset on top of bpf-next/master root@amd:/home/jolsa/bpf-next/tools/testing/selftests/bpf# ./bench -w2 -d5 -a trig-kprobe-multi Setting up benchmark 'trig-kprobe-multi'... Benchmark 'trig-kprobe-multi' started. Iter 0 ( 55.712us): hits 7.224M/s ( 7.224M/prod), drops 0.000M/s, total operations 7.224M/s Iter 1 ( 15.226us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s Iter 2 ( -1.353us): hits 7.221M/s ( 7.221M/prod), drops 0.000M/s, total operations 7.221M/s Iter 3 (-13.029us): hits 7.223M/s ( 7.223M/prod), drops 0.000M/s, total operations 7.223M/s Iter 4 ( 9.111us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s Iter 5 ( -0.106us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s Iter 6 ( 0.734us): hits 7.221M/s ( 7.221M/prod), drops 0.000M/s, total operations 7.221M/s Iter 7 (-10.233us): hits 7.220M/s ( 7.220M/prod), drops 0.000M/s, total operations 7.220M/s Summary: hits 7.221 ± 0.001M/s ( 7.221M/prod), drops 0.000 ± 0.000M/s, total operations 7.221 ± 0.001M/s I did perf profile (attached), but nothing stands out on the first look,
Hm, we may need to compare both before and after, and need annotate in the code to find the bottleneck. Anyway, performance optimization can be done afterwards, so I'm not so worried it :)
I'll try to check on that later this week
Thank you for your help! Thanks,
jirka
---
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 42K of event 'cycles:k'
# Event count (approx.): 44580427937
#
# Overhead Command Shared Object Symbol
# ........ ............... .................................................... ........................................................
#
17.47% bench [kernel.vmlinux] [k] find_kallsyms_symbol
|
---find_kallsyms_symbol
module_address_lookup
kallsyms_lookup_buildid
kallsyms_lookup
print_rec
t_show
seq_read_iter
seq_read
vfs_read
ksys_read
__x64_sys_read
x64_sys_call
do_syscall_64
entry_SYSCALL_64
read
0
11.71% bench [kernel.vmlinux] [k] kprobe_multi_link_prog_run
|
---kprobe_multi_link_prog_run
|
--11.56%--kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
7.65% bench [kernel.vmlinux] [k] function_graph_enter_regs
|
---function_graph_enter_regs
|
--7.65%--ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
7.55% bench [kernel.vmlinux] [k] kprobe_multi_link_handler
|
---kprobe_multi_link_handler
|
|--4.35%--function_graph_enter_regs
| ftrace_graph_func
| 0xffffffffc29850ab
| bpf_get_numa_node_id
| bpf_prog_d9703036495d54b0_trigger_driver
| __bpf_prog_test_run_raw_tp
| bpf_prog_test_run_raw_tp
| __sys_bpf
| __x64_sys_bpf
| x64_sys_call
| do_syscall_64
| entry_SYSCALL_64
| syscall
| bpf_prog_test_run_opts
| trigger_producer_batch
| 0x7a2a73a94ac3
|
--3.20%--fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
7.50% bench [kernel.vmlinux] [k] fprobe_entry
|
---fprobe_entry
|
--7.50%--function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
7.08% bench bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi [k] bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi
|
---bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi
|
|--4.10%--kprobe_multi_link_handler
| fprobe_entry
| function_graph_enter_regs
| ftrace_graph_func
| 0xffffffffc29850ab
| bpf_get_numa_node_id
| bpf_prog_d9703036495d54b0_trigger_driver
| __bpf_prog_test_run_raw_tp
| bpf_prog_test_run_raw_tp
| __sys_bpf
| __x64_sys_bpf
| x64_sys_call
| do_syscall_64
| entry_SYSCALL_64
| syscall
| bpf_prog_test_run_opts
| trigger_producer_batch
| 0x7a2a73a94ac3
|
--2.97%--kprobe_multi_link_prog_run
kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
6.11% bench [kernel.vmlinux] [k] migrate_enable
|
---migrate_enable
|
|--3.37%--kprobe_multi_link_handler
| fprobe_entry
| function_graph_enter_regs
| ftrace_graph_func
| 0xffffffffc29850ab
| bpf_get_numa_node_id
| bpf_prog_d9703036495d54b0_trigger_driver
| __bpf_prog_test_run_raw_tp
| bpf_prog_test_run_raw_tp
| __sys_bpf
| __x64_sys_bpf
| x64_sys_call
| do_syscall_64
| entry_SYSCALL_64
| syscall
| bpf_prog_test_run_opts
| trigger_producer_batch
| 0x7a2a73a94ac3
|
--2.73%--kprobe_multi_link_prog_run
kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
5.90% bench [kernel.vmlinux] [k] ftrace_graph_func
|
---ftrace_graph_func
|
--5.75%--0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
4.03% bench [kernel.vmlinux] [k] __rcu_read_lock
|
---__rcu_read_lock
|
--4.00%--kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
3.61% bench [kernel.vmlinux] [k] bpf_get_numa_node_id
|
---bpf_get_numa_node_id
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
2.99% bench [nf_conntrack] [k] 0x000000000002c0b3
|
---0xffffffffc29850b3
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
2.69% bench bpf_prog_d9703036495d54b0_trigger_driver [k] bpf_prog_d9703036495d54b0_trigger_driver
|
---bpf_prog_d9703036495d54b0_trigger_driver
|
--2.68%--__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
2.04% bench [kernel.vmlinux] [k] t_start
|
---t_start
seq_read_iter
seq_read
vfs_read
ksys_read
__x64_sys_read
x64_sys_call
do_syscall_64
entry_SYSCALL_64
read
0
1.90% bench [kernel.vmlinux] [k] srso_untrain_ret
|
---srso_untrain_ret
|
|--0.68%--fprobe_entry
| function_graph_enter_regs
| ftrace_graph_func
| 0xffffffffc29850ab
| bpf_get_numa_node_id
| bpf_prog_d9703036495d54b0_trigger_driver
| __bpf_prog_test_run_raw_tp
| bpf_prog_test_run_raw_tp
| __sys_bpf
| __x64_sys_bpf
| x64_sys_call
| do_syscall_64
| entry_SYSCALL_64
| syscall
| bpf_prog_test_run_opts
| trigger_producer_batch
| 0x7a2a73a94ac3
|
--0.54%--seq_read_iter
seq_read
vfs_read
ksys_read
__x64_sys_read
x64_sys_call
do_syscall_64
entry_SYSCALL_64
read
0
1.14% bench [kernel.vmlinux] [k] __rcu_read_unlock
|
---__rcu_read_unlock
|
--1.12%--kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
1.02% bench [kernel.vmlinux] [k] migrate_disable
|
---migrate_disable
kprobe_multi_link_handler
fprobe_entry
function_graph_enter_regs
ftrace_graph_func
0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
0.99% bench [kernel.vmlinux] [k] t_func_next.isra.0
|
---t_func_next.isra.0
|
--0.83%--t_start
seq_read_iter
seq_read
vfs_read
ksys_read
__x64_sys_read
x64_sys_call
do_syscall_64
entry_SYSCALL_64
read
0
0.78% bench [nf_conntrack] [k] 0x000000000002c0d4
|
---0xffffffffc29850d4
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
0.75% bench [nf_conntrack] [k] 0x000000000002c0ab
|
---0xffffffffc29850ab
bpf_get_numa_node_id
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
0.68% bench [nf_conntrack] [k] 0x000000000002c00d
|
---0xffffffffc298500d
bpf_prog_d9703036495d54b0_trigger_driver
__bpf_prog_test_run_raw_tp
bpf_prog_test_run_raw_tp
__sys_bpf
__x64_sys_bpf
x64_sys_call
do_syscall_64
entry_SYSCALL_64
syscall
bpf_prog_test_run_opts
trigger_producer_batch
0x7a2a73a94ac3
0.52% bench [kernel.vmlinux] [k] __x86_indirect_thunk_array
|
---__x86_indirect_thunk_array
0.42% bench [nf_conntrack] [k] 0x000000000002c005
0.38% bench [kernel.vmlinux] [k] __sys_bpf
0.37% bench [kernel.vmlinux] [k] srso_return_thunk
0.24% bench [nf_conntrack] [k] 0x000000000002c001
0.23% bench [kernel.vmlinux] [k] read_hpet
0.21% bench [kernel.vmlinux] [k] entry_SYSCALL_64
0.19% bench [kernel.vmlinux] [k] bpf_prog_test_run_raw_tp
0.17% bench [nf_conntrack] [k] 0x000000000002c000
0.15% bench [kernel.vmlinux] [k] _copy_from_user
0.15% bench [kernel.vmlinux] [k] syscall_exit_to_user_mode
0.14% bench [nf_conntrack] [k] 0x000000000002c027
0.14% bench [nf_conntrack] [k] 0x000000000002c0ca
0.13% swapper [kernel.vmlinux] [k] read_hpet
0.12% bench [kernel.vmlinux] [k] _copy_to_user
0.12% bench [nf_conntrack] [k] 0x000000000002c086
0.11% bench [nf_conntrack] [k] 0x000000000002c0c0
0.11% bench [kernel.vmlinux] [k] __fdget
0.11% bench bench [.] bpf_prog_test_run_opts
0.11% bench [kernel.vmlinux] [k] __bpf_prog_test_run_raw_tp
0.10% bench [nf_conntrack] [k] 0x000000000002c059
0.09% bench [kernel.vmlinux] [k] get_symbol_offset
0.09% bench [kernel.vmlinux] [k] do_syscall_64
0.08% bench [kernel.vmlinux] [k] __bpf_prog_get
0.08% bench [kernel.vmlinux] [k] memchr_inv
0.07% bench [kernel.vmlinux] [k] syscall_return_via_sysret
0.06% bench [kernel.vmlinux] [k] x64_sys_call
0.06% bench [kernel.vmlinux] [k] rep_movs_alternative
0.06% bench [kernel.vmlinux] [k] ftrace_replace_code
0.04% bench [kernel.vmlinux] [k] __x64_sys_bpf
0.04% bench [kernel.vmlinux] [k] get_symbol_pos
0.04% bench bench [.] sys_bpf
0.04% bench [kernel.vmlinux] [k] vsnprintf
0.04% bench [kernel.vmlinux] [k] number
0.04% bench bench [.] ptr_to_u64
0.04% bench [kernel.vmlinux] [k] kfree
0.03% bench [kernel.vmlinux] [k] seq_read_iter
0.03% bench [kernel.vmlinux] [k] security_bpf
0.03% bench libc.so.6 [.] syscall
0.03% bench [kernel.vmlinux] [k] kallsyms_expand_symbol.constprop.0
0.03% kworker/2:2-eve [kernel.vmlinux] [k] memcpy_orig
0.03% swapper [amdgpu] [k] 0x0000000000002f75
0.03% bench bench [.] trigger_producer_batch
0.03% bench [kernel.vmlinux] [k] __bpf_prog_put
0.02% bench [kernel.vmlinux] [k] seq_printf
0.02% bench [kernel.vmlinux] [k] smp_call_function_many_cond
0.02% kworker/2:2-eve [amdgpu] [k] 0x0000000000002f75
0.02% bench [kernel.vmlinux] [k] __check_object_size
0.02% bench [kernel.vmlinux] [k] ftrace_test_record
0.02% bench [kernel.vmlinux] [k] seq_write
0.02% bench [kernel.vmlinux] [k] ftrace_lookup_ip
0.02% bench [kernel.vmlinux] [k] t_show
0.02% bench [kernel.vmlinux] [k] print_rec
0.02% swapper [kernel.vmlinux] [k] io_idle
0.01% bench [amdgpu] [k] 0x0000000000002f75
0.01% kworker/2:2-eve [kernel.vmlinux] [k] memcmp
0.01% bench [kernel.vmlinux] [k] format_decode
0.01% bench [kernel.vmlinux] [k] memcpy_orig
0.01% bench [kernel.vmlinux] [k] __ftrace_hash_rec_update.part.0
0.01% bench [kernel.vmlinux] [k] mod_find
0.01% swapper [kernel.vmlinux] [k] srso_untrain_ret
0.01% bench [kernel.vmlinux] [k] kallsyms_lookup_buildid
0.01% bench [kernel.vmlinux] [k] __seq_puts
0.01% bench [kernel.vmlinux] [k] native_read_msr
0.01% bench [kernel.vmlinux] [k] check_stack_object
0.01% bench [kernel.vmlinux] [k] fput
0.01% bench bench [.] libbpf_err_errno
0.01% bench [kernel.vmlinux] [k] fpregs_assert_state_consistent
0.01% bench [kernel.vmlinux] [k] ftrace_check_record
0.01% bench [kernel.vmlinux] [k] t_next
0.01% bench [kernel.vmlinux] [k] strlen
0.01% swapper [kernel.vmlinux] [k] native_sched_clock
0.01% bench [kernel.vmlinux] [k] __sysvec_apic_timer_interrupt
0.01% bench libc.so.6 [.] 0x00000000001a0f10
0.01% bench [kernel.vmlinux] [k] native_write_msr
0.01% bench [kernel.vmlinux] [k] __update_load_avg_se
0.01% bench [kernel.vmlinux] [k] amd_pmu_addr_offset
0.01% bench [kernel.vmlinux] [k] module_address_lookup
0.01% bench [kernel.vmlinux] [k] ftrace_rec_iter_next
0.01% bench [kernel.vmlinux] [k] string
0.01% bench [kernel.vmlinux] [k] asm_exc_page_fault
0.01% swapper [kernel.vmlinux] [k] __get_next_timer_interrupt
0.01% bench [kernel.vmlinux] [k] clear_page_rep
0.01% kworker/2:2-eve [kernel.vmlinux] [k] delay_halt_mwaitx
0.01% bench [kernel.vmlinux] [k] __handle_mm_fault
0.01% swapper [kernel.vmlinux] [k] ktime_get
0.01% kworker/2:2-eve [kernel.vmlinux] [k] clear_page_rep
0.01% swapper [kernel.vmlinux] [k] menu_select
0.00% perf [kernel.vmlinux] [k] clear_page_rep
0.00% bench [kernel.vmlinux] [k] ftrace_rec_iter_record
0.00% bench [kernel.vmlinux] [k] account_process_tick
0.00% bench [kernel.vmlinux] [k] __update_load_avg_cfs_rq
0.00% bench [kernel.vmlinux] [k] update_curr
0.00% bench [kernel.vmlinux] [k] do_sync_core
0.00% bench [kernel.vmlinux] [k] kallsyms_lookup
0.00% bench [kernel.vmlinux] [k] memset_orig
0.00% kworker/6:1-eve [kernel.vmlinux] [k] memcpy_orig
0.00% perf [kernel.vmlinux] [k] rep_movs_alternative
0.00% sshd [kernel.vmlinux] [k] p4d_offset
0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000002f75
0.00% bench [kernel.vmlinux] [k] __mod_memcg_lruvec_state
0.00% swapper [kernel.vmlinux] [k] _raw_spin_unlock
0.00% swapper [kernel.vmlinux] [k] cpuidle_enter_state
0.00% swapper [kernel.vmlinux] [k] rb_erase
0.00% containerd [kernel.vmlinux] [k] read_hpet
0.00% kworker/6:1-eve [kernel.vmlinux] [k] memcmp
0.00% swapper [kernel.vmlinux] [k] irq_entries_start
0.00% swapper [kernel.vmlinux] [k] do_idle
0.00% kworker/2:2-eve [kernel.vmlinux] [k] read_hpet
0.00% swapper [kernel.vmlinux] [k] update_rq_clock
0.00% swapper [kernel.vmlinux] [k] psi_group_change
0.00% swapper [kernel.vmlinux] [k] sched_clock_cpu
0.00% swapper [kernel.vmlinux] [k] do_sync_core
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ec11
0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% swapper [kernel.vmlinux] [k] asm_sysvec_call_function_single
0.00% swapper [kernel.vmlinux] [k] update_sd_lb_stats.constprop.0
0.00% perf [kernel.vmlinux] [k] __filemap_get_folio
0.00% swapper [kernel.vmlinux] [k] irqentry_enter
0.00% bench [kernel.vmlinux] [k] amd_pmu_test_overflow_topbit
0.00% bench [amdgpu] [k] 0x0000000000124f4c
0.00% bench [kernel.vmlinux] [k] update_rq_clock
0.00% bench [kernel.vmlinux] [k] perf_event_task_tick
0.00% bench [amdgpu] [k] 0x0000000000126f3a
0.00% bench [kernel.vmlinux] [k] sched_balance_update_blocked_averages
0.00% bench [kernel.vmlinux] [k] hrtimer_active
0.00% bench [kernel.vmlinux] [k] _copy_to_iter
0.00% bench [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% bench [kernel.vmlinux] [k] calc_global_load_tick
0.00% bench [kernel.vmlinux] [k] ftrace_shutdown.part.0
0.00% bench libc.so.6 [.] 0x00000000001a0fb5
0.00% bench [kernel.vmlinux] [k] task_tick_mm_cid
0.00% bench libc.so.6 [.] 0x00000000001a0fba
0.00% bench [kernel.vmlinux] [k] sched_clock_cpu
0.00% bench [kernel.vmlinux] [k] update_cfs_group
0.00% bench [kernel.vmlinux] [k] irq_exit_rcu
0.00% bench [kernel.vmlinux] [k] native_sched_clock
0.00% bench [kernel.vmlinux] [k] sched_tick
0.00% bench [kernel.vmlinux] [k] sized_strscpy
0.00% bench [kernel.vmlinux] [k] task_tick_fair
0.00% bench [nf_conntrack] [k] 0x000000000002c036
0.00% bench [kernel.vmlinux] [k] perf_adjust_freq_unthr_context
0.00% bench [kernel.vmlinux] [k] idle_cpu
0.00% bench [kernel.vmlinux] [k] bpf_check_uarg_tail_zero
0.00% swapper [kernel.vmlinux] [k] __hrtimer_next_event_base
0.00% swapper [kernel.vmlinux] [k] native_write_msr
0.00% bench [kernel.vmlinux] [k] __module_address.part.0
0.00% bench [kernel.vmlinux] [k] lru_gen_add_folio
0.00% bench [kernel.vmlinux] [k] set_pte_range
0.00% bench [kernel.vmlinux] [k] pfn_pte
0.00% swapper [kernel.vmlinux] [k] error_entry
0.00% swapper [kernel.vmlinux] [k] idle_cpu
0.00% bench [kernel.vmlinux] [k] unmap_page_range
0.00% bench [kernel.vmlinux] [k] __count_memcg_events
0.00% bench [kernel.vmlinux] [k] __irq_exit_rcu
0.00% perf [kernel.vmlinux] [k] __perf_event_enable
0.00% bench [kernel.vmlinux] [k] seq_putc
0.00% perf [kernel.vmlinux] [k] srso_untrain_ret
0.00% swapper [kernel.vmlinux] [k] __irq_exit_rcu
0.00% kworker/2:2-eve [kernel.vmlinux] [k] __kmalloc_cache_noprof
0.00% bench libc.so.6 [.] 0x00000000001a0bd7
0.00% perf [kernel.vmlinux] [k] _raw_spin_lock
0.00% kworker/u65:2-e [kernel.vmlinux] [k] __bio_split_to_limits
0.00% bench [kernel.vmlinux] [k] do_anonymous_page
0.00% perf [kernel.vmlinux] [k] __es_insert_extent
0.00% bench [kernel.vmlinux] [k] __mem_cgroup_charge
0.00% kworker/2:2-eve [kernel.vmlinux] [k] __wait_for_common
0.00% bench [kernel.vmlinux] [k] __folio_throttle_swaprate
0.00% swapper [kernel.vmlinux] [k] __switch_to_asm
0.00% swapper [kernel.vmlinux] [k] add_interrupt_randomness
0.00% kworker/2:2-eve [kernel.vmlinux] [k] srso_untrain_ret
0.00% kworker/2:2-eve [kernel.vmlinux] [k] memset_orig
0.00% kworker/u65:2-e [kernel.vmlinux] [k] percpu_counter_add_batch
0.00% swapper [kernel.vmlinux] [k] tick_irq_enter
0.00% swapper [kernel.vmlinux] [k] tick_check_broadcast_expired
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d28e
0.00% perf [kernel.vmlinux] [k] __lruvec_stat_mod_folio
0.00% bench [kernel.vmlinux] [k] kernfs_fop_read_iter
0.00% bench [kernel.vmlinux] [k] __mod_zone_page_state
0.00% kworker/2:2-eve [kernel.vmlinux] [k] delay_halt
0.00% perf [kernel.vmlinux] [k] __mod_memcg_lruvec_state
0.00% perf [kernel.vmlinux] [k] xas_load
0.00% perf [kernel.vmlinux] [k] workingset_update_node
0.00% perf [kernel.vmlinux] [k] _raw_spin_unlock
0.00% swapper [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% kworker/2:2-eve [kernel.vmlinux] [k] post_alloc_hook
0.00% bench [kernel.vmlinux] [k] refill_obj_stock
0.00% perf [kernel.vmlinux] [k] __alloc_pages_noprof
0.00% swapper [kernel.vmlinux] [k] cpuidle_enter
0.00% swapper [kernel.vmlinux] [k] sched_balance_domains
0.00% perf [kernel.vmlinux] [k] __handle_mm_fault
0.00% swapper [kernel.vmlinux] [k] ct_kernel_enter.constprop.0
0.00% bench [kernel.vmlinux] [k] mab_mas_cp
0.00% swapper [amdgpu] [k] 0x00000000004619a7
0.00% swapper [kernel.vmlinux] [k] tick_nohz_irq_exit
0.00% perf [kernel.vmlinux] [k] __memcg_slab_post_alloc_hook
0.00% bench [kernel.vmlinux] [k] __alloc_pages_noprof
0.00% swapper [kernel.vmlinux] [k] irq_exit_rcu
0.00% bench [kernel.vmlinux] [k] memcmp
0.00% swapper [kernel.vmlinux] [k] asm_sysvec_call_function
0.00% perf [kernel.vmlinux] [k] perf_poll
0.00% swapper [kernel.vmlinux] [k] enqueue_task_fair
0.00% bench [kernel.vmlinux] [k] __rmqueue_pcplist
0.00% kworker/u65:4-e [kernel.vmlinux] [k] __iommu_map
0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000992cd2
0.00% kworker/6:1-eve [amdgpu] [k] 0x00000000004b2ade
0.00% swapper [kernel.vmlinux] [k] wakeup_preempt
0.00% migration/11 [kernel.vmlinux] [k] enqueue_task
0.00% swapper [kernel.vmlinux] [k] hrtimer_update_next_event
0.00% swapper [kernel.vmlinux] [k] native_apic_mem_eoi
0.00% bench [kernel.vmlinux] [k] is_sync_callback_calling_insn
0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000992cb3
0.00% systemd-network libc.so.6 [.] clock_gettime
0.00% kworker/u65:4-e [kernel.vmlinux] [k] filemap_get_folios_tag
0.00% kworker/u65:4-e [kernel.vmlinux] [k] fscrypt_mergeable_bio_bh
0.00% swapper [kernel.vmlinux] [k] hrtimer_start_range_ns
0.00% multipathd [kernel.vmlinux] [k] cpuacct_charge
0.00% swapper [kernel.vmlinux] [k] hrtimer_get_next_event
0.00% kworker/12:1-ev [amdgpu] [k] 0x00000000004b09fc
0.00% kworker/12:1-ev [kernel.vmlinux] [k] memcpy_orig
0.00% kworker/u65:4-e [kernel.vmlinux] [k] wbt_track
0.00% kworker/u65:4-e [kernel.vmlinux] [k] __lruvec_stat_mod_folio
0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000002f75
0.00% bench [kernel.vmlinux] [k] do_jit
0.00% kworker/u65:4-e [kernel.vmlinux] [k] __folio_start_writeback
0.00% bench [kernel.vmlinux] [k] uncharge_folio
0.00% perf [kernel.vmlinux] [k] do_fault
0.00% perf [kernel.vmlinux] [k] __kmalloc_noprof
0.00% containerd [kernel.vmlinux] [k] blkcg_maybe_throttle_current
0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000523901
0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000538414
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000465f98
0.00% irqbalance [kernel.vmlinux] [k] format_decode
0.00% swapper [kernel.vmlinux] [k] handle_edge_irq
0.00% kworker/12:1-ev [kernel.vmlinux] [k] srso_untrain_ret
0.00% kworker/u65:4-e [kernel.vmlinux] [k] __find_get_block
0.00% systemd-journal [kernel.vmlinux] [k] __put_user_nocheck_4
0.00% systemd-journal libsystemd-shared-249.so [.] 0x00000000001c9c3e
0.00% swapper [kernel.vmlinux] [k] local_touch_nmi
0.00% bench [kernel.vmlinux] [k] __mem_cgroup_uncharge_folios
0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock
0.00% swapper [kernel.vmlinux] [k] select_task_rq_fair
0.00% swapper [kernel.vmlinux] [k] need_update
0.00% bench [kernel.vmlinux] [k] rcu_segcblist_enqueue
0.00% swapper [kernel.vmlinux] [k] tick_nohz_get_sleep_length
0.00% containerd [kernel.vmlinux] [k] _raw_spin_unlock
0.00% containerd [kernel.vmlinux] [k] ep_autoremove_wake_function
0.00% kworker/u66:1-e [kernel.vmlinux] [k] __percpu_counter_sum
0.00% swapper [kernel.vmlinux] [k] ktime_get_mono_fast_ns
0.00% swapper [kernel.vmlinux] [k] sysvec_apic_timer_interrupt
0.00% swapper [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% migration/10 [kernel.vmlinux] [k] dequeue_entity
0.00% swapper [kernel.vmlinux] [k] handle_irq_event
0.00% perf [kernel.vmlinux] [k] srso_return_thunk
0.00% perf [kernel.vmlinux] [k] xas_find_conflict
0.00% perf-exec [kernel.vmlinux] [k] mas_next_slot
0.00% swapper [kernel.vmlinux] [k] fetch_next_timer_interrupt.constprop.0
0.00% bench [kernel.vmlinux] [k] get_page_from_freelist
0.00% swapper [amdgpu] [k] 0x0000000000125079
0.00% kworker/u65:4-e [kernel.vmlinux] [k] sched_balance_newidle
0.00% migration/6 [kernel.vmlinux] [k] raw_spin_rq_unlock
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000444a05
0.00% swapper [kernel.vmlinux] [k] fetch_pte
0.00% perf [kernel.vmlinux] [k] __pte_offset_map
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000534985
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000003ac5
0.00% bench [kernel.vmlinux] [k] perf_iterate_sb
0.00% swapper [kernel.vmlinux] [k] get_cpu_device
0.00% perf [kernel.vmlinux] [k] xas_start
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000098fea5
0.00% perf [kernel.vmlinux] [k] mark_buffer_dirty
0.00% containerd containerd [.] runtime.cleantimers
0.00% bench [kernel.vmlinux] [k] __cond_resched
0.00% kworker/2:2-eve [drm_kms_helper] [k] 0x00000000000008eb
0.00% kworker/2:2-eve [raid6_pq] [k] 0x000000000000c24b
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ea79
0.00% perf [kernel.vmlinux] [k] cpu_util
0.00% swapper [kernel.vmlinux] [k] asm_common_interrupt
0.00% perf [kernel.vmlinux] [k] __filemap_add_folio
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000007d1eab
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051a45f
0.00% perf [kernel.vmlinux] [k] ext4_es_lookup_extent
0.00% swapper [kernel.vmlinux] [k] __flush_smp_call_function_queue
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000519af6
0.00% swapper [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477759
0.00% perf-exec [kernel.vmlinux] [k] kmem_cache_free
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000539303
0.00% kworker/2:2-eve [drm] [k] 0x0000000000020c1c
0.00% swapper [kernel.vmlinux] [k] asm_sysvec_apic_timer_interrupt
0.00% bench [kernel.vmlinux] [k] alloc_pages_mpol_noprof
0.00% bench [kernel.vmlinux] [k] filemap_map_pages
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477767
0.00% swapper [kernel.vmlinux] [k] update_load_avg
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d3d6
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051d905
0.00% kworker/2:2-eve [drm_kms_helper] [k] 0x0000000000004ae8
0.00% kworker/u65:2-e [kernel.vmlinux] [k] __mod_memcg_lruvec_state
0.00% kworker/u65:2-e [kernel.vmlinux] [k] bio_associate_blkg
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004916dd
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000465ecb
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b780a
0.00% swapper [kernel.vmlinux] [k] sysvec_call_function
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e764
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ed72
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051c7bf
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ebe0
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000523fba
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000832660
0.00% swapper [kernel.vmlinux] [k] acpi_idle_do_entry
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097ddfc
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d334
0.00% kworker/u65:2-e [kernel.vmlinux] [k] soft_cursor
0.00% perf [kernel.vmlinux] [k] sched_balance_rq
0.00% kworker/u65:2-e [kernel.vmlinux] [k] __fprop_add_percpu
0.00% swapper [kernel.vmlinux] [k] rb_next
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000977ece
0.00% perf [kernel.vmlinux] [k] __folio_mark_dirty
0.00% perf [kernel.vmlinux] [k] filemap_alloc_folio_noprof
0.00% perf [kernel.vmlinux] [k] get_mem_cgroup_from_mm
0.00% perf [kernel.vmlinux] [k] ext4_da_reserve_space
0.00% perf [kernel.vmlinux] [k] read_hpet
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051cd3e
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000532f0f
0.00% kworker/2:2-eve [kernel.vmlinux] [k] __mod_node_page_state
0.00% systemd-journal [kernel.vmlinux] [k] mod_objcg_state
0.00% jbd2/dm-0-8 [kernel.vmlinux] [k] dm_submit_bio
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000517b6d
0.00% kcompactd0 [kernel.vmlinux] [k] idle_cpu
0.00% kworker/2:2-eve [kernel.vmlinux] [k] free_tail_page_prepare
0.00% swapper [amdgpu] [k] 0x00000000002f7f65
0.00% swapper [kernel.vmlinux] [k] tick_program_event
0.00% containerd [kernel.vmlinux] [k] psi_group_change
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000053776f
0.00% swapper [kernel.vmlinux] [k] srso_return_thunk
0.00% perf [kernel.vmlinux] [k] ext4_claim_free_clusters
0.00% perf [kernel.vmlinux] [k] generic_perform_write
0.00% perf [kernel.vmlinux] [k] mem_cgroup_commit_charge
0.00% perf [kernel.vmlinux] [k] file_modified
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e779
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000053974c
0.00% perf [kernel.vmlinux] [k] ext4_da_write_begin
0.00% perf [kernel.vmlinux] [k] filemap_add_folio
0.00% perf libc.so.6 [.] write
0.00% perf [kernel.vmlinux] [k] __es_remove_extent
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000052244a
0.00% swapper [kernel.vmlinux] [k] ct_nmi_exit
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000034bee5
0.00% kworker/2:2-eve [kernel.vmlinux] [k] __rcu_read_lock
0.00% perf [kernel.vmlinux] [k] __dquot_alloc_space
0.00% swapper [kernel.vmlinux] [k] native_read_msr
0.00% perf [kernel.vmlinux] [k] folio_unlock
0.00% perf [kernel.vmlinux] [k] radix_tree_node_ctor
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000006c5b82
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000527259
0.00% perf [kernel.vmlinux] [k] __mod_node_page_state
0.00% perf [kernel.vmlinux] [k] __radix_tree_lookup
0.00% kworker/u65:2-e [kernel.vmlinux] [k] __rcu_read_unlock
0.00% perf [kernel.vmlinux] [k] ext4_get_reserved_space
0.00% swapper [kernel.vmlinux] [k] hrtimer_cancel
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004ba8ce
0.00% swapper [amdgpu] [k] 0x0000000000126f3a
0.00% perf perf [.] perf_cpu_map__idx
0.00% swapper [kernel.vmlinux] [k] sched_balance_update_blocked_averages
0.00% systemd-journal [kernel.vmlinux] [k] do_timerfd_settime
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097ddc5
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097e5a4
0.00% perf [kernel.vmlinux] [k] folio_batch_move_lru
0.00% swapper [amdgpu] [k] 0x000000000046f8a5
0.00% perf [kernel.vmlinux] [k] xas_nomem
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051d24c
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097df7c
0.00% tuned [kernel.vmlinux] [k] psi_task_switch
0.00% swapper [kernel.vmlinux] [k] __dequeue_entity
0.00% swapper [amdgpu] [k] 0x00000000006e7c5d
0.00% swapper [kernel.vmlinux] [k] rcu_preempt_deferred_qs
0.00% kworker/7:1-eve [kernel.vmlinux] [k] __free_one_page
0.00% swapper [kernel.vmlinux] [k] memchr_inv
0.00% perf [kernel.vmlinux] [k] psi_task_switch
0.00% swapper [kernel.vmlinux] [k] _nohz_idle_balance.isra.0
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000009a6fbd
0.00% swapper [kernel.vmlinux] [k] acpi_idle_enter
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000456a9f
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004ac642
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477888
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000529a52
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b36d2
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000525b33
0.00% swapper [kernel.vmlinux] [k] tmigr_active_up
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000452444
0.00% swapper [amdgpu] [k] 0x000000000046f865
0.00% swapper [kernel.vmlinux] [k] next_timer_interrupt
0.00% kworker/6:1-eve [kernel.vmlinux] [k] __free_pages_ok
0.00% perf [kernel.vmlinux] [k] exc_page_fault
0.00% swapper [kernel.vmlinux] [k] __schedule
0.00% irqbalance [kernel.vmlinux] [k] __kmalloc_cache_noprof
0.00% swapper [kernel.vmlinux] [k] cpuidle_not_available
0.00% containerd [kernel.vmlinux] [k] pick_next_task_fair
0.00% kworker/6:1-eve [amdgpu] [k] 0x00000000009a70b7
0.00% swapper [kernel.vmlinux] [k] tick_nohz_next_event
0.00% swapper [kernel.vmlinux] [k] tick_check_oneshot_broadcast_this_cpu
0.00% swapper [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% irqbalance [kernel.vmlinux] [k] show_stat
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b8295
0.00% swapper [amdgpu] [k] 0x000000000045d385
0.00% swapper [kernel.vmlinux] [k] tick_nohz_idle_stop_tick
0.00% containerd [kernel.vmlinux] [k] __schedule
0.00% swapper [kernel.vmlinux] [k] switch_mm_irqs_off
0.00% swapper [amdgpu] [k] 0x0000000000461a21
0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000051d7c2
0.00% swapper [kernel.vmlinux] [k] try_to_wake_up
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e7cf
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ed4e
0.00% perf [kernel.vmlinux] [k] perf_mmap_to_page
0.00% migration/7 [kernel.vmlinux] [k] update_sd_lb_stats.constprop.0
0.00% irqbalance [kernel.vmlinux] [k] __rcu_read_lock
0.00% migration/3 [kernel.vmlinux] [k] __update_load_avg_cfs_rq
0.00% swapper [br_netfilter] [k] 0x0000000000000775
0.00% bench [kernel.vmlinux] [k] remove_vma
0.00% swapper [kernel.vmlinux] [k] sched_clock_noinstr
0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000529169
0.00% swapper [amdgpu] [k] 0x000000000000df07
0.00% swapper [kernel.vmlinux] [k] __switch_to
0.00% swapper [kernel.vmlinux] [k] place_entity
0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000097dd9a
0.00% swapper [amdgpu] [k] 0x00000000006e7c5a
0.00% irqbalance [kernel.vmlinux] [k] __rcu_read_unlock
0.00% migration/6 [kernel.vmlinux] [k] enqueue_entity
0.00% swapper [kernel.vmlinux] [k] __update_load_avg_cfs_rq
0.00% perf [kernel.vmlinux] [k] event_function
0.00% perf [kernel.vmlinux] [k] do_user_addr_fault
0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064eaee
0.00% migration/4 [kernel.vmlinux] [k] __update_load_avg_cfs_rq
0.00% swapper [amdgpu] [k] 0x00000000006e79ff
0.00% containerd [kernel.vmlinux] [k] schedule_hrtimeout_range_clock
0.00% irqbalance [kernel.vmlinux] [k] security_file_open
0.00% perf [kernel.vmlinux] [k] _find_next_and_bit
0.00% multipathd [kernel.vmlinux] [k] __rseq_handle_notify_resume
0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_r12
0.00% swapper [kernel.vmlinux] [k] rcu_note_context_switch
0.00% bench [kernel.vmlinux] [k] free_unref_folios
0.00% migration/0 [kernel.vmlinux] [k] migration_cpu_stop
0.00% migration/1 [kernel.vmlinux] [k] rcu_note_context_switch
0.00% kworker/u66:1-e [kernel.vmlinux] [k] n_tty_receive_buf2
0.00% swapper [kernel.vmlinux] [k] __common_interrupt
0.00% swapper [drm] [k] 0x000000000003d658
0.00% migration/5 [kernel.vmlinux] [k] psi_group_change
0.00% swapper [kernel.vmlinux] [k] clockevents_program_event
0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_array
0.00% perf [kernel.vmlinux] [k] set_pte_range
0.00% perf [kernel.vmlinux] [k] __schedule
0.00% swapper [kernel.vmlinux] [k] __update_load_avg_se
0.00% swapper [kernel.vmlinux] [k] asm_sysvec_reschedule_ipi
0.00% swapper [kernel.vmlinux] [k] local_clock_noinstr
0.00% swapper [kernel.vmlinux] [k] tick_nohz_stop_idle
0.00% swapper [amdgpu] [k] 0x000000000044a039
0.00% kworker/4:1-mm_ [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
0.00% rcu_preempt [kernel.vmlinux] [k] __update_idle_core
0.00% rcu_preempt [kernel.vmlinux] [k] pick_next_task_fair
0.00% rcu_preempt [kernel.vmlinux] [k] prepare_to_swait_event
0.00% rcu_preempt [kernel.vmlinux] [k] sched_balance_rq
0.00% rcu_preempt [kernel.vmlinux] [k] update_rq_clock
0.00% rs:main Q:Reg [kernel.vmlinux] [k] ext4_inode_csum
0.00% swapper [kernel.vmlinux] [k] tmigr_inactive_up
0.00% swapper [kernel.vmlinux] [k] hrtimer_forward
0.00% swapper [kernel.vmlinux] [k] cpuidle_reflect
0.00% perf [kernel.vmlinux] [k] percpu_counter_add_batch
0.00% swapper [amdgpu] [k] 0x000000000031e745
0.00% swapper [kernel.vmlinux] [k] sched_idle_set_state
0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000045675c
0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000051cbb1
0.00% swapper [amdgpu] [k] 0x000000000002dce5
0.00% systemd-journal [kernel.vmlinux] [k] __ext4_ext_check
0.00% swapper [kernel.vmlinux] [k] sched_core_idle_cpu
0.00% swapper [drm] [k] 0x000000000003de52
0.00% swapper [kernel.vmlinux] [k] __msecs_to_jiffies
0.00% swapper [amdgpu] [k] 0x000000000046f9c4
0.00% swapper [kernel.vmlinux] [k] call_cpuidle
0.00% swapper [kernel.vmlinux] [k] cpuidle_governor_latency_req
0.00% kworker/7:1-eve [kernel.vmlinux] [k] __switch_to_asm
0.00% dockerd [kernel.vmlinux] [k] apparmor_file_permission
0.00% swapper [amdgpu] [k] 0x0000000000126f37
0.00% swapper [amdgpu] [k] 0x00000000003218b6
0.00% swapper [amdgpu] [k] 0x00000000006e8d65
0.00% swapper [amdgpu] [k] 0x000000000096f62d
0.00% systemd-journal [kernel.vmlinux] [k] __es_tree_search.isra.0
0.00% swapper [kernel.vmlinux] [k] update_group_capacity
0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000005225c3
0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000990c4f
0.00% swapper [kernel.vmlinux] [k] timer_base_try_to_set_idle
0.00% swapper [kernel.vmlinux] [k] ct_idle_exit
0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_rbx
0.00% swapper [kernel.vmlinux] [k] fast_mix
0.00% swapper [kernel.vmlinux] [k] sched_clock_idle_sleep_event
0.00% swapper [kernel.vmlinux] [k] touch_softlockup_watchdog_sched
0.00% containerd [kernel.vmlinux] [k] psi_task_switch
0.00% swapper [amdgpu] [k] 0x0000000000123820
0.00% swapper [kernel.vmlinux] [k] sched_clock
0.00% swapper [amdgpu] [k] 0x00000000002f46a5
0.00% swapper [amdgpu] [k] 0x0000000000449ff5
0.00% containerd [kernel.vmlinux] [k] _copy_to_user
0.00% swapper [kernel.vmlinux] [k] enqueue_task
0.00% swapper [amdgpu] [k] 0x000000000096f625
0.00% swapper [amdgpu] [k] 0x00000000006e7c66
0.00% swapper [kernel.vmlinux] [k] hrtimer_interrupt
0.00% swapper [kernel.vmlinux] [k] irq_enter_rcu
0.00% perf [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% perf [kernel.vmlinux] [k] poll_freewait
0.00% rcu_preempt [kernel.vmlinux] [k] __rcu_read_lock
0.00% swapper [kernel.vmlinux] [k] wake_q_add
0.00% kworker/7:1H-kb [kernel.vmlinux] [k] __switch_to_asm
0.00% swapper [amdgpu] [k] 0x00000000002f8058
0.00% swapper [kernel.vmlinux] [k] sched_clock_tick
0.00% swapper [kernel.vmlinux] [k] slab_update_freelist.constprop.0.isra.0
0.00% swapper [kernel.vmlinux] [k] tick_nohz_get_next_hrtimer
0.00% swapper [kernel.vmlinux] [k] ct_kernel_enter_state
0.00% swapper [kernel.vmlinux] [k] hrtimer_next_event_without
0.00% swapper [kernel.vmlinux] [k] resched_curr
0.00% swapper [amdgpu] [k] 0x0000000000003a55
0.00% swapper [amdgpu] [k] 0x000000000045d3df
0.00% swapper [amdgpu] [k] 0x000000000098aeb9
0.00% containerd [kernel.vmlinux] [k] __switch_to
0.00% swapper [kernel.vmlinux] [k] rcu_sched_clock_irq
0.00% kworker/9:1-eve [kernel.vmlinux] [k] sched_clock_cpu
0.00% swapper [amdgpu] [k] 0x0000000000124ef5
0.00% containerd [kernel.vmlinux] [k] update_curr
0.00% swapper [drm] [k] 0x0000000000035f4e
0.00% swapper [kernel.vmlinux] [k] tmigr_update_events
0.00% swapper [kernel.vmlinux] [k] profile_pc
0.00% swapper [nvme] [k] 0x0000000000004315
0.00% swapper [kernel.vmlinux] [k] raw_spin_rq_unlock
0.00% containerd [kernel.vmlinux] [k] posix_get_monotonic_timespec
0.00% swapper [amdgpu] [k] 0x0000000000002f4b
0.00% swapper [amdgpu] [k] 0x000000000034bf25
0.00% perf [kernel.vmlinux] [k] get_page_from_freelist
0.00% rcu_preempt [kernel.vmlinux] [k] __mod_timer
0.00% perf [kernel.vmlinux] [k] mutex_unlock
0.00% swapper [amdgpu] [k] 0x0000000000460f85
0.00% swapper [amdgpu] [k] 0x0000000000977ec5
0.00% swapper [amdgpu] [k] 0x0000000000977ed6
0.00% swapper [kernel.vmlinux] [k] sched_clock_idle_wakeup_event
0.00% swapper [amdgpu] [k] 0x0000000000124016
0.00% swapper [kernel.vmlinux] [k] pick_next_task_fair
0.00% swapper [raid6_pq] [k] 0x000000000001acf5
0.00% swapper [amdgpu] [k] 0x0000000000977ec6
0.00% systemd-journal [kernel.vmlinux] [k] security_inode_setattr
0.00% swapper [kernel.vmlinux] [k] __sysvec_apic_timer_interrupt
0.00% swapper [amdgpu] [k] 0x0000000000461985
0.00% swapper [amdgpu] [k] 0x0000000000123825
0.00% swapper [amdgpu] [k] 0x0000000000126ee9
0.00% swapper [amdgpu] [k] 0x000000000096f6cd
0.00% perf [kernel.vmlinux] [k] do_syscall_64
0.00% systemd-journal [kernel.vmlinux] [k] unmap_mapping_range
0.00% systemd-journal libc.so.6 [.] 0x0000000000090a64
0.00% containerd [kernel.vmlinux] [k] dequeue_entity
0.00% systemd-journal [kernel.vmlinux] [k] dax_layout_busy_page
0.00% swapper [kernel.vmlinux] [k] apic_ack_irq
0.00% swapper [drm] [k] 0x000000000003d83b
0.00% containerd [kernel.vmlinux] [k] __futex_queue
0.00% swapper [kernel.vmlinux] [k] complete_all
0.00% swapper [amdgpu] [k] 0x000000000098b0d6
0.00% swapper [kernel.vmlinux] [k] can_stop_idle_tick
0.00% swapper [amdgpu] [k] 0x000000000000dee5
0.00% swapper [amdgpu] [k] 0x000000000044ecef
0.00% swapper [amdgpu] [k] 0x000000000000dee6
0.00% migration/14 [kernel.vmlinux] [k] update_curr_common
0.00% rs:main Q:Reg [kernel.vmlinux] [k] ext4_get_group_desc
0.00% swapper [kernel.vmlinux] [k] tick_do_update_jiffies64
0.00% swapper [kernel.vmlinux] [k] tmigr_cpu_new_timer
0.00% kworker/7:1-eve [kernel.vmlinux] [k] free_pcppages_bulk
0.00% dockerd [kernel.vmlinux] [k] srso_untrain_ret
0.00% dockerd [kernel.vmlinux] [k] update_curr
0.00% swapper [amdgpu] [k] 0x0000000000124f0c
0.00% swapper [drm] [k] 0x000000000003f455
0.00% swapper [amdgpu] [k] 0x00000000003218b5
0.00% swapper [kernel.vmlinux] [k] ct_irq_enter
0.00% swapper [kernel.vmlinux] [k] ct_irq_exit
0.00% swapper [kernel.vmlinux] [k] blk_stat_add
0.00% swapper [amdgpu] [k] 0x00000000006e8dd0
0.00% swapper [amdgpu] [k] 0x0000000000124f4c
0.00% swapper [amdgpu] [k] 0x000000000098aef6
0.00% swapper [drm] [k] 0x000000000003f468
0.00% swapper [kernel.vmlinux] [k] nohz_balance_enter_idle
0.00% swapper [kernel.vmlinux] [k] __wake_up_common
0.00% swapper [amdgpu] [k] 0x00000000001240ea
0.00% swapper [kernel.vmlinux] [k] __wake_up
0.00% swapper [amdgpu] [k] 0x00000000002f7f81
0.00% swapper [amdgpu] [k] 0x000000000046f97f
0.00% swapper [drm] [k] 0x000000000003f45c
0.00% swapper [kernel.vmlinux] [k] __mod_memcg_lruvec_state
0.00% swapper [kernel.vmlinux] [k] mempool_free_slab
0.00% swapper [amdgpu] [k] 0x0000000000977ef0
0.00% swapper [kernel.vmlinux] [k] x86_pmu_disable_all
0.00% swapper [amdgpu] [k] 0x00000000002f7f71
0.00% kworker/3:1H-kb [kernel.vmlinux] [k] read_hpet
0.00% swapper [kernel.vmlinux] [k] psi_task_change
0.00% swapper [kernel.vmlinux] [k] perf_adjust_freq_unthr_context
0.00% swapper [kernel.vmlinux] [k] account_idle_ticks
0.00% swapper [amdgpu] [k] 0x000000000045d365
0.00% swapper [kernel.vmlinux] [k] ttwu_do_activate
0.00% swapper [raid6_pq] [k] 0x000000000001acc5
0.00% swapper [kernel.vmlinux] [k] enqueue_entity
0.00% wpa_supplicant [kernel.vmlinux] [k] read_hpet
0.00% swapper [amdgpu] [k] 0x0000000000012205
0.00% swapper [drm] [k] 0x000000000003d649
0.00% containerd [kernel.vmlinux] [k] __update_load_avg_se
0.00% rs:main Q:Reg [kernel.vmlinux] [k] futex_wake
0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_r13
0.00% swapper [kernel.vmlinux] [k] sched_balance_softirq
0.00% swapper [kernel.vmlinux] [k] timerqueue_del
0.00% swapper [kernel.vmlinux] [k] cpu_util
#
# (Tip: Limit to show entries above 5% only: perf report --percent-limit 5)
#-- Masami Hiramatsu (Google) [off-list ref]