On Fri, 30 Jul 2021 13:32:12 +0800
Gang Li [off-list ref] wrote:

Thanks! I have tried your suggestion. They are great, especially 
synthetic-events.

If don't print ip per event, we can only guess which one cause the 
contention by "hitcount".

 >   
(https://www.kernel.org/doc/html/latest/trace/histogram.html#synthetic-events)

But it seems that they only support histogram, can I print the
synthetic-events args per event in /sys/kernel/debug/tracing/trace
like other events? I haven't found that in kernel doc.

Yes, synthetic events are just like normal events, and have triggers,
stack traces, and do pretty much anything that another event can do.

I'm just finishing up a libtracfs called tracefs_sql() (hopefully
posting it today), that allows you to create a synthetic event via an
SQL statement. But I don't think this is what you are looking for.

What about using function tracing? Because the tracepoint is called
from __mmap_lock* helper functions that function tracer can see, you
can just do the following:

 # trace-cmd start -e mmap_lock -p function -l '__mmap_lock_*' 
 # trace-cmd show
[..]
       trace-cmd-1840    [006] ....   194.576801: __mmap_lock_do_trace_start_locking <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576805: mmap_lock_start_locking: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false

       trace-cmd-1840    [006] ....   194.576806: __mmap_lock_do_trace_acquire_returned <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576807: mmap_lock_acquire_returned: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false success=true

       trace-cmd-1840    [006] ....   194.576811: __mmap_lock_do_trace_released <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576812: mmap_lock_released: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false

       trace-cmd-1840    [006] ....   194.576815: __mmap_lock_do_trace_start_locking <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576816: mmap_lock_start_locking: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false

       trace-cmd-1840    [006] ....   194.576816: __mmap_lock_do_trace_acquire_returned <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576817: mmap_lock_acquire_returned: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false success=true

       trace-cmd-1840    [006] ....   194.576820: __mmap_lock_do_trace_released <-do_user_addr_fault
       trace-cmd-1840    [006] ...1   194.576821: mmap_lock_released: mm=000000006515cb1f memcg_path=/user.slice/user-0.slice/session-2.scope write=false


This looks exactly like the robots you are looking for.

-- Steve