Re: 'perf probe' and symbols from .text.<something>

Hi Evgenii,

On Thu, 18 Feb 2021 20:09:17 +0300
Evgenii Shatokhin [off-list ref] wrote:

Hi,

It seems, 'perf probe' can only see functions from .text section in the 
kernel modules, but not from .text.unlikely or other .text.* sections.

For example, with kernel 5.11 and nf_conntrack.ko with debug info, 'perf 
probe' succeeds for nf_conntrack_attach() from .text and fails for 
nf_ct_resolve_clash() from .text.unlikely:

Thanks for reporting it!

------------
# perf probe -v -m nf_conntrack nf_ct_resolve_clash
probe-definition(0): nf_ct_resolve_clash
symbol:nf_ct_resolve_clash file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Failed to get build-id from nf_conntrack.
Cache open error: -1
Open Debuginfo file: 
/lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko
Try to find probe point from debuginfo.
Matched function: nf_ct_resolve_clash [33616]
Probe point found: nf_ct_resolve_clash+0
Found 1 probe_trace_events.
Post processing failed or all events are skipped. (-2)
Probe point 'nf_ct_resolve_clash' not found.
   Error: Failed to add events. Reason: No such file or directory (Code: -2)

The above log shows, an error occured while post_process_probe_trace_events(),
and the error code is -ENOENT (-2).
  ----
                pr_debug("Found %d probe_trace_events.\n", ntevs);
                ret = post_process_probe_trace_events(pev, *tevs, ntevs,
                                        pev->target, pev->uprobes, dinfo);
                if (ret < 0 || ret == ntevs) {
                        pr_debug("Post processing failed or all events are skipped. (%d)\n", ret);
  ----

In that function, map__find_symbol() failure will return -ENOENT.

  ----
/* Adjust symbol name and address */
static int post_process_probe_trace_point(struct probe_trace_point *tp,
                                           struct map *map, unsigned long offs)
{
        struct symbol *sym;
        u64 addr = tp->address - offs;

        sym = map__find_symbol(map, addr);
        if (!sym)
                return -ENOENT;
  ----

So it seems "map" may not load the symbol out of ".text".
This need to be fixed, since the map is widely used in the perf.

Anyway, since this is on a module, so even if it can not find the symbol
from map (or failed to load a map), it can fail back to the original symbol.
Let me fix that.

# perf probe -v -m nf_conntrack nf_conntrack_attach
probe-definition(0): nf_conntrack_attach
symbol:nf_conntrack_attach file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Failed to get build-id from nf_conntrack.
Cache open error: -1
Open Debuginfo file: 
/lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko
Try to find probe point from debuginfo.
Matched function: nf_conntrack_attach [2c8c3]
Probe point found: nf_conntrack_attach+0
Found 1 probe_trace_events.
Opening /sys/kernel/tracing//kprobe_events write=1
Opening /sys/kernel/tracing//README write=0
Writing event: p:probe/nf_conntrack_attach 
nf_conntrack:nf_conntrack_attach+0
Added new event:
   probe:nf_conntrack_attach (on nf_conntrack_attach in nf_conntrack)
------------

Is there a way to allow probing of functions in .text.<something> ?

I need to check how machine__kernel_maps() generated maps cut down .text.unlikely.
Arnaldo, I thought the maps in machine__kernel_maps() are generated from
kallsyms (doesn't check .text) right?

Of course, one could place probes using absolute addresses of the 
functions but that would be less convenient.

This also affects many livepatch modules where the kernel code can be 
compiled with -ffunction-sections and each function may end up in a 
separate section .text.<function_name>. 'perf probe' cannot be used 
there, except with the absolute addresses.

Moreover, if FGKASLR patches are merged 
(https://lwn.net/Articles/832434/) and the kernel is built with FGKASLR 
enabled, -ffunction-sections will be used too. 'perf probe' will be 
unable to see the kernel functions then.

Hmm, if the FGKASLAR really randomizes the symbol address, perf-probe
should give up "_text-relative" probe for that kernel, and must fallback
to the "symbol-based" probe. (Are there any way to check the FGKASLR is on?)
The problem of "symbol-based" probe is that local (static) symbols
may share a same name sometimes. In that case, it can not find correct
symbol. (Maybe I can find a candidate from its size.)
Anyway, sometimes the security and usability are trade-off.

Thank you,

-- 
Masami Hiramatsu [off-list ref]