[perf] perf probe fails sometimes on 4.9

On Mon, 29 Jan 2018 13:40:34 +0530
Pintu Kumar [off-list ref] wrote:

Hi All,

'perf probe' is failing sometimes on 4.9.20 with AMD-64.
# perf probe --add schedule
schedule is out of .text, skip it.
  Error: Failed to add events.

If any one have come across this problem please let me know the cause.

Hi Pintu,

Could you run it with --vv?

Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?

Without it, you can not probe source-level probe nor trace local variable.

However, I manually copied the vmlinux file to /boot/ directory, but
still it does not work.

That doesn't work.
CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
Without that option, debuginfo is not generated with vmlinux.

I checked upstream patches until 4.15 but could not find any clue.
Please let me know if there is any fixes available for this.

Could you also ensure that you run perf by root user?

Thank you,

Thank You!
Regards,
Pintu


On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar [off-list ref] wrote:

quoted

Hi,

** Changed the subject now, since these issues are related to general
perf commands.

Following are the issues:

1) perf probe --add schedule - FAILED
output:
schedule is out of .text, skip it.
  Error: Failed to add events.

what is the issue here?
Sometimes it pass and sometimes it fails...
Similar is the case of 'perf inject' as well.

2) perf test - 1 FAILURE
37.1: Test basic BPF filtering                               : FAILED!
37.2: Test BPF prologue generation                           : Skip
37.3: Test BPF relocation checker                            : Skip

bpf: config program 'func=SyS_epoll_wait'
symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
bpf: config 'func=SyS_epoll_wait' is ok
Looking at the vmlinux_path (8 entries long)
Using /boot/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
SyS_epoll_wait is out of .text, skip it.
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 0: FAILED!

Looks like both 1,2 are related.
Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
to /boot/ folder.

-------
Some more info:

Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
unset env: KBUILD_OPTS
include option is set to  -nostdinc -isystem
/usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
-I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
-I./include -I./arch/x86/include/uapi -I./include/uapi
-I./include/generated/uapi -include ./include/linux/kconfig.h
set env: NR_CPUS=8
set env: LINUX_VERSION_CODE=0x40914
set env: CLANG_EXEC=/usr/bin/clang
set env: CLANG_OPTIONS=-xc
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
/usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
-I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
-I./include -I./arch/x86/include/uapi -I./include/uapi
-I./include/generated/uapi -include ./include/linux/kconfig.h
set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build


If you have any clue about these failure please help me.


Thanks,
Pintu


On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar [off-list ref] wrote:

quoted

Hi,

Thanks for your help.
Yes it was a sub version issue.

Earlier, while building the kernel I left the SUBLEVEL as blank.

make -j8 bindeb-pkg SUBLEVEL=20

After passing the correct sublevel now the test is working.

But still following are failing:

16: Try 'import perf' in python, checking link problems      : FAILED!
37.2: Test BPF prologue generation                           : FAILED!


This is the error I get:

16: Try 'import perf' in python, checking link problems      :

--- start ---

test child forked, pid 7637
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named perf
test child finished with -1
---- end ----
Try 'import perf' in python, checking link problems: FAILED!


Looking at the vmlinux_path (8 entries long)
symsrc__init: cannot get elf header.
Failed to find the path for kernel: Invalid ELF file
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 1: FAILED!



Thanks,
Pintu


On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) [off-list ref] wrote:

quoted

On 2018/1/23 20:37, Pintu Kumar wrote:

quoted

Hi All,

I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
kernel version 4.9.20.
I have installed several others packages including: clang, llvm

But, when I run 'perf test' I get some FAILURE.
Specially, 'perf test LLVM' is failing.

Please check the below error logs:

----------------------------------------------------
# perf test LLVM
35: Test LLVM searching and compiling                        :
35.1: Basic BPF llvm compiling test                          : FAILED!
35.2: Test kbuild searching                                  : Skip
35.3: Compile source for BPF prologue generation test        : Skip
35.4: Compile source for BPF relocation test                 : Skip
----------------------------------------------------

When I run with -v I get this:

-----------------------------------------------------
# perf test -v LLVM
35: Test LLVM searching and compiling                        :
35.1: Basic BPF llvm compiling test                          :

--- start ---

test child forked, pid 3304
Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
WARNING:        unable to get correct kernel building directory.
Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
         section of ~/.perfconfig or set it to "" to suppress kbuild
         detection.

Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'


int _version SEC("version") = LINUX_VERSION_CODE;
' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
-DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
$KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
-working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
-
libbpf: 2129190-4a0 doesn't provide kernel version
Failed to parse test case 'Basic BPF llvm compiling test'
test child finished with -1
---- end ----
Test LLVM searching and compiling subtest 0: FAILED!
-----------------------------------------------------


Basic BPF LLVM compiling test is failing.
It reports that bpf could not able to read the kernel version from uname.
Is it because of this that 'perf test LLVM' is failing ?

My uname says:
4.9-xxxx-amd-x86-64

'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
tools/perf/fetch_kernel_version . If your distro uses another method to
report
kernel version number (from your uname string, sublevel can't be
determined),
please help us improve that function like what we have done for ubuntu
(checking /proc/version_signature).

quoted

My kernel version is: 4.9.20 (from Makefile)

So, I dont think there is any problem with kernel version.

If any body have come across this problem please let me know how to
resolve this issue.



Thank You!

Regards,
Pintu

-- 
Masami Hiramatsu [off-list ref]