Add some missing glue logic to teach bpf about bare tracepoints - tracepoints
without any trace event associated with them.
Bare tracepoints are declare with DECLARE_TRACE(). Full tracepoints are declare
with TRACE_EVENT().
BPF can attach to these tracepoints as RAW_TRACEPOINT() only as there's no
events in tracefs created with them.
Qais Yousef (2):
trace: bpf: Allow bpf to attach to bare tracepoints
selftests: bpf: Add a new test for bare tracepoints
Documentation/bpf/bpf_design_QA.rst | 6 ++++++
include/trace/bpf_probe.h | 12 ++++++++++--
.../selftests/bpf/bpf_testmod/bpf_testmod-events.h | 6 ++++++
.../testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 2 ++
.../testing/selftests/bpf/prog_tests/module_attach.c | 1 +
.../testing/selftests/bpf/progs/test_module_attach.c | 10 ++++++++++
6 files changed, 35 insertions(+), 2 deletions(-)
--
2.25.1
Some subsystems only have bare tracepoints (a tracepoint with no
associated trace event) to avoid the problem of trace events being an
ABI that can't be changed.
From bpf presepective, bare tracepoints are what it calls
RAW_TRACEPOINT().
Since bpf assumed there's 1:1 mapping, it relied on hooking to
DEFINE_EVENT() macro to create bpf mapping of the tracepoints. Since
bare tracepoints use DECLARE_TRACE() to create the tracepoint, bpf had
no knowledge about their existence.
By teaching bpf_probe.h to parse DECLARE_TRACE() in a similar fashion to
DEFINE_EVENT(), bpf can find and attach to the new raw tracepoints.
Enabling that comes with the contract that changes to raw tracepoints
don't constitute a regression if they break existing bpf programs.
We need the ability to continue to morph and modify these raw
tracepoints without worrying about any ABI.
Update Documentation/bpf/bpf_design_QA.rst to document this contract.
Signed-off-by: Qais Yousef <redacted>
---
Documentation/bpf/bpf_design_QA.rst | 6 ++++++
include/trace/bpf_probe.h | 12 ++++++++++--
2 files changed, 16 insertions(+), 2 deletions(-)
@@ -208,6 +208,12 @@ data structures and compile with kernel internal headers. Both of these kernel internals are subject to change and can break with newer kernels such that the program needs to be adapted accordingly.+Q: Are tracepoints part of the stable ABI?+------------------------------------------+A: NO. Tracepoints are tied to internal implementation details hence they are+subject to change and can break with newer kernels. BPF programs need to change+accordingly when this happens.+ Q: How much stack space a BPF program uses? ------------------------------------------- A: Currently all program types are limited to 512 bytes of stack
@@ -55,8 +55,7 @@/* tracepoints with more than 12 arguments will hit build error */#define CAST_TO_U64(...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__)-#undef DECLARE_EVENT_CLASS-#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \+#define __BPF_DECLARE_TRACE(call, proto, args) \staticnotracevoid\__bpf_trace_##call(void*__data,proto)\{\
Reuse module_attach infrastructure to add a new bare tracepoint to check
we can attach to it as a raw tracepoint.
Signed-off-by: Qais Yousef <redacted>
---
Andrii
I was getting the error below when I was trying to run the test.
I had to comment out all related fentry* code to be able to test the raw_tp
stuff. Not sure something I've done wrong or it's broken for some reason.
I was on v5.11-rc2.
$ sudo ./test_progs -v -t module_attach
bpf_testmod.ko is already unloaded.
Loading bpf_testmod.ko...
Successfully loaded bpf_testmod.ko.
test_module_attach:PASS:skel_open 0 nsec
test_module_attach:PASS:set_attach_target 0 nsec
test_module_attach:PASS:skel_load 0 nsec
libbpf: prog 'handle_fentry': failed to attach: ERROR: strerror_r(-524)=22
libbpf: failed to auto-attach program 'handle_fentry': -524
test_module_attach:FAIL:skel_attach skeleton attach failed: -524
#58 module_attach:FAIL
Successfully unloaded bpf_testmod.ko.
Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED
.../selftests/bpf/bpf_testmod/bpf_testmod-events.h | 6 ++++++
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 2 ++
tools/testing/selftests/bpf/prog_tests/module_attach.c | 1 +
tools/testing/selftests/bpf/progs/test_module_attach.c | 10 ++++++++++
4 files changed, 19 insertions(+)
@@ -28,6 +28,12 @@ TRACE_EVENT(bpf_testmod_test_read,__entry->pid,__entry->comm,__entry->off,__entry->len));+/* A bare tracepoint with no event associated with it */+DECLARE_TRACE(bpf_testmod_test_read_bare,+TP_PROTO(structtask_struct*task,structbpf_testmod_test_read_ctx*ctx),+TP_ARGS(task,ctx)+);+#endif /* _BPF_TESTMOD_EVENTS_H */#undef TRACE_INCLUDE_PATH
On Mon, Jan 11, 2021 at 10:20 AM Qais Yousef [off-list ref] wrote:
Reuse module_attach infrastructure to add a new bare tracepoint to check
we can attach to it as a raw tracepoint.
Signed-off-by: Qais Yousef <redacted>
---
Andrii
I was getting the error below when I was trying to run the test.
I had to comment out all related fentry* code to be able to test the raw_tp
stuff. Not sure something I've done wrong or it's broken for some reason.
I was on v5.11-rc2.
Check that you have all the required Kconfig options from
tools/testing/selftests/bpf/config. And also you will need to build
pahole from master, 1.19 doesn't have some fixes that add kernel
module support. I think pahole is the reasons why you have the failure
below.
$ sudo ./test_progs -v -t module_attach
use -vv when debugging stuff like that with test_progs, it will output
libbpf detailed logs, that often are very helpful
@@ -28,6 +28,12 @@ TRACE_EVENT(bpf_testmod_test_read,__entry->pid,__entry->comm,__entry->off,__entry->len));+/* A bare tracepoint with no event associated with it */+DECLARE_TRACE(bpf_testmod_test_read_bare,+TP_PROTO(structtask_struct*task,structbpf_testmod_test_read_ctx*ctx),+TP_ARGS(task,ctx)+);+#endif /* _BPF_TESTMOD_EVENTS_H */#undef TRACE_INCLUDE_PATH
It's kind of boring to have two read tracepoints :) Do you mind adding
a write tracepoint and use bare tracepoint there? You won't need this
ctx.len++ hack as well. Feel free to add identical
bpf_testmod_test_write_ctx (renaming it is more of a pain).
On Mon, Jan 11, 2021 at 10:20 AM Qais Yousef [off-list ref] wrote:
quoted
Reuse module_attach infrastructure to add a new bare tracepoint to check
we can attach to it as a raw tracepoint.
Signed-off-by: Qais Yousef <redacted>
---
Andrii
I was getting the error below when I was trying to run the test.
I had to comment out all related fentry* code to be able to test the raw_tp
stuff. Not sure something I've done wrong or it's broken for some reason.
I was on v5.11-rc2.
Check that you have all the required Kconfig options from
tools/testing/selftests/bpf/config. And also you will need to build
Yep I have merged this config snippet using merge_config.sh script.
pahole from master, 1.19 doesn't have some fixes that add kernel
module support. I think pahole is the reasons why you have the failure
below.
I am using pahole 1.19. I have built it from tip of master though.
/trying using v1.19 tag
Still fails the same.
quoted
$ sudo ./test_progs -v -t module_attach
use -vv when debugging stuff like that with test_progs, it will output
libbpf detailed logs, that often are very helpful
Sorry about that. I did a last minute change because of checkpatch.pl error and
it seems I either forgot to rebuild or missed that the rebuild failed :/
@@ -28,6 +28,12 @@ TRACE_EVENT(bpf_testmod_test_read,__entry->pid,__entry->comm,__entry->off,__entry->len));+/* A bare tracepoint with no event associated with it */+DECLARE_TRACE(bpf_testmod_test_read_bare,+TP_PROTO(structtask_struct*task,structbpf_testmod_test_read_ctx*ctx),+TP_ARGS(task,ctx)+);+#endif /* _BPF_TESTMOD_EVENTS_H */#undef TRACE_INCLUDE_PATH
It's kind of boring to have two read tracepoints :) Do you mind adding
Hehe boring is good :p
a write tracepoint and use bare tracepoint there? You won't need this
ctx.len++ hack as well. Feel free to add identical
bpf_testmod_test_write_ctx (renaming it is more of a pain).
It was easy to get this done. So I think it should be easy to make it a write
too :)
Thanks
--
Qais Yousef
From: Yonghong Song <hidden> Date: 2021-01-12 21:56:30
On 1/11/21 10:20 AM, Qais Yousef wrote:
quoted hunk
Some subsystems only have bare tracepoints (a tracepoint with no
associated trace event) to avoid the problem of trace events being an
ABI that can't be changed.
From bpf presepective, bare tracepoints are what it calls
RAW_TRACEPOINT().
Since bpf assumed there's 1:1 mapping, it relied on hooking to
DEFINE_EVENT() macro to create bpf mapping of the tracepoints. Since
bare tracepoints use DECLARE_TRACE() to create the tracepoint, bpf had
no knowledge about their existence.
By teaching bpf_probe.h to parse DECLARE_TRACE() in a similar fashion to
DEFINE_EVENT(), bpf can find and attach to the new raw tracepoints.
Enabling that comes with the contract that changes to raw tracepoints
don't constitute a regression if they break existing bpf programs.
We need the ability to continue to morph and modify these raw
tracepoints without worrying about any ABI.
Update Documentation/bpf/bpf_design_QA.rst to document this contract.
Signed-off-by: Qais Yousef <redacted>
---
Documentation/bpf/bpf_design_QA.rst | 6 ++++++
include/trace/bpf_probe.h | 12 ++++++++++--
2 files changed, 16 insertions(+), 2 deletions(-)
@@ -208,6 +208,12 @@ data structures and compile with kernel internal headers. Both of these kernel internals are subject to change and can break with newer kernels such that the program needs to be adapted accordingly.+Q: Are tracepoints part of the stable ABI?+------------------------------------------+A: NO. Tracepoints are tied to internal implementation details hence they are+subject to change and can break with newer kernels. BPF programs need to change+accordingly when this happens.+ Q: How much stack space a BPF program uses? ------------------------------------------- A: Currently all program types are limited to 512 bytes of stack
@@ -55,8 +55,7 @@/* tracepoints with more than 12 arguments will hit build error */#define CAST_TO_U64(...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__)-#undef DECLARE_EVENT_CLASS-#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \+#define __BPF_DECLARE_TRACE(call, proto, args) \staticnotracevoid\__bpf_trace_##call(void*__data,proto)\{\
@@ -111,6 +114,11 @@ __DEFINE_EVENT(template, call, PARAMS(proto), PARAMS(args), size)#define DEFINE_EVENT_PRINT(template, name, proto, args, print) \DEFINE_EVENT(template,name,PARAMS(proto),PARAMS(args))+#undef DECLARE_TRACE+#define DECLARE_TRACE(call, proto, args) \+(__BPF_DECLARE_TRACE(call,PARAMS(proto),PARAMS(args))\+__DEFINE_EVENT(call,call,PARAMS(proto),PARAMS(args),0))
I applied the patch to my local bpf-next repo, and got the following
compilation error:
In file included from
/data/users/yhs/work/net-next/include/trace/define_trace.h:104,
from
/data/users/yhs/work/net-next/include/trace/events/sched.h:740,
from
/data/users/yhs/work/net-next/kernel/sched/core.c:10:
/data/users/yhs/work/net-next/include/trace/bpf_probe.h:59:1: error:
expected identifier or ‘(’ before ‘static’
static notrace void \
^~~~~~
/data/users/yhs/work/net-next/include/trace/bpf_probe.h:119:3: note: in
expansion of macro ‘__BPF_DECLARE_TRACE’
(__BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \
^~~~~~~~~~~~~~~~~~~
/data/users/yhs/work/net-next/include/trace/events/sched.h:693:1: note:
in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(pelt_cfs_tp,
^~~~~~~~~~~~~
/data/users/yhs/work/net-next/include/trace/bpf_probe.h:59:1: error:
expected identifier or ‘(’ before ‘static’
static notrace void \
^~~~~~
/data/users/yhs/work/net-next/include/trace/bpf_probe.h:119:3: note: in
expansion of macro ‘__BPF_DECLARE_TRACE’
(__BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \
^~~~~~~~~~~~~~~~~~~
/data/users/yhs/work/net-next/include/trace/events/sched.h:697:1: note:
in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(pelt_rt_tp,
^~~~~~~~~~~~~
/data/users/yhs/work/net-next/include/trace/bpf_probe.h:59:1: error:
expected identifier or ‘(’ before ‘static’
static notrace void \
I dumped preprecessor result but after macro expansion, the code
becomes really complex and I have not figured out why it failed.
Do you know what is the possible reason?
On Tue, Jan 12, 2021 at 11:27 AM Qais Yousef [off-list ref] wrote:
On 01/11/21 23:26, Andrii Nakryiko wrote:
quoted
On Mon, Jan 11, 2021 at 10:20 AM Qais Yousef [off-list ref] wrote:
quoted
Reuse module_attach infrastructure to add a new bare tracepoint to check
we can attach to it as a raw tracepoint.
Signed-off-by: Qais Yousef <redacted>
---
Andrii
I was getting the error below when I was trying to run the test.
I had to comment out all related fentry* code to be able to test the raw_tp
stuff. Not sure something I've done wrong or it's broken for some reason.
I was on v5.11-rc2.
Check that you have all the required Kconfig options from
tools/testing/selftests/bpf/config. And also you will need to build
Yep I have merged this config snippet using merge_config.sh script.
quoted
pahole from master, 1.19 doesn't have some fixes that add kernel
module support. I think pahole is the reasons why you have the failure
below.
I am using pahole 1.19. I have built it from tip of master though.
/trying using v1.19 tag
Still fails the same.
quoted
quoted
$ sudo ./test_progs -v -t module_attach
use -vv when debugging stuff like that with test_progs, it will output
libbpf detailed logs, that often are very helpful
It did help a bit for me to make sure that you have bpf_testmod
properly loaded and its BTF was found, so the problem is somewhere
else. Also, given load succeeded and attach failed with OPNOTSUPP, I
suspect you are missing some of FTRACE configs, which seems to be
missing from selftests's config as well. Check that you have
CONFIG_FTRACE=y and CONFIG_DYNAMIC_FTRACE=y, and you might need some
more. See [0] for a real config we are using to run all tests in
libbpf CI. If you figure out what you were missing, please also
contribute a patch to selftests' config.
[0] https://github.com/libbpf/libbpf/blob/master/travis-ci/vmtest/configs/latest.config
Sorry about that. I did a last minute change because of checkpatch.pl error and
it seems I either forgot to rebuild or missed that the rebuild failed :/
no worries, just fix and re-submit. Good that we have CI that caught
this early on.
@@ -28,6 +28,12 @@ TRACE_EVENT(bpf_testmod_test_read,__entry->pid,__entry->comm,__entry->off,__entry->len));+/* A bare tracepoint with no event associated with it */+DECLARE_TRACE(bpf_testmod_test_read_bare,+TP_PROTO(structtask_struct*task,structbpf_testmod_test_read_ctx*ctx),+TP_ARGS(task,ctx)+);+#endif /* _BPF_TESTMOD_EVENTS_H */#undef TRACE_INCLUDE_PATH
It's kind of boring to have two read tracepoints :) Do you mind adding
Hehe boring is good :p
quoted
a write tracepoint and use bare tracepoint there? You won't need this
ctx.len++ hack as well. Feel free to add identical
bpf_testmod_test_write_ctx (renaming it is more of a pain).
It was easy to get this done. So I think it should be easy to make it a write
too :)
yep, having two tracepoints allow more flexibility over longer term,
so I think it's good to do (regardless of boring or not ;) )
I applied the patch to my local bpf-next repo, and got the following
compilation error:
[...]
I dumped preprecessor result but after macro expansion, the code
becomes really complex and I have not figured out why it failed.
Do you know what is the possible reason?
Yeah I did a last minute fix to address a checkpatch.pl error and my
verification of the change wasn't good enough obviously.
If you're keen to try out I can send you a patch with the fix. I should send v2
by the weekend too.
Thanks for having a look.
Cheers
--
Qais Yousef
It did help a bit for me to make sure that you have bpf_testmod
properly loaded and its BTF was found, so the problem is somewhere
else. Also, given load succeeded and attach failed with OPNOTSUPP, I
suspect you are missing some of FTRACE configs, which seems to be
missing from selftests's config as well. Check that you have
CONFIG_FTRACE=y and CONFIG_DYNAMIC_FTRACE=y, and you might need some
more. See [0] for a real config we are using to run all tests in
libbpf CI. If you figure out what you were missing, please also
contribute a patch to selftests' config.
[0] https://github.com/libbpf/libbpf/blob/master/travis-ci/vmtest/configs/latest.config
Yeah that occurred to me too. I do have all necessary FTRACE options enabled,
including DYNAMIC_FTRACE. I think I did try enabling fault injection too just
in case. I have CONFIG_FAULT_INJECTION=y and CONFIG_FUNCTION_ERROR_INJECTION=y.
I will look at the CI config and see if I can figure it out.
I will likely get a chance to look at all of this and send v2 over the
weekend.
Thanks
--
Qais Yousef
From: Yonghong Song <hidden> Date: 2021-01-13 16:08:10
On 1/13/21 2:16 AM, Qais Yousef wrote:
On 01/12/21 12:19, Yonghong Song wrote:
quoted
I applied the patch to my local bpf-next repo, and got the following
compilation error:
[...]
quoted
I dumped preprecessor result but after macro expansion, the code
becomes really complex and I have not figured out why it failed.
Do you know what is the possible reason?
Yeah I did a last minute fix to address a checkpatch.pl error and my
verification of the change wasn't good enough obviously.
If you're keen to try out I can send you a patch with the fix. I should send v2
by the weekend too.
Thanks. I can wait and will check v2 once it is available.
It did help a bit for me to make sure that you have bpf_testmod
properly loaded and its BTF was found, so the problem is somewhere
else. Also, given load succeeded and attach failed with OPNOTSUPP, I
suspect you are missing some of FTRACE configs, which seems to be
missing from selftests's config as well. Check that you have
CONFIG_FTRACE=y and CONFIG_DYNAMIC_FTRACE=y, and you might need some
more. See [0] for a real config we are using to run all tests in
libbpf CI. If you figure out what you were missing, please also
contribute a patch to selftests' config.
[0] https://github.com/libbpf/libbpf/blob/master/travis-ci/vmtest/configs/latest.config
Yeah that occurred to me too. I do have all necessary FTRACE options enabled,
including DYNAMIC_FTRACE. I think I did try enabling fault injection too just
in case. I have CONFIG_FAULT_INJECTION=y and CONFIG_FUNCTION_ERROR_INJECTION=y.
Could it come from lack of fentry support on arm64 (or are you testing on
x86?) Since the arm64 JIT doesn't have trampoline support at the moment, a
lot of bpf selftests fail with ENOTSUPP.
Thanks,
Jean
It did help a bit for me to make sure that you have bpf_testmod
properly loaded and its BTF was found, so the problem is somewhere
else. Also, given load succeeded and attach failed with OPNOTSUPP, I
suspect you are missing some of FTRACE configs, which seems to be
missing from selftests's config as well. Check that you have
CONFIG_FTRACE=y and CONFIG_DYNAMIC_FTRACE=y, and you might need some
more. See [0] for a real config we are using to run all tests in
libbpf CI. If you figure out what you were missing, please also
contribute a patch to selftests' config.
[0] https://github.com/libbpf/libbpf/blob/master/travis-ci/vmtest/configs/latest.config
Yeah that occurred to me too. I do have all necessary FTRACE options enabled,
including DYNAMIC_FTRACE. I think I did try enabling fault injection too just
in case. I have CONFIG_FAULT_INJECTION=y and CONFIG_FUNCTION_ERROR_INJECTION=y.
Could it come from lack of fentry support on arm64 (or are you testing on
x86?) Since the arm64 JIT doesn't have trampoline support at the moment, a
lot of bpf selftests fail with ENOTSUPP.
I am on arm64. I honestly have no clue about this. I'll try to dig out.
Thanks
--
Qais Yousef