From: Steven Rostedt <rostedt@goodmis.org> Date: 2021-06-29 13:55:49
From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
All internal use cases for tracepoint_probe_register() is set to not ever
be called with the same function and data. If it is, it is considered a
bug, as that means the accounting of handling tracepoints is corrupted.
If the function and data for a tracepoint is already registered when
tracepoint_probe_register() is called, it will call WARN_ON_ONCE() and
return with EEXISTS.
The BPF system call can end up calling tracepoint_probe_register() with
the same data, which now means that this can trigger the warning because
of a user space process. As WARN_ON_ONCE() should not be called because
user space called a system call with bad data, there needs to be a way to
register a tracepoint without triggering a warning.
Enter tracepoint_probe_register_may_exist(), which can be called, but will
not cause a WARN_ON() if the probe already exists. It will still error out
with EEXIST, which will then be sent to the user space that performed the
BPF system call.
This keeps the previous testing for issues with other users of the
tracepoint code, while letting BPF call it with duplicated data and not
warn about it.
Link: https://lore.kernel.org/lkml/20210626135845.4080-1-penguin-kernel@I-love.SAKURA.ne.jp/
Link: https://syzkaller.appspot.com/bug?id=41f4318cf01762389f4d1c1c459da4f542fe5153 [1]`
Cc: stable@vger.kernel.org
Fixes: c4f6699dfcb85 ("bpf: introduce BPF_RAW_TRACEPOINT")
Reported-by: syzbot <redacted>
Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
include/linux/tracepoint.h | 10 ++++++++++
kernel/trace/bpf_trace.c | 3 ++-
kernel/tracepoint.c | 33 ++++++++++++++++++++++++++++++---
3 files changed, 42 insertions(+), 4 deletions(-)
On Tue, Jun 29, 2021 at 6:55 AM Steven Rostedt [off-list ref] wrote:
From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
All internal use cases for tracepoint_probe_register() is set to not ever
be called with the same function and data. If it is, it is considered a
bug, as that means the accounting of handling tracepoints is corrupted.
If the function and data for a tracepoint is already registered when
tracepoint_probe_register() is called, it will call WARN_ON_ONCE() and
return with EEXISTS.
The BPF system call can end up calling tracepoint_probe_register() with
the same data, which now means that this can trigger the warning because
of a user space process. As WARN_ON_ONCE() should not be called because
user space called a system call with bad data, there needs to be a way to
register a tracepoint without triggering a warning.
Enter tracepoint_probe_register_may_exist(), which can be called, but will
not cause a WARN_ON() if the probe already exists. It will still error out
with EEXIST, which will then be sent to the user space that performed the
BPF system call.
This keeps the previous testing for issues with other users of the
tracepoint code, while letting BPF call it with duplicated data and not
warn about it.
There doesn't seem to be anything conceptually wrong with attaching
the same BPF program twice to the same tracepoint. Is it a hard
requirement to have a unique tp+callback combination, or was it done
mostly to detect an API misuse? How hard is it to support such use
cases?
I was surprised to discover this is not supported (though I never had
a use for this, had to construct a test to see the warning).
There doesn't seem to be anything conceptually wrong with attaching
the same BPF program twice to the same tracepoint. Is it a hard
requirement to have a unique tp+callback combination, or was it done
mostly to detect an API misuse? How hard is it to support such use
cases?
I was surprised to discover this is not supported (though I never had
a use for this, had to construct a test to see the warning).
The callback is identified by the function and its data combination. If
there's two callbacks calling the same function with the same data on
the same tracepoint, one question is, why? And the second is how do you
differentiate the two?
-- Steve
There doesn't seem to be anything conceptually wrong with attaching
the same BPF program twice to the same tracepoint. Is it a hard
requirement to have a unique tp+callback combination, or was it done
mostly to detect an API misuse? How hard is it to support such use
cases?
I was surprised to discover this is not supported (though I never had
a use for this, had to construct a test to see the warning).
The callback is identified by the function and its data combination. If
there's two callbacks calling the same function with the same data on
the same tracepoint, one question is, why? And the second is how do you
differentiate the two?
For places where multiple BPF programs can be attached (kprobes,
cgroup programs, etc), we don't put a restriction that all programs
have to be unique. It's totally legal to have the same program
attached multiple times. So having this for tracepoints will be a
one-off behavior.
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
-- Steve
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
But not for the kprobe case.
And it might not always be possible to know that the same BPF program
is being attached. It could be attached by different processes that
re-use pinned program (without being aware of each other). Or it could
be done from some generic library that just accepts prog_fd and
doesn't really know the exact BPF program and whether it was already
attached.
Not sure why it doesn't matter that attachment will fail where it is
expected to succeed. The question is rather why such restriction?
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
But not for the kprobe case.
What do you mean "not for the kprobe case"? What kprobe case?
You attach the same program twice to the same kprobe? Or do you create
two kprobes at the same location?
And it might not always be possible to know that the same BPF program
is being attached. It could be attached by different processes that
re-use pinned program (without being aware of each other). Or it could
be done from some generic library that just accepts prog_fd and
doesn't really know the exact BPF program and whether it was already
attached.
Not sure why it doesn't matter that attachment will fail where it is
expected to succeed. The question is rather why such restriction?
Why is it expected to succeed? It never did. And why such a
restriction? Because it complicates the code, and there's no good use
case to do so. Why complicate something for little reward?
-- Steve
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
But not for the kprobe case.
And it might not always be possible to know that the same BPF program
is being attached. It could be attached by different processes that
re-use pinned program (without being aware of each other). Or it could
be done from some generic library that just accepts prog_fd and
doesn't really know the exact BPF program and whether it was already
attached.
Not sure why it doesn't matter that attachment will fail where it is
expected to succeed. The question is rather why such restriction?
Before eBPF came to exist, all in-kernel users of the tracepoint API never
required multiple registrations for a given (tracepoint, probe, data) tuple.
This allowed us to expose an API which can consider that the (tracepoint, probe, data)
tuple is unique for each registration/unregistration pair, and therefore use that same
tuple for unregistration. Refusing multiple registrations for a given tuple allows us to
forgo the complexity of reference counting for duplicate registrations, and provide
immediate feedback to misbehaving tracers which have duplicate registration or
unbalanced registration/unregistration pairs.
From the perspective of a ring buffer tracer, the notion of multiple instances of
a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given
tracepoint hit would generate many instances of the exact same event into the
same trace buffer.
AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse
allowed Steven to find a few unbalanced registration/unregistration issues while developing
ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well.
Considering that allowing duplicates would add complexity to the tracepoint code,
what is the use-case justifying allowing many instances of the exact same callback
and data for a given tracepoint ?
One key difference I notice here between eBPF and ring buffer tracers is what eBPF
considers a "program". AFAIU (please let me know if I'm mistaken), the "callback"
argument provided by eBPF to the tracepoint API is a limited set of trampoline routines.
The bulk of the eBPF "program" is provided in the "data" argument. So this means the
"program" is both the eBPF code and some context.
So I understand that a given eBPF code could be loaded more than once for a given
tracepoint, but I would expect that each registration on a given tracepoint be
provided with its own "context", otherwise we end up in a similar situation as the
ring buffer's duplicated events scenario I explained above.
Also, we should discuss whether kprobes might benefit from being more strict by
rejecting duplicated (instrumentation site, probe, data) tuples.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
But not for the kprobe case.
What do you mean "not for the kprobe case"? What kprobe case?
You attach the same program twice to the same kprobe? Or do you create
two kprobes at the same location?
I meant attaching the same BPF program twice to the same kernel
function through the kprobe mechanism (through perf_event_open()
syscall). From user perspective it's one BPF program attached twice to
the same kprobe. Not entirely sure if two perf_event_open() calls will
create two kprobes or re-use one internally.
quoted
And it might not always be possible to know that the same BPF program
is being attached. It could be attached by different processes that
re-use pinned program (without being aware of each other). Or it could
be done from some generic library that just accepts prog_fd and
doesn't really know the exact BPF program and whether it was already
attached.
Not sure why it doesn't matter that attachment will fail where it is
expected to succeed. The question is rather why such restriction?
Why is it expected to succeed? It never did. And why such a
restriction? Because it complicates the code, and there's no good use
case to do so. Why complicate something for little reward?
See above about kprobe for why it was my expectation.
But it was my original question whether this causes some complications
or it's just an attempt to detect API mis-use. Seems like it's the
former, alright.
As for why the user might need that, it's up to the user and I don't
want to speculate because it will always sound contrived without a
specific production use case. But people are very creative and we try
not to dictate how and what can be done if it doesn't break any
fundamental assumption and safety.
I guess it doesn't matter, because if they try to do it, the second
attachment will simply fail to attach.
But not for the kprobe case.
And it might not always be possible to know that the same BPF program
is being attached. It could be attached by different processes that
re-use pinned program (without being aware of each other). Or it could
be done from some generic library that just accepts prog_fd and
doesn't really know the exact BPF program and whether it was already
attached.
Not sure why it doesn't matter that attachment will fail where it is
expected to succeed. The question is rather why such restriction?
Before eBPF came to exist, all in-kernel users of the tracepoint API never
required multiple registrations for a given (tracepoint, probe, data) tuple.
This allowed us to expose an API which can consider that the (tracepoint, probe, data)
tuple is unique for each registration/unregistration pair, and therefore use that same
tuple for unregistration. Refusing multiple registrations for a given tuple allows us to
forgo the complexity of reference counting for duplicate registrations, and provide
immediate feedback to misbehaving tracers which have duplicate registration or
unbalanced registration/unregistration pairs.
From the perspective of a ring buffer tracer, the notion of multiple instances of
a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given
tracepoint hit would generate many instances of the exact same event into the
same trace buffer.
AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse
allowed Steven to find a few unbalanced registration/unregistration issues while developing
ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well.
Considering that allowing duplicates would add complexity to the tracepoint code,
what is the use-case justifying allowing many instances of the exact same callback
and data for a given tracepoint ?
It wasn't clear to me if supporting this would cause any added
complexity, which is why I asked.
One key difference I notice here between eBPF and ring buffer tracers is what eBPF
considers a "program". AFAIU (please let me know if I'm mistaken), the "callback"
argument provided by eBPF to the tracepoint API is a limited set of trampoline routines.
The bulk of the eBPF "program" is provided in the "data" argument. So this means the
"program" is both the eBPF code and some context.
So I understand that a given eBPF code could be loaded more than once for a given
No, it turns out it can't, I was just surprised to learn that.
Surprised, because AFAIK we don't have such restrictions on uniqueness
of attached BPF programs anywhere else where multiple BPF programs are
allowed.
tracepoint, but I would expect that each registration on a given tracepoint be
provided with its own "context", otherwise we end up in a similar situation as the
ring buffer's duplicated events scenario I explained above.
Also, we should discuss whether kprobes might benefit from being more strict by
rejecting duplicated (instrumentation site, probe, data) tuples.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com