On Fri, Mar 01, 2024 at 09:01:07AM -0800, Alexei Starovoitov wrote:

On Fri, Mar 1, 2024 at 12:18 AM Jiri Olsa [off-list ref] wrote:

quoted

On Thu, Feb 29, 2024 at 04:25:17PM -0800, Andrii Nakryiko wrote:

quoted

On Thu, Feb 29, 2024 at 6:39 AM Jiri Olsa [off-list ref] wrote:

quoted

One of uprobe pain points is having slow execution that involves
two traps in worst case scenario or single trap if the original
instruction can be emulated. For return uprobes there's one extra
trap on top of that.

My current idea on how to make this faster is to follow the optimized
kprobes and replace the normal uprobe trap instruction with jump to
user space trampoline that:

  - executes syscall to call uprobe consumers callbacks

Did you get a chance to measure relative performance of syscall vs
int3 interrupt handling? If not, do you think you'll be able to get
some numbers by the time the conference starts? This should inform the
decision whether it even makes sense to go through all the trouble.

right, will do that

I believe Yusheng measured syscall vs uprobe performance
difference during LPC. iirc it was something like 3x.
Certainly necessary to have a benchmark.
selftests/bpf/bench has one for uprobe.
Probably should extend with sys_bpf.

ok, did not know there was uprobe benchmark, will check

Regarding:

quoted

replace the normal uprobe trap instruction with jump to

user space trampoline

it should probably be a call to trampoline instead of a jump.
Unless you plan to generate a different trampoline for every location ?

I wanted to store the ip of the uprobe as argument for the syscall,
but the call instruction will push return address on stack and we
can use it to get uprobe's address.. great

Also how would you pick a space for a trampoline in the target process ?
Analyze /proc/pid/maps and look for gaps in executable sections?

As Andrii mentioned in other response there's already one page mapped
as '[uprobes]' mapping, it's used as trampoline for return uprobes
(contains just int3 instruction) and as buffers to hold the original
instruction for the single step execution

I think if we endup with just single trampoline we can just use some
of the space from that page, our trampoline should not be big

We can start simple with a USDT that uses nop5 instead of nop1
and explicit single trampoline for all USDT locations
that saves all (callee and caller saved) registers and
then does sys_bpf with a new cmd.

ah, I did not realize USDTs are like that, will check, good idea

To replace nop5 with a call to trampoline we can use text_poke_bp
approach: replace 1st byte with int3, replace 2-5 with target addr,
replace 1st byte to make an actual call insn.

I'm bit in the dark in here, but uprobe_write_opcode stores the int3
byte by allocating new page, copying the contents of the old page over
and updating it with int3 byte.. then calls __replace_page to put new
page in place

should that be enough also for 5 bytes update? the cpu executing that
exact page will page fault and get the new updated page? I discussed
with Oleg and got this understanding, I might be wrong

hm what if the cpu is just executing the address in the middle of the
uprobe's original instructions and the page gets updated.. I need to
check more on this ;-)

Once patched there will be no simulation of insns or kernel traps.
Just normal user code that calls into trampoline, that calls sys_bpf,
and returns back.

I saw this as generic uprobe enhancement, should it be sys_bpf syscall,
not a some generic one? we will call all the uprobe's handlers/consumers

thanks,
jirka