On 4/21/21 5:57 AM, Jinpu Wang wrote:

On Wed, Apr 21, 2021 at 1:55 PM Pavel Begunkov [off-list ref] wrote:

quoted

On 4/21/21 12:50 PM, Jinpu Wang wrote:
[snip]

quoted

quoted

quoted

Hi Jens,
The problem with using blktrace at production may cause a performance
drop ~30%. while with the block stats here, we only see ~3% when
enabled.

It's probably was asked before, but let's refresh as the discussion
erupted again.

I get your problem with blktrace(8), IIRC it definitely can deteriorate
performance if run constantly, but did you try to write a bpf program
that does smarter accumulation in the kernel? Like making bpf to collect
a latency table (right as in your patches do) and flushing it to the
disk periodically?

Hi Pavel,

Thanks for the suggestion.

We did test with ebpf with kprobe in the past (~kernel 4.4/4.14), we
saw 10% performance drop, that's the reason we develop this
stats patches.

But I just did another test with bpftrace on k 5.10.30, I do not see
performance lost.
It must be ebpf is improving very much since then.

So to summarize, we can use bpftrace to do the drop in latest kernel,
there is no need to have it build into the kernel.

Perfect, and I'm sure it will be even more convenient for you, for
instance to gather other stats or do it somehow differently

Yeah, agree.
Thanks again!

Perfect, thanks! I'll drop the series.

-- 
Jens Axboe