Hi Steven, thanks for having a look at this.

On Fri, 2021-05-28 at 13:32 -0400, Steven Rostedt wrote:

On Wed, 19 May 2021 19:57:55 +0200
Nicolas Saenz Julienne [off-list ref] wrote:

quoted

To minimize trace's effect on isolated CPUs. That is, CPUs were only a
handful or a single, process are allowed to run. Introduce a new trace
option: 'poll-rb'.

This option changes the heuristic used to wait for data on trace
buffers. The default one, based on wait queues, will trigger an IPI[1]
on the CPU responsible for new data, which will take care of waking up
the trace gathering process (generally trace-cmd). Whereas with
'poll-rb' we will poll (as in busy-wait) the ring buffers from the trace
gathering process, releasing the CPUs writing trace data from doing any
wakeup work.

This wakeup work, although negligible in the vast majority of workloads,
may cause unwarranted latencies on systems running trace on isolated
CPUs. This is made worse on PREEMPT_RT kernels, as they defer the IPI
handling into a kernel thread, forcing unwarranted context switches on
otherwise extremely busy CPUs.

To illustrate this, tracing with PREEMPT_RT=y on an isolated CPU with a
single process pinned to it (NO_HZ_FULL=y, and plenty more isolation
options enabled). I see:
  - 50-100us latency spikes with the default trace-cmd options
  - 14-10us latency spikes with 'poll-rb'
  - 11-8us latency spikes with no tracing at all

The obvious drawback of 'poll-rb' is putting more pressure on the
housekeeping CPUs. Wasting cycles. Hence the notice in the documentation
discouraging its use in general.

[1] The IPI, in this case, an irq_work, is needed since trace might run
in NMI context. Which is not suitable for wake-ups.

Can't this simply be done in user-space?

Set the reading of the trace buffers to O_NONBLOCK and it wont wait for
buffering to happen, and should prevent it from causing the IPI wake ups.

Yes, I hadn't thought of O_NONBLOCK...

If you need this for trace-cmd, we can add a --poll option that would do
this.

I just sent a patch with my attempt at implementing --poll.

-- 
Nicolás Sáenz