On Thu, Jul 08, 2021 at 03:46:14PM +0000, Matthias Klein wrote:

The configuration is attached.

I've quickly inspected the .config file resulting from the defconfig
above.  You should disable the enabled symbols below:

  - CONFIG_PROVE_LOCKING
  - CONFIG_DEBUG_ALIGN_RODATA
  - CONFIG_DEBUG_FS_ALLOW_ALL
  - CONFIG_DEBUG_MISC
  - CONFIG_DEBUG_PREEMPT
  - CONFIG_DEBUG_RT_MUTEXES
  - CONFIG_DEBUG_SPINLOCK
  - CONFIG_DEBUG_MUTEXES
  - CONFIG_DEBUG_WW_MUTEX_SLOWPATH
  - CONFIG_DEBUG_RWSEMS
  - CONFIG_DEBUG_LOCK_ALLOC
  - CONFIG_DEBUG_USER
  - CONFIG_NO_HZ_COMMON
  - CONFIG_NO_HZ_IDLE
  - CONFIG_NO_HZ

You also have a lot of CPU frequency governors enabled. Disable all of
them except the performance governor and make sure that
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y.

The above, especially disabling CONFIG_PROVE_LOCKING and
CONFIG_DEBUG_PREEMPT, should really fix your problem.

If you still have issues, play around with disabling CONFIG_CPU_IDLE and
its governors. Note though that CPU manufacturers sometimes advise
against doing this.

If the latency is still higher than desired, see if CONFIG_DRM and high
GPU workloads negatively affects it . On some architectures, latency can
increase by 100 microseconds or more due to sharing of the Last Level
Cache and memory bandwidth between the GPU and the CPU.

After all of this, if latency is still higher than the 100->150
microseconds range, it's possible that the custom code you've added is
doing shady stuff; e.g., manually disabling interrupts or preemption.

If it aint so, and the latency is still higher than the expected range
(even with a pure RT release), then you'll need to manually trace the
kernel and catch the offending call chain.

Good luck,

--
Ahmed S. Darwish
Linutronix GmbH