* Brian Silverman | 2016-05-16 14:05:54 [-0700]:

Thank you for the explanation.

It happens when realtime (SCHED_RR/SCHED_FIFO) processes which have
called mlockall(MCL_CURRENT | MCL_FUTURE) exit.

Why can't the application drop the RT priority before its exit? Wouldn't
that be appropriate?

One of them:
queue_cyclictes-5298  [000] ....1..   290.366030: remove_vma <-exit_mmap
queue_cyclictes-5298  [000] ....1..   290.366032: __might_sleep <-remove_vma
queue_cyclictes-5298  [000] ....1..   290.366033: ___might_sleep
<-__might_sleep
queue_cyclictes-5298  [000] ....1..   290.366035: kmem_cache_free <-remove_vma
queue_cyclictes-5298  [000] ....1..   290.366036: preempt_count_add
<-kmem_cache_free
queue_cyclictes-5298  [000] ....2..   290.366037: preempt_count_sub
<-kmem_cache_free
queue_cyclictes-5298  [000] ....1..   290.366039: __slab_free <-kmem_cache_free
queue_cyclictes-5298  [000] d...1..   290.366040: preempt_count_add
<-__slab_free
queue_cyclictes-5298  [000] d...2..   290.366041: preempt_count_sub
<-__slab_free
queue_cyclictes-5298  [000] ....1..   290.366043: kmem_cache_free:
call_site=c0147054 ptr=ebb93cb8

In this trace and the others you attached, you have a preempt count of
at least one. Is this a function trace? For events I recently figure out
that the counter in event tracing is recorded wrongly [0]. So my
question here is basically is this a long preempt disable section or
not.

[0] http://permalink.gmane.org/gmane.linux.kernel/2227774

Sebastian