From: Thomas Gleixner <hidden> Date: 2017-02-16 08:37:10
On Thu, 16 Feb 2017, Mike Galbraith wrote:
4.9.10-rt6-virgin on 72 core +SMT box.
Below is 1 line per minute, box idling along daintily nibbling, I fire
up a parallel kbuild loop at 40465, and box gobbles greedily.
I have entries bumped to 128k, and chain bits to 18 so box will get
booted and run for a while before lockdep says "I quit". With stock
settings, this box will barely get booted. Seems the bigger the box,
the sooner you're gonna run out. A NOPREEMPT kernel seems to nibble
entries too, but nowhere remotely near as greedily as RT.
Right. RT adds a bunch of locks through the local lock mechanism.
With stacktrace on, buffer contains 1010 __lru_cache_add+0x4f...
(gdb) list *__lru_cache_add+0x4f
0xffffffff811dca9f is in __lru_cache_add (./include/linux/locallock.h:59).
54
55 static inline void __local_lock(struct local_irq_lock *lv)
56 {
57 if (lv->owner != current) {
58 spin_lock_local(&lv->lock);
59 LL_WARN(lv->owner);
60 LL_WARN(lv->nestcnt);
61 lv->owner = current;
62 }
63 lv->nestcnt++;
...which seems to be this.
0xffffffff811dca80 is in __lru_cache_add (mm/swap.c:397).
392 }
393 EXPORT_SYMBOL(mark_page_accessed);
394
395 static void __lru_cache_add(struct page *page)
396 {
397 struct pagevec *pvec = &get_locked_var(swapvec_lock, lru_add_pvec);
398
399 get_page(page);
400 if (!pagevec_add(pvec, page) || PageCompound(page))
401 __pagevec_lru_add(pvec);
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Thanks,
tglx
From: Mike Galbraith <hidden> Date: 2017-02-16 08:50:11
On Thu, 2017-02-16 at 09:37 +0100, Thomas Gleixner wrote:
On Thu, 16 Feb 2017, Mike Galbraith wrote:
...
quoted
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Adding explicit local_irq_lock_init() makes things heaps better, so
presumably we need better lockdep-foo in DEFINE_LOCAL_IRQ_LOCK().
-Mike
From: Thomas Gleixner <hidden> Date: 2017-02-16 09:01:22
On Thu, 16 Feb 2017, Mike Galbraith wrote:
On Thu, 2017-02-16 at 09:37 +0100, Thomas Gleixner wrote:
quoted
On Thu, 16 Feb 2017, Mike Galbraith wrote:
...
quoted
quoted
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Adding explicit local_irq_lock_init() makes things heaps better, so
presumably we need better lockdep-foo in DEFINE_LOCAL_IRQ_LOCK().
From: Mike Galbraith <hidden> Date: 2017-02-16 09:27:17
On Thu, 2017-02-16 at 10:01 +0100, Thomas Gleixner wrote:
On Thu, 16 Feb 2017, Mike Galbraith wrote:
quoted
On Thu, 2017-02-16 at 09:37 +0100, Thomas Gleixner wrote:
quoted
On Thu, 16 Feb 2017, Mike Galbraith wrote:
...
quoted
quoted
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Adding explicit local_irq_lock_init() makes things heaps better, so
presumably we need better lockdep-foo in DEFINE_LOCAL_IRQ_LOCK().
Bah.
Hm, "bah" sounds kinda like it might be a synonym for -EDUMMY :) Fair
enough, I know spit about about lockdep, so that's likely the case, but
the below has me down to ~17k (and climbing, but not as fast).
berio:/sys/kernel/debug/tracing/:[0]# grep -A 1 'stack trace' trace|grep '=>'|sort|uniq
=> ___slab_alloc+0x171/0x5c0
=> __percpu_counter_add+0x56/0xd0
=> __schedule+0xb0/0x7b0
=> __slab_free+0xd8/0x200
=> cgroup_idr_alloc.constprop.39+0x37/0x80
=> hrtimer_start_range_ns+0xe6/0x400
=> idr_preload+0x6c/0x300
=> jbd2_journal_extend+0x4c/0x310 [jbd2]
=> lock_hrtimer_base.isra.28+0x29/0x50
=> rcu_note_context_switch+0x2b8/0x5c0
=> rcu_report_unblock_qs_rnp+0x6e/0xa0
=> rt_mutex_slowunlock+0x25/0xc0
=> rt_spin_lock_slowlock+0x52/0x330
=> rt_spin_lock_slowlock+0x94/0x330
=> rt_spin_lock_slowunlock+0x3c/0xc0
=> swake_up+0x21/0x40
=> task_blocks_on_rt_mutex+0x42/0x1e0
=> try_to_wake_up+0x2d/0x920
berio:/sys/kernel/debug/tracing/:[0]# grep nr_list_entries: trace|tail -1
irq/66-eth2-TxR-3670 [115] d....14 1542.321173: add_lock_to_list.isra.24.constprop.42+0x20/0x100: nr_list_entries: 17839
Got rid of the really pesky growth anyway.
--- a/kernel/workqueue.c+++ b/kernel/workqueue.c
@@ -5522,6 +5522,7 @@ static int __init init_workqueues(void)pwq_cache=KMEM_CACHE(pool_workqueue,SLAB_PANIC);+local_irq_lock_init(pendingb_lock);wq_numa_init();/* initialize CPU pools */---a/lib/radix-tree.c+++b/lib/radix-tree.c
From: Peter Zijlstra <peterz@infradead.org> Date: 2017-02-16 11:06:27
On Thu, Feb 16, 2017 at 10:01:18AM +0100, Thomas Gleixner wrote:
On Thu, 16 Feb 2017, Mike Galbraith wrote:
quoted
On Thu, 2017-02-16 at 09:37 +0100, Thomas Gleixner wrote:
quoted
On Thu, 16 Feb 2017, Mike Galbraith wrote:
...
quoted
quoted
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Adding explicit local_irq_lock_init() makes things heaps better, so
presumably we need better lockdep-foo in DEFINE_LOCAL_IRQ_LOCK().
From: Mike Galbraith <hidden> Date: 2017-02-16 14:43:39
On Thu, 2017-02-16 at 12:06 +0100, Peter Zijlstra wrote:
On Thu, Feb 16, 2017 at 10:01:18AM +0100, Thomas Gleixner wrote:
quoted
On Thu, 16 Feb 2017, Mike Galbraith wrote:
quoted
On Thu, 2017-02-16 at 09:37 +0100, Thomas Gleixner wrote:
quoted
On Thu, 16 Feb 2017, Mike Galbraith wrote:
...
quoted
quoted
swapvec_lock? Oodles of 'em? Nope.
Well, it's a per cpu lock and the lru_cache_add() variants might be called
from a gazillion of different call chains, but yes, it does not make a lot
of sense. We'll have a look.
Adding explicit local_irq_lock_init() makes things heaps better, so
presumably we need better lockdep-foo in DEFINE_LOCAL_IRQ_LOCK().
Weeell, I'm trying to cobble something kinda like that together using
__RT_SPIN_INITIALIZER() instead, but seems mean ole Mr. Compiler NAKs
the PER_CPU_DEP_MAP_INIT() thingy.
CC mm/swap.o
mm/swap.c:54:689: error: braced-group within expression allowed only
inside a function
-Mike
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Date: 2017-02-16 14:54:06
On 2017-02-16 15:42:59 [+0100], Mike Galbraith wrote:
Weeell, I'm trying to cobble something kinda like that together using
__RT_SPIN_INITIALIZER() instead, but seems mean ole Mr. Compiler NAKs
the PER_CPU_DEP_MAP_INIT() thingy.
CC mm/swap.o
mm/swap.c:54:689: error: braced-group within expression allowed only
inside a function
so this is what I have now. I need to get the `static' symbol working
again and PER_CPU_DEP_MAP_INIT but aside from that it seems to do its
job.
From: Mike Galbraith <hidden> Date: 2017-02-16 18:07:08
On Thu, 2017-02-16 at 15:53 +0100, Sebastian Andrzej Siewior wrote:
On 2017-02-16 15:42:59 [+0100], Mike Galbraith wrote:
quoted
Weeell, I'm trying to cobble something kinda like that together using
__RT_SPIN_INITIALIZER() instead, but seems mean ole Mr. Compiler NAKs
the PER_CPU_DEP_MAP_INIT() thingy.
CC mm/swap.o
mm/swap.c:54:689: error: braced-group within expression allowed only
inside a function
so this is what I have now. I need to get the `static' symbol working
again and PER_CPU_DEP_MAP_INIT but aside from that it seems to do its
job.
...
Yeah, works, I should be able to do an ltp run with stock lockdep
settings without it taking it's toys and going home in a snit.
berio:/sys/kernel/debug/tracing/:[0]# !while
while sleep 60; do tail -1 trace; done
<...>-10315 [064] d...1.. 226.953935: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14223
w-13148 [120] d...111 287.414978: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14465
w-16492 [089] d...111 347.128742: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14653
(starts kbuild loop)
btrfs-transacti-1964 [016] d...1.. 411.101549: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 17011
<...>-100268 [127] d...112 472.271769: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18153
w-18864 [011] d...1.. 534.386443: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18543
<...>-50390 [035] dN..2.. 597.794164: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18765
<...>-80098 [127] d...111 659.912145: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18977
checkproc-11123 [017] d...1.. 721.483463: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19247
<idle>-0 [055] d..h5.. 782.685953: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19383
<...>-93632 [055] d...111 835.527817: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19441
From: Mike Galbraith <hidden> Date: 2017-02-16 18:17:12
BTW, this ain't gone. I'll take a peek. It doesn't happen in my tree,
seems likely to be because whether running sirqs fully threaded or not,
I don't let one any thread handle what another exists to handle.
[ 638.107293] NOHZ: local_softirq_pending 80
[ 939.729684] NOHZ: local_softirq_pending 80
[ 945.600869] NOHZ: local_softirq_pending 80
[ 1387.101178] NOHZ: local_softirq_pending 80
[ 1387.101343] NOHZ: local_softirq_pending 80
[ 1387.101549] NOHZ: local_softirq_pending 80
[ 1413.313212] NOHZ: local_softirq_pending 80
[ 1413.313305] NOHZ: local_softirq_pending 80
[ 1413.313347] NOHZ: local_softirq_pending 80
From: Mike Galbraith <hidden> Date: 2017-02-17 20:56:28
On Thu, 2017-02-16 at 19:06 +0100, Mike Galbraith wrote:
On Thu, 2017-02-16 at 15:53 +0100, Sebastian Andrzej Siewior wrote:
quoted
On 2017-02-16 15:42:59 [+0100], Mike Galbraith wrote:
quoted
Weeell, I'm trying to cobble something kinda like that together using
__RT_SPIN_INITIALIZER() instead, but seems mean ole Mr. Compiler NAKs
the PER_CPU_DEP_MAP_INIT() thingy.
CC mm/swap.o
mm/swap.c:54:689: error: braced-group within expression allowed only
inside a function
so this is what I have now. I need to get the `static' symbol working
again and PER_CPU_DEP_MAP_INIT but aside from that it seems to do its
job.
...
Yeah, works, I should be able to do an ltp run with stock lockdep
settings without it taking it's toys and going home in a snit.
berio:/sys/kernel/debug/tracing/:[0]# !while
while sleep 60; do tail -1 trace; done
<...>-10315 [064] d...1.. 226.953935: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14223
w-13148 [120] d...111 287.414978: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14465
w-16492 [089] d...111 347.128742: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 14653
(starts kbuild loop)
btrfs-transacti-1964 [016] d...1.. 411.101549: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 17011
<...>-100268 [127] d...112 472.271769: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18153
w-18864 [011] d...1.. 534.386443: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18543
<...>-50390 [035] dN..2.. 597.794164: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18765
<...>-80098 [127] d...111 659.912145: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 18977
checkproc-11123 [017] d...1.. 721.483463: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19247
-0 [055] d..h5.. 782.685953: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19383
<...>-93632 [055] d...111 835.527817: add_lock_to_list.isra.24.constprop.42: nr_list_entries: 19441