Re: kernel 2.6.25-rc7 highly unstable on high load
From: Eric Dumazet <hidden>
Date: 2008-03-27 16:29:56
Also in:
netfilter-devel
Subsystem:
networking [general], the rest · Maintainers:
"David S. Miller", Eric Dumazet, Jakub Kicinski, Paolo Abeni, Linus Torvalds
Possibly related (same subject, not in this thread)
- 2008-03-29 · Re: kernel 2.6.25-rc7 highly unstable on high load · Eric Dumazet <hidden>
- 2008-03-27 · Re: kernel 2.6.25-rc7 highly unstable on high load · Patrick McHardy <hidden>
- 2008-03-27 · Re: kernel 2.6.25-rc7 highly unstable on high load · Eric Dumazet <hidden>
- 2008-03-27 · Re: kernel 2.6.25-rc7 highly unstable on high load · Denys Fedoryshchenko <hidden>
- 2008-03-27 · Re: kernel 2.6.25-rc7 highly unstable on high load · Denys Fedoryshchenko <hidden>
Eric Dumazet a écrit :
Denys Fedoryshchenko a écrit :quoted
I did this tuning. Not sure what does mean rt_cache_entries, but seems they are still increasing. Kup /config # rtstat -c1000 -i60 rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache| rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache| rt_cache| entries| in_hit|in_slow_|in_slow_|in_no_ro| in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist| out_hlis| | | tot| mc| ute| | an_dst| an_src| | _tot| _mc| | ed| miss| verflow| _search|t_search| 274234| 1590009| 271953| 0| 1809| 6| 0| 0| 24551| 7365| 0| 279991| 279867| 24| 0| 5438630| 89656| 280535| 25463| 19956| 0| 187| 0| 0| 0| 485| 263| 0| 20407| 20404| 3| 0| 12796| 242| 288072| 25367| 19830| 0| 163| 0| 0| 0| 493| 256| 0| 20248| 20245| 3| 0| 17146| 346| 294943| 25689| 19807| 0| 154| 0| 0| 0| 438| 250| 0| 20211| 20208| 3| 0| 21554| 432| 301118| 25287| 19645| 0| 170| 0| 0| 0| 466| 236| 0| 20052| 20048| 3| 0| 25713| 528| 307651| 25663| 20264| 0| 180| 0| 0| 0| 472| 234| 0| 20678| 20675| 3| 0| 30588| 607| 313767| 25810| 20014| 0| 175| 0| 0| 0| 459| 237| 0| 20425| 20422| 3| 0| 33701| 699|Then try to lower gc_elasticity, to 3, or even 2 echo 3 >/proc/sys/net/ipv4/route/gc_elasticity
One more idea before leaving :) If rt_cache_entries still increasing, we might have a dst leak somewhere : gc tries to evict entries that have a non null refcount -> they are put in dst_garbage.list Following patch will show us how dst_garbage behaves. ( printk(KERN_DEBUG "dst_total: %d delayed: %d work_perf: %d" ...)
diff --git a/net/core/dst.c b/net/core/dst.c
index 7deef48..e634e5f 100644
--- a/net/core/dst.c
+++ b/net/core/dst.c@@ -31,6 +31,8 @@ * 3) This list is guarded by a mutex, * so that the gc_task and dst_dev_event() can be synchronized. */ +#undef RT_CACHE_DEBUG +#define RT_CACHE_DEBUG 2 #if RT_CACHE_DEBUG >= 2 static atomic_t dst_total = ATOMIC_INIT(0); #endif