Re: [PATCH 00/16] Remove the ipv4 routing cache
From: Eric Dumazet <hidden>
Date: 2012-07-26 08:13:40
On Wed, 2012-07-25 at 17:54 -0700, David Miller wrote:
From: David Miller <davem@davemloft.net> Date: Wed, 25 Jul 2012 16:39:39 -0700 (PDT)quoted
From: David Miller <davem@davemloft.net> Date: Wed, 25 Jul 2012 16:17:32 -0700 (PDT)quoted
From: Alexander Duyck <redacted> Date: Wed, 25 Jul 2012 16:02:45 -0700quoted
Since your patches are in I have started to re-run my tests. I am seeing a significant drop in throughput with 8 flows which I expected, however it looks like one of the biggest issues I am seeing is that the dst_hold and dst_release calls seem to be causing some serious cache thrash. I was at 12.5Mpps w/ 8 flows before the patches, after your patches it drops to 8.3Mpps.Yes, this is something we knew would start happening. One idea is to make cached dsts be per-cpu in the nexthops.Actually I think what really kills your case is the removal of the noref path for route lookups. I'll work on a patch to restore that in the case where we use cached routes from the FIB nexthops.Alex, here is something I tossed together, does it help with the dst_hold()/dst_release() overhead at all?
seems good to me, only one question :
quoted hunk ↗ jump to hunk
static void rt_cache_route(struct fib_nh *nh, struct rtable *rt)@@ -1216,9 +1215,15 @@ static void rt_cache_route(struct fib_nh *nh, struct rtable *rt) prev = cmpxchg(p, orig, rt); if (prev == orig) { - dst_clone(&rt->dst); if (orig) - call_rcu_bh(&orig->dst.rcu_head, rt_release_rcu); + rt_free(orig); + } else { + /* Routes we intend to cache in the FIB nexthop have + * the DST_NOCACHE bit set. However, if we are + * unsuccessful at storing this route into the cache + * we really need to clear that bit. + */ + rt->dst.flags &= ~DST_NOCACHE; } }
Not sure why you removed the dst_clone(&rt->dst) ?
If it is not needed, we might need to release a reference in the else {}
clause, no ?