Re: Memory leak in ip_dst_cache

Hi Eric,
I did not flush those routes explicitly. They are flushed out automatically by calling "rt_run_flush". It inturn calls rt_free(), but __cache_free() never gets called after that. Basically, I have 512 MBytes of memory on board. I want to restrict the cache to around 64k, with max_size. But if I do that due to this leak, TCP/IP communication stops working when it reaches 64k.
 
Thanks
psk


----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: 
Sent: Friday, September 9, 2011 12:33 PM
Subject: Re: Memory leak in ip_dst_cache

Le vendredi 09 septembre 2011 à 11:39 -0700, Kumar S a écrit :

Hi Eric,
Thanks for the response. Here is the output of the commands you mentioned.

Hmm, you should have included netdev in this message.

After injecting around11k routes:
-sh-2.05b#
-sh-2.05b# ip route ls cache | wc
  23256  162772 1166903
-sh-2.05b#

After route flush out:

-sh-2.05b#
-sh-2.05b# ip route ls cache | wc
     14      88     710
-sh-2.05b#
-sh-2.05b#
-sh-2.05b#
-sh-2.05b# cat /proc/slabinfo | grep ip_dst
ip_dst_cache       11673  11685    256   15    1 : tunables  120   60    0 : slabdata    779    779      0

Thats OK : Each tcp session has a pointer to a dst.

When you "ip route flush cache", busy dst are all invalidated.
"ip route ls cache" doesnt output invalidated dsts.

As soon as tcp session will try to send a packet, it will notice the dst
is obsolete and 'free' it.

-sh-2.05b#
-sh-2.05b# grep . /proc/sys/net/ipv4/route/*
/proc/sys/net/ipv4/route/error_burst:1250
/proc/sys/net/ipv4/route/error_cost:250
grep: /proc/sys/net/ipv4/route/flush: Permission denied
/proc/sys/net/ipv4/route/gc_elasticity:8
/proc/sys/net/ipv4/route/gc_interval:60
/proc/sys/net/ipv4/route/gc_min_interval:0
/proc/sys/net/ipv4/route/gc_min_interval_ms:500
/proc/sys/net/ipv4/route/gc_thresh:16384
/proc/sys/net/ipv4/route/gc_timeout:300
/proc/sys/net/ipv4/route/max_delay:10
/proc/sys/net/ipv4/route/max_size:2097152
/proc/sys/net/ipv4/route/min_adv_mss:256
/proc/sys/net/ipv4/route/min_delay:2
/proc/sys/net/ipv4/route/min_pmtu:552
/proc/sys/net/ipv4/route/mtu_expires:600
/proc/sys/net/ipv4/route/redirect_load:5
/proc/sys/net/ipv4/route/redirect_number:9
/proc/sys/net/ipv4/route/redirect_silence:5120
/proc/sys/net/ipv4/route/secret_interval:600
-sh-2.05b# rtstat -c10 -i1
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
 entries|  in_hit|in_slow_|in_slow_|in_no_ro|  in_brd|in_marti|in_marti| out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
        |        |     tot|      mc|     ute|        |  an_dst|  an_src|        |    _tot|     _mc|        |      ed|    miss| verflow| _search|t_search|
   11675|   27700|  101014|       0|       0|     130|       0|       0|   36284|   23436|       0|       0|       0|       0|       0|    9658|   23140|
   11675|       6|      18|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       2|      10|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       8|      18|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       6|      18|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       4|      16|       0|       0|       0|       0|       0|       4|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       8|      12|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       6|      10|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       2|      14|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
   11675|       8|      12|       0|       0|       0|       0|       0|       2|       0|       0|       0|       0|       0|       0|       0|       0|
-sh-2.05b#

Thats pretty low ip route usage, you dont need to tweak your tunables.

Thanks

Le vendredi 09 septembre 2011 à 13:48 -0700, Kumar S a écrit :

Hi Eric,
I did not flush those routes explicitly. They are flushed out
automatically by calling "rt_run_flush". It inturn calls rt_free(),
but __cache_free() never gets called after that. Basically, I have 512
MBytes of memory on board. I want to restrict the cache to around 64k,
with max_size. But if I do that due to this leak, TCP/IP communication
stops working when it reaches 64k.
 

Your dsts use about 3Mb of memory out of 512Mb. Its certainly not the
reason of the failures you have.

Its not a leak.

All "in use" dst are queued in dst_gc, check net/core/dst.c for details.