Re: Memory leak in ip_dst_cache

Le vendredi 09 septembre 2011 à 14:58 -0700, Kumar S a écrit :

I agree, 3 MB is not a concern. But actually this is exceeding, we are
seeing more than 1200000 objects in a span of 24 hours, which is close
to 300 MB.

Hmm, please include netdev in your mails.

Then there is something else.

Maybe a timer bug preventing dst_gc_task() to make its duty.

 Other tasks require somewhere around 200 MB+. When it reaches this
number, the board reboots. In lab we could simulate this. That is how
I'm trying to understand the problem. I'm injecting around 10k routes,
and waiting for them to ageout.
 

I dont know, you could try disable IP route cache.

On recent (2.6.29+) kernels, thats very easy

# Disable Ip route cache (negative value in rebuild_count)
echo 3000000000 >/proc/sys/net/ipv4/rt_cache_rebuild_count

On an old kernel, you probably need to patch your kernel, or backport
commit 1080d709fb9d8cd43 (net: implement emergency route cache rebulds
when gc_elasticity is exceeded)

Yes Eric, this command doesn't work on 2.6.24. Which timer are you suspecting?


----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: netdev <redacted>
Sent: Friday, September 9, 2011 3:08 PM
Subject: Re: Memory leak in ip_dst_cache

Le vendredi 09 septembre 2011 à 14:58 -0700, Kumar S a écrit :

I agree, 3 MB is not a concern. But actually this is exceeding, we are
seeing more than 1200000 objects in a span of 24 hours, which is close
to 300 MB.

Hmm, please include netdev in your mails.

Then there is something else.

Maybe a timer bug preventing dst_gc_task() to make its duty.

 Other tasks require somewhere around 200 MB+. When it reaches this
number, the board reboots. In lab we could simulate this. That is how
I'm trying to understand the problem. I'm injecting around 10k routes,
and waiting for them to ageout.
 

I dont know, you could try disable IP route cache.

On recent (2.6.29+) kernels, thats very easy

# Disable Ip route cache (negative value in rebuild_count)
echo 3000000000 >/proc/sys/net/ipv4/rt_cache_rebuild_count

On an old kernel, you probably need to patch your kernel, or backport
commit 1080d709fb9d8cd43 (net: implement emergency route cache rebulds
when gc_elasticity is exceeded)

On Fri, Sep 09, 2011 at 03:53:45PM -0700, Kumar S wrote:

Yes Eric, this command doesn't work on 2.6.24. Which timer are you suspecting?

He means that the timer that starts the the workqueue which calls dst_gc_task is
either delayed indefinately, or otherwise not run, which means that the route
cache will never be scanned for old entries.  The implication being that, while
the dst entries aren't leaked per-se, but never expired, so they just sit
around.  You can check this by instrumenting gc_dst_task with a printk or two
and watch to see if it ever pops up.

If you only have 512Mb of ram and want to really restrict you're cache size, you
should definately consider backporting the route cache rebuild patch to disable
the route cache usage entirely.  It shouldn't be too hard to do.
Neil

Thanks Neil. I did try with prink(). I do see entries getting aged out, but they are not getting deallocated. This seems to be happening because of "ref_cnt". When the route entries are added the ref_cnt is set to 1. Looks this is causing trouble clearing the entries completely. If I set the ref_cnt to 0, I can see it working. Now I'm trying to understand whether this is right. Please let me know if you have any thoughts on it.


----- Original Message -----
From: Neil Horman <nhorman@tuxdriver.com>
To: Kumar S <redacted>
Cc: Eric Dumazet <redacted>; netdev <redacted>
Sent: Saturday, September 10, 2011 6:04 AM
Subject: Re: Memory leak in ip_dst_cache

On Fri, Sep 09, 2011 at 03:53:45PM -0700, Kumar S wrote:

Yes Eric, this command doesn't work on 2.6.24. Which timer are you suspecting?

He means that the timer that starts the the workqueue which calls dst_gc_task is
either delayed indefinately, or otherwise not run, which means that the route
cache will never be scanned for old entries.  The implication being that, while
the dst entries aren't leaked per-se, but never expired, so they just sit
around.  You can check this by instrumenting gc_dst_task with a printk or two
and watch to see if it ever pops up.

If you only have 512Mb of ram and want to really restrict you're cache size, you
should definately consider backporting the route cache rebuild patch to disable
the route cache usage entirely.  It shouldn't be too hard to do.
Neil

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Le dimanche 11 septembre 2011 à 20:38 -0700, Kumar S a écrit :

Please dont top post.

Thanks Neil. I did try with prink(). I do see entries getting aged
out, but they are not getting deallocated. This seems to be happening
because of "ref_cnt". When the route entries are added the ref_cnt is
set to 1. Looks this is causing trouble clearing the entries
completely. If I set the ref_cnt to 0, I can see it working. Now I'm
trying to understand whether this is right. Please let me know if you
have any thoughts on it.

I believe I already explained what was happening.

A tcp socket has a pointer to a dst, so it holds a reference on it, to
make sure no freeing of dst can happen while at least some socket still
can reference dst. (It could reference freed memory and crash)

As soon as the tcp socket will try to transmit some data, the dst will
be checked and we notice its obsolete : We then release the refcount and
dst pointer.

Later, the garbage collector can notice dst refcount is zero and can
free dst.

If you have dormant tcp sockets (no trafic at all), they hold their dst.

A dormant tcp socket has a pretty more expensive memory cost than its
dst. (Socket structure, dentry, inode, and probably in user land a
thread or process, and data)

----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: Neil Horman <nhorman@tuxdriver.com>; netdev <redacted>
Sent: Sunday, September 11, 2011 10:40 PM
Subject: Re: Memory leak in ip_dst_cache

Le dimanche 11 septembre 2011 à 20:38 -0700, Kumar S a écrit :

Please dont top post.

quoted

Thanks Neil. I did try with prink(). I do see entries getting aged
out, but they are not getting deallocated. This seems to be happening
because of "ref_cnt". When the route entries are added the ref_cnt is
set to 1. Looks this is causing trouble clearing the entries
completely. If I set the ref_cnt to 0, I can see it working. Now I'm
trying to understand whether this is right. Please let me know if you
have any thoughts on it.

I believe I already explained what was happening.

A tcp socket has a pointer to a dst, so it holds a reference on it, to
make sure no freeing of dst can happen while at least some socket still
can reference dst. (It could reference freed memory and crash)

As soon as the tcp socket will try to transmit some data, the dst will
be checked and we notice its obsolete : We then release the refcount and
dst pointer.

Later, the garbage collector can notice dst refcount is zero and can
free dst.

If you have dormant tcp sockets (no trafic at all), they hold their dst.
A dormant tcp socket has a pretty more expensive memory cost than its
dst. (Socket structure, dentry, inode, and probably in user land a
thread or process, and data)

 
Thanks Eric for detailed explanation. You did mention this before. What I see is the cache entries related to the TCP sockets are getting cleared, whenever they age out. But the issue we see here is with the broadcast messages such as SMB messages and network neighbor hood messages. They never get freed. There is no traffic to those destinations from our board. 

Le dimanche 11 septembre 2011 à 23:07 -0700, Kumar S a écrit :

----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: Neil Horman <nhorman@tuxdriver.com>; netdev <redacted>
Sent: Sunday, September 11, 2011 10:40 PM
Subject: Re: Memory leak in ip_dst_cache

Le dimanche 11 septembre 2011 à 20:38 -0700, Kumar S a écrit :

Please dont top post.

quoted

quoted

Thanks Neil. I did try with prink(). I do see entries getting aged
out, but they are not getting deallocated. This seems to be happening
because of "ref_cnt". When the route entries are added the ref_cnt is
set to 1. Looks this is causing trouble clearing the entries
completely. If I set the ref_cnt to 0, I can see it working. Now I'm
trying to understand whether this is right. Please let me know if you
have any thoughts on it.

quoted

I believe I already explained what was happening.

quoted

A tcp socket has a pointer to a dst, so it holds a reference on it, to
make sure no freeing of dst can happen while at least some socket still
can reference dst. (It could reference freed memory and crash)

quoted

As soon as the tcp socket will try to transmit some data, the dst will
be checked and we notice its obsolete : We then release the refcount and
dst pointer.

quoted

Later, the garbage collector can notice dst refcount is zero and can
free dst.

quoted

If you have dormant tcp sockets (no trafic at all), they hold their dst.
A dormant tcp socket has a pretty more expensive memory cost than its
dst. (Socket structure, dentry, inode, and probably in user land a
thread or process, and data)

 
Thanks Eric for detailed explanation. You did mention this before.
What I see is the cache entries related to the TCP sockets are getting
cleared, whenever they age out. But the issue we see here is with the
broadcast messages such as SMB messages and network neighbor hood
messages. They never get freed. There is no traffic to those
destinations from our board. 

What do you mean ? Your box is a router only ?

Those SMB messages are going through it ?

----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: Neil Horman <nhorman@tuxdriver.com>; netdev <redacted>
Sent: Sunday, September 11, 2011 11:28 PM
Subject: Re: Memory leak in ip_dst_cache

Le dimanche 11 septembre 2011 à 23:07 -0700, Kumar S a écrit :

quoted

----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: Neil Horman <nhorman@tuxdriver.com>; netdev <redacted>
Sent: Sunday, September 11, 2011 10:40 PM

Subject: Re: Memory leak in ip_dst_cache

quoted

Le dimanche 11 septembre 2011 à 20:38 -0700, Kumar S a écrit :

Please dont top post.

quoted

quoted

Thanks Neil. I did try with prink(). I do see entries getting aged
out, but they are not getting deallocated. This seems to be happening
because of "ref_cnt". When the route entries are added the ref_cnt is
set to 1. Looks this is causing trouble clearing the entries
completely. If I set the ref_cnt to 0, I can see it working. Now I'm
trying to understand whether this is right. Please let me know if you
have any thoughts on it.

quoted

I believe I already explained what was happening.

quoted

A tcp socket has a pointer to a dst, so it holds a reference on it, to
make sure no freeing of dst can happen while at least some socket still
can reference dst. (It could reference freed memory and crash)

quoted

As soon as the tcp socket will try to transmit some data, the dst will
be checked and we notice its obsolete : We then release the refcount and
dst pointer.

quoted

Later, the garbage collector can notice dst refcount is zero and can
free dst.

quoted

If you have dormant tcp sockets (no trafic at all), they hold their dst.
A dormant tcp socket has a pretty more expensive memory cost than its
dst. (Socket structure, dentry, inode, and probably in user land a
thread or process, and data)

Thanks Eric for detailed explanation. You did mention this before.
What I see is the cache entries related to the TCP sockets are getting
cleared, whenever they age out. But the issue we see here is with the
broadcast messages such as SMB messages and network neighbor hood
messages. They never get freed. There is no traffic to those
destinations from our board. 

What do you mean ? Your box is a router only ?

Those SMB messages are going through it ?

 
Our box is a stand-alone system with L2 Quick Engine. This QE forwards all broadcast to the other ports and also a copy to the CPU port. 

Le lundi 12 septembre 2011 à 10:16 -0700, Kumar S a écrit :

----- Original Message -----
From: Eric Dumazet <redacted>

quoted

What do you mean ? Your box is a router only ?

quoted

Those SMB messages are going through it ?

 
Our box is a stand-alone system with L2 Quick Engine. This QE forwards
all broadcast to the other ports and also a copy to the CPU port. 

It sounds like a modified kernel, maybe you added a bug in the code...

----- Original Message -----
From: Eric Dumazet <redacted>
To: Kumar S <redacted>
Cc: Neil Horman <nhorman@tuxdriver.com>; netdev <redacted>
Sent: Monday, September 12, 2011 10:57 AM
Subject: Re: Memory leak in ip_dst_cache

Le lundi 12 septembre 2011 à 10:16 -0700, Kumar S a écrit :

quoted

----- Original Message -----
From: Eric Dumazet <redacted>

quoted

What do you mean ? Your box is a router only ?

quoted

Those SMB messages are going through it ?

Our box is a stand-alone system with L2 Quick Engine. This QE forwards
all broadcast to the other ports and also a copy to the CPU port. 

It sounds like a modified kernel, maybe you added a bug in the code...

It is possible. I'm trying to isolate the same. Your input is helping a lot in understanding the flow related to ip_dst_cache.