From: Eric Dumazet <hidden> Date: 2011-01-22 08:59:49
Le vendredi 21 janvier 2011 à 22:30 -0800, PK a écrit :
Creating many ipv6 connections hits a ceiling on connections/fds ; okay, fine.
But in my case I'm seeing millions of entries spring up within a few seconds and
then vanish within a few minutes, in /proc/net/tcp6 (vanish due to garbage
collection?)
Furthermore I can trigger this easily on vanilla kernels from 2.6.36 to
2.6.38-rc1-next-20110121 inside a ubuntu 10.10 amd64 vm, causing the kernel to
spew warnings. There is also some corruption in the logs (see kernel-sample.log
line 296), but that may be unrelated.
More explanation, kernel config of the primary machine I saw this on, sample
ruby script to reproduce (inside the ubuntu VMs I apt-get and use ruby-1.9.1),
are located at
https://github.com/runningdogx/net6-bug
Seems to only affect 64-bit. So far I have not been able to reproduce on 32-bit
ubuntu VMs of any kernel version.
Seems to only affect IPv6. So far I have not been able to reproduce using IPv4
connections (and watching /proc/net/tcp of course).
Does not trigger the bug if the connections are made to ::1. Only externally
routable local and global IPv6 addresses seem to cause problems.
Seems to have been introduced between 2.6.35 and 2.6.36 (see README on github
for more kernels I've tried)
All the tested Ubuntu VMs are stock 10.10 userland, with vanilla kernels (the
latest ubuntu kernel is 2.6.35-something, and my initial test didn't show it
suffering from this problem)
Originally noticed on separate Gentoo 64-bit non-vm system when doing web
benchmarking.
not subscribed, so please keep me in cc although I'll try to follow the thread
Hi PK (Sorry, your real name is hidden)
I could not reproduce this on current linux-2.6 kernel.
How many vcpus running in your VM, and memory ?
Note : a recent commit did fix /proc/net/tcp[6] behavior
commit 1bde5ac49398a064c753bb490535cfad89e99a5f
Author: Eric Dumazet [off-list ref]
Date: Thu Dec 23 09:32:46 2010 -0800
tcp: fix listening_get_next()
Alexey Vlasov found /proc/net/tcp could sometime loop and display
millions of sockets in LISTEN state.
In 2.6.29, when we converted TCP hash tables to RCU, we left two
sk_next() calls in listening_get_next().
We must instead use sk_nulls_next() to properly detect an end of chain.
Reported-by: Alexey Vlasov [off-list ref]
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
From: Eric Dumazet <hidden> Date: 2011-01-22 15:15:52
Le samedi 22 janvier 2011 à 09:59 +0100, Eric Dumazet a écrit :
Le vendredi 21 janvier 2011 à 22:30 -0800, PK a écrit :
quoted
Creating many ipv6 connections hits a ceiling on connections/fds ; okay, fine.
But in my case I'm seeing millions of entries spring up within a few seconds and
then vanish within a few minutes, in /proc/net/tcp6 (vanish due to garbage
collection?)
Furthermore I can trigger this easily on vanilla kernels from 2.6.36 to
2.6.38-rc1-next-20110121 inside a ubuntu 10.10 amd64 vm, causing the kernel to
spew warnings. There is also some corruption in the logs (see kernel-sample.log
line 296), but that may be unrelated.
More explanation, kernel config of the primary machine I saw this on, sample
ruby script to reproduce (inside the ubuntu VMs I apt-get and use ruby-1.9.1),
are located at
https://github.com/runningdogx/net6-bug
Seems to only affect 64-bit. So far I have not been able to reproduce on 32-bit
ubuntu VMs of any kernel version.
Seems to only affect IPv6. So far I have not been able to reproduce using IPv4
connections (and watching /proc/net/tcp of course).
Does not trigger the bug if the connections are made to ::1. Only externally
routable local and global IPv6 addresses seem to cause problems.
Seems to have been introduced between 2.6.35 and 2.6.36 (see README on github
for more kernels I've tried)
All the tested Ubuntu VMs are stock 10.10 userland, with vanilla kernels (the
latest ubuntu kernel is 2.6.35-something, and my initial test didn't show it
suffering from this problem)
Originally noticed on separate Gentoo 64-bit non-vm system when doing web
benchmarking.
not subscribed, so please keep me in cc although I'll try to follow the thread
I had some incidents, after hours of testing...
After following patch, I could not reproduce it.
I cant believe this bug was not noticed before today.
Thanks !
[PATCH] tcp: fix bug in listening_get_next()
commit a8b690f98baf9fb19 (tcp: Fix slowness in read /proc/net/tcp)
introduced a bug in handling of SYN_RECV sockets.
st->offset represents number of sockets found since beginning of
listening_hash[st->bucket].
We should not reset st->offset when iterating through
syn_table[st->sbucket], or else if more than ~25 sockets (if
PAGE_SIZE=4096) are in SYN_RECV state, we exit from listening_get_next()
with a too small st->offset
Next time we enter tcp_seek_last_pos(), we are not able to seek past
already found sockets.
Reported-by: PK <redacted>
CC: Tom Herbert <redacted>
Signed-off-by: Eric Dumazet <redacted>
---
net/ipv4/tcp_ipv4.c | 1 -
1 file changed, 1 deletion(-)
From: Eric Dumazet <hidden> Date: 2011-01-22 21:20:45
Le samedi 22 janvier 2011 à 11:42 -0800, PK a écrit :
Eric Dumazet wrote:
quoted
I had some incidents, after hours of testing...
After following patch, I could not reproduce it.
Looks like that patch solved the /proc/net/tcp6 problem. The causal commit was
the one you identified... confirmed with bisect.
These warnings show up when I run the script (or I presume any tcp6 connection
flooder) with /proc/sys/net/tcp/ipv4/tcp_tw_recycle enabled. There's textual
corruption of the traces a lot of the time. Here's a sample trace that doesn't
appear to be corrupt. All the warnings I've seen are from route.c:209, and I
don't see how that would cause memory corruption.
From: Eric Dumazet <hidden> Date: 2011-01-22 21:40:53
Le samedi 22 janvier 2011 à 22:20 +0100, Eric Dumazet a écrit :
Le samedi 22 janvier 2011 à 11:42 -0800, PK a écrit :
quoted
Eric Dumazet wrote:
quoted
I had some incidents, after hours of testing...
After following patch, I could not reproduce it.
Looks like that patch solved the /proc/net/tcp6 problem. The causal commit was
the one you identified... confirmed with bisect.
These warnings show up when I run the script (or I presume any tcp6 connection
flooder) with /proc/sys/net/tcp/ipv4/tcp_tw_recycle enabled. There's textual
corruption of the traces a lot of the time. Here's a sample trace that doesn't
appear to be corrupt. All the warnings I've seen are from route.c:209, and I
don't see how that would cause memory corruption.
From: David Miller <davem@davemloft.net> Date: 2011-01-24 22:31:08
From: Eric Dumazet <redacted>
Date: Sat, 22 Jan 2011 22:40:44 +0100
Le samedi 22 janvier 2011 à 22:20 +0100, Eric Dumazet a écrit :
quoted
Le samedi 22 janvier 2011 à 11:42 -0800, PK a écrit :
quoted
Eric Dumazet wrote:
quoted
I had some incidents, after hours of testing...
After following patch, I could not reproduce it.
Looks like that patch solved the /proc/net/tcp6 problem. The causal commit was
the one you identified... confirmed with bisect.
These warnings show up when I run the script (or I presume any tcp6 connection
flooder) with /proc/sys/net/tcp/ipv4/tcp_tw_recycle enabled. There's textual
corruption of the traces a lot of the time. Here's a sample trace that doesn't
appear to be corrupt. All the warnings I've seen are from route.c:209, and I
don't see how that would cause memory corruption.
From: David Miller <davem@davemloft.net> Date: 2011-01-24 22:39:51
From: Eric Dumazet <redacted>
Date: Sat, 22 Jan 2011 22:40:44 +0100
In my testings, I even have crashes in cleanup_once() if I
enable /proc/sys/net/ipv4/tcp_tw_recycle
Luckily, this bug was easy to fix, I've just committed the
following to net-2.6
The other crash (the !RTF_CACHE WARN assertion) I'm looking
into now.
--------------------
From 1c5642cf754939c318a0230b0f546a9e20888292 Mon Sep 17 00:00:00 2001
From: David S. Miller <davem@davemloft.net>
Date: Mon, 24 Jan 2011 14:37:46 -0800
Subject: [PATCH] inetpeer: Use correct AVL tree base pointer in inet_getpeer().
Family was hard-coded to AF_INET but should be daddr->family.
This fixes crashes when unlinking ipv6 peer entries, since the
unlink code was looking up the base pointer properly.
Reported-by: Eric Dumazet <redacted>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
net/ipv4/inetpeer.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
@@ -475,7 +475,7 @@ static int cleanup_once(unsigned long ttl)structinet_peer*inet_getpeer(structinetpeer_addr*daddr,intcreate){structinet_peer__rcu**stack[PEER_MAXDEPTH],***stackptr;-structinet_peer_base*base=family_to_base(AF_INET);+structinet_peer_base*base=family_to_base(daddr->family);structinet_peer*p;/* Look up for the address quickly, lockless.
From: David Miller <davem@davemloft.net> Date: 2011-01-24 22:41:26
From: Eric Dumazet <redacted>
Date: Sat, 22 Jan 2011 16:15:44 +0100
[PATCH] tcp: fix bug in listening_get_next()
commit a8b690f98baf9fb19 (tcp: Fix slowness in read /proc/net/tcp)
introduced a bug in handling of SYN_RECV sockets.
st->offset represents number of sockets found since beginning of
listening_hash[st->bucket].
We should not reset st->offset when iterating through
syn_table[st->sbucket], or else if more than ~25 sockets (if
PAGE_SIZE=4096) are in SYN_RECV state, we exit from listening_get_next()
with a too small st->offset
Next time we enter tcp_seek_last_pos(), we are not able to seek past
already found sockets.
Reported-by: PK <redacted>
CC: Tom Herbert <redacted>
Signed-off-by: Eric Dumazet <redacted>
These warnings show up when I run the script (or I presume any tcp6 connection
flooder) with /proc/sys/net/tcp/ipv4/tcp_tw_recycle enabled. There's textual
corruption of the traces a lot of the time. Here's a sample trace that doesn't
appear to be corrupt. All the warnings I've seen are from route.c:209, and I
don't see how that would cause memory corruption.
Please give this patch a try:
--------------------
From d80bc0fd262ef840ed4e82593ad6416fa1ba3fc4 Mon Sep 17 00:00:00 2001
From: David S. Miller <davem@davemloft.net>
Date: Mon, 24 Jan 2011 16:01:58 -0800
Subject: [PATCH] ipv6: Always clone offlink routes.
Do not handle PMTU vs. route lookup creation any differently
wrt. offlink routes, always clone them.
Reported-by: PK <redacted>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
net/ipv6/route.c | 9 +--------
1 files changed, 1 insertions(+), 8 deletions(-)