Re: RFC Hanging clean-up of a namespace

On Friday 20 January 2012 11:08:37 Eric W. Biederman wrote:

Hans Schillstrom [off-list ref] writes:

quoted

On Thursday 19 January 2012 22:40:53 Hagen Paul Pfeifer wrote:

quoted

* Eric W. Biederman | 2012-01-19 13:24:13 [-0800]:

quoted

This thread is a fascinating disconnect from reality all of the way
around.

- inet_twsk_purge already implements throwing out of timewait sockets
 when a network namespaces is being cleaned up.  So the RFC is nonsense.

This is how it is implemented, not how it should be. TIME_WAIT is not the
problem, it is there to keep the stack from sending wrong RST messages. Maybe
the 2*MSL could be fixed by a more accurate 2*RTT.

I was only refering to my printk's i.e. the last sockets leaving the namespace was
from tcp_timer() with state 7, 2 minutes after free_nsproxy() was called.
(and assumed that was the time_wait)

Which kernel are you running?  

 3.2.0 

I can't find a mention of a function
named tcp_timer() anywhere in the kernel since 2.6.16 when the kernel
was put into git.

Sorry, it was tcp_write_timer() in tcp_timer.c

There is a file named net/ipv4/tcp_timer.c

But if you are actually describing normal sockets and not timewait
sockets then it is remotely possible that something like what you are
talking about is happening. 

Hmm, state 7 is TCP_CLOSE I simply assumed that it was TCP_WAIT ...

Normal sockets keep the network namespace
alive.  So if something was keeping the sockets open.  Like perhaps a
process that has one of your sockets from your network namespace open
then it could happen.

We had a number of procs. with tcp connections open, and kill proc 1 (lxc-init)
i.e. all procs. in the ns got killed within a few ms. 
(or at least no visible traces left)

nsproxy is not the only place that references to the network namespace
are allowed to live that keep the network namespace alive.

quoted

quoted

quoted

- Keeping the timewait sockets at that point we purge them in the code
 can achieve nothing.  We don't have any userspace processes or network
 devices associated with the timewait sockets at the point we get rid
 of them.  The network namespace exists so long as a userspace process
 can find it.  The network namespace exit is asynchronous in it's own
 workqueue so userspace definitely is not blocked.

One example of a real life problem is when a container crash where a VLAN from
a physical interface is used in the container, and you automatically reboot
that container.  A new namespace is created with that VLAN again and what happens ?
That VLAN id is busy (waiting for tcp_timer) and the continer start fails ...
So you have to wait a couple of minutes :-(

Yes the vlan is busy until that the network namespace is cleaned up, and
we get as far as calling dellink on the network namespace.

There are a lot of reasons why a network namespace would not be cleaned
up immediately.  Especially in older kernels.

One problem people running older kernels had troubles with was vsftp
created an empty network namespace for every connection.  On kernels pre
2.6.34 I think before we had batching support for cleaning up network
devices and network namespaces the kernel could simply not keep up with
the rate that vsftp was creating and destroying network namespaces, and
would slowly fall farther and farther behind in it's cleanup.

If you are running an older kernel it is quite possible that you are
missing some cleanups.  It is also possible that you are hitting one of
the cases where we can only destroy 4 network devices a second and you
have lots of network devices dying with your network namespace.

We started with 2.6.32 but the cleanup process didn't work we always end up
with ref-counts on loopback

Thanks
Hans