Bug? TCP shutdown behaviour when deleting local IP addresses

Hi all,

I sent this to the list yesterday (from another address) but didn't get any responses.  Accordingly I'm expanding the receiver list to the listed maintainers for IPv4/IPv6.

I'm seeing some unexpected (to me, at least) behaviour with local TCP connections.  The scenario goes as follows:

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address
5) kill server with ctrl-C.  At this point it appears that because the address was removed the shutdown message isn't processed properly. netstat shows the server socket as FIN_WAIT1, but the client socket is still ESTABLISHED.
6) client writes to the connected socket (this passes with no error)
7) client waits for response from server, and waits forever or until keepalive expires



A few points:

This was originally seen on 2.6.27, but I've verified it on 2.6.35. I'll see about trying it on current git.  I've got really simple client/server code if anyone wants to try reproducing.

If we don't remove the address in step 4, then step 5 results in the server socket going to FIN_WAIT2 and the client socket going to CLOSE_WAIT and step 7 returns right away with zero bytes.

It seems like the waiting forever behaviour in step 7 might be legitimate since the address was removed before shutting down the server, but it also seems like we should be able to do better given that everything is local.  In the "remove IP address" case maybe step 6 should cause some sort of error since the IP address no longer exists?

Incidentally, if we do this sort of scenario with the client and server on different hosts then we get a "no route to host" error at step 6.

Curious how this is supposed to work...

Chris



-- 

Chris Friesen
Software Designer
GENBAND
www.genband.com

From: Chris Friesen <redacted>
Date: Wed, 17 Oct 2012 17:01:40 -0600

I sent this to the list yesterday (from another address) but didn't
get any responses.  Accordingly I'm expanding the receiver list to the
listed maintainers for IPv4/IPv6.

It can take more than 24 hours to get a response from people who are
volunteers, and who will reply to your report because they want to
rather than because they are obligated.

Posting it again just irritates such people, and will place your
report at a much lower priority, just FYI.

On Wed, 2012-10-17 at 17:01 -0600, Chris Friesen wrote:

Hi all,

I sent this to the list yesterday (from another address) but didn't get 
any responses.  Accordingly I'm expanding the receiver list to the 
listed maintainers for IPv4/IPv6.

I'm seeing some unexpected (to me, at least) behaviour with local TCP 
connections.  The scenario goes as follows:

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address
5) kill server with ctrl-C.  At this point it appears that because the 
address was removed the shutdown message isn't processed properly. 
netstat shows the server socket as FIN_WAIT1, but the client socket is 
still ESTABLISHED.
6) client writes to the connected socket (this passes with no error)
7) client waits for response from server, and waits forever or until 
keepalive expires



A few points:

This was originally seen on 2.6.27, but I've verified it on 2.6.35. I'll 
see about trying it on current git.  I've got really simple 
client/server code if anyone wants to try reproducing.

If we don't remove the address in step 4, then step 5 results in the 
server socket going to FIN_WAIT2 and the client socket going to 
CLOSE_WAIT and step 7 returns right away with zero bytes.

It seems like the waiting forever behaviour in step 7 might be 
legitimate since the address was removed before shutting down the 
server, but it also seems like we should be able to do better given that 
everything is local.  In the "remove IP address" case maybe step 6 
should cause some sort of error since the IP address no longer exists?

Incidentally, if we do this sort of scenario with the client and server 
on different hosts then we get a "no route to host" error at step 6.

Curious how this is supposed to work...

Chris

I see no real problem here.

Its like you cut the cable somewhere in the path.

Only timeouts will apply.

And its not keeepalive timeouts in 7) but normal retransmits with
exponential backoff.

Extract of Documentation/networking/ip-sysctl.txt :

tcp_retries1 - INTEGER
        This value influences the time, after which TCP decides, that
        something is wrong due to unacknowledged RTO retransmissions,
        and reports this suspicion to the network layer.
        See tcp_retries2 for more details.

        RFC 1122 recommends at least 3 retransmissions, which is the
        default.

tcp_retries2 - INTEGER
        This value influences the timeout of an alive TCP connection,
        when RTO retransmissions remain unacknowledged.
        Given a value of N, a hypothetical TCP connection following
        exponential backoff with an initial RTO of TCP_RTO_MIN would
        retransmit N times before killing the connection at the (N+1)th
RTO.

        The default value of 15 yields a hypothetical timeout of 924.6
        seconds and is a lower bound for the effective timeout.
        TCP will effectively time out at the first RTO which exceeds the
        hypothetical timeout.

        RFC 1122 recommends at least 100 seconds for the timeout,
        which corresponds to a value of at least 8.

On 10/17/2012 05:27 PM, David Miller wrote:

From: Chris Friesen<redacted>
Date: Wed, 17 Oct 2012 17:01:40 -0600

quoted

I sent this to the list yesterday (from another address) but didn't
get any responses.  Accordingly I'm expanding the receiver list to the
listed maintainers for IPv4/IPv6.

It can take more than 24 hours to get a response from people who are
volunteers, and who will reply to your report because they want to
rather than because they are obligated.

Posting it again just irritates such people, and will place your
report at a much lower priority, just FYI.

Thanks for the pointers.  I was worried that it had gone unnoticed, but I guess the level of traffic on netdev is small enough (compared to something like lkml) that that's generally not an issue.

Chris

On 10/18/2012 12:05 AM, Eric Dumazet wrote:

On Wed, 2012-10-17 at 17:01 -0600, Chris Friesen wrote:

quoted

I'm seeing some unexpected (to me, at least) behaviour with local TCP
connections.  The scenario goes as follows:

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address
5) kill server with ctrl-C.  At this point it appears that because the
address was removed the shutdown message isn't processed properly.
netstat shows the server socket as FIN_WAIT1, but the client socket is
still ESTABLISHED.
6) client writes to the connected socket (this passes with no error)
7) client waits for response from server, and waits forever or until
keepalive expires

quoted

It seems like the waiting forever behaviour in step 7 might be
legitimate since the address was removed before shutting down the
server, but it also seems like we should be able to do better given that
everything is local.  In the "remove IP address" case maybe step 6
should cause some sort of error since the IP address no longer exists?

Incidentally, if we do this sort of scenario with the client and server
on different hosts then we get a "no route to host" error at step 6.

I see no real problem here.

Its like you cut the cable somewhere in the path.

Only timeouts will apply.

While I agree generally, it's a bit unfortunate that we can't (as a quality of implementation thing) give an earlier notice of failure since the kernel knows about both ends of the connection even though the IP address is gone.  On the other hand, I imagine that would mean special-casing things and presumably that would open a whole can of worms.

Thanks for the reply,

Chris

On Thu, 2012-10-18 at 01:08 -0600, Chris Friesen wrote:

While I agree generally, it's a bit unfortunate that we can't (as a 
quality of implementation thing) give an earlier notice of failure since 
the kernel knows about both ends of the connection even though the IP 
address is gone.  On the other hand, I imagine that would mean 
special-casing things and presumably that would open a whole can of worms.

Really what is the difference between a cable cut and what you are
doing ?

Some frames are lost (Dropped), and sender doesnt 'know' that is
definitive or temporary failure.

If you want faster response, you need to send RST messages, not dropping
frames.

So change your strategy, and add an iptables rule for example ?

On Wed, 17 Oct 2012, Chris Friesen wrote:

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address

I'm a network engineer, as in I work primarily with IP routing. Ever since I started running Linux back in the mid 90ties I've had a love/hate relationship with how Linux handles disappearing network connectivity or IP addresses.

In my mind there are two ways to handle outage:

1. When a network connection (physical interface) goes down, keep everything as it is, it might come back up again shortly and then we can continue as if basically nothing happened. TCP was designed for this considering timeouts can be in hours.

2. When a network connection (physical interface) goes down, wait a few seconds, give up, reset all connectivity related to that connection, basically give up.

Now to my question for the netdev people:

Is there functionality in the kernel for a connection manager to easily accomplish 2, in that when it tries to deconfigure the IP address on the interface to also kill all TCP connections terminated at that IP? On my laptop, I regularily have to kill my ssh client after suspend/resume cycle, because it's been down for quite a while, and the ssh client doesn't know the TCP connection is now not functional anymore (TCP session is still up and retransmit won't happen for a while, so the TCP RST from the server (I use keepalives within SSH) isn't seen for a long time).

Without knowing what's in place right now, I see some behaviours that I'd like to have:

After resume (or otherwise network connectivity re-established), connection manager should be able to tell the kernel to:

a) kill all TCP/UDP/other sessions existing which doesn't currently have an active IP address on the machine. This is for the sake of local clients.
b) the TCP/SCTP sessions that *do* have an IP, should have their retransmit timers "reset", so that whatever needs to be sent, should be sent immediately (or shortly, within a few seconds).
c) tell the kernel to kill all TCP sessions bound to a certain IP, because the connection manager is going to remove it shortly. Send TCP RSTs or whatever and close the TCP session, so both ends know that network connectivity is going down.

This would mean that if I resume my laptop and it establishes network connectivity, I will then *know* within a few seconds what TCP sessions are still valid (the ones that aren't will be killed either because they're bound to an IP that is not available, or a TCP packet is sent out to which there will be a TCP RST reply from the other side if the TCP session is closed at that end).

All of this also has implications on IPv6 and autoconfiguration, I don't know if currently we are closing TCP sessions bound to IPv6 addresses that are going away due to the RA the addresses were autoconfigured based on, now doesn't have a valid lifetime anymore and the kernel is going to remove them. It also applies to both DHCPv6 and DHCPv4 when a lease is expiring and the IPv4/IPv6 address is going to be removed.

Right now I think the situation with a lot of "hanging" TCP sessions after connectivity is restored is suboptimal and negatively impacts user experience. Probably there should be knobs to turn for different user needs (server or desktop/laptop) because desired behaviour is different in different use cases.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

2. When a network connection (physical interface) goes down, wait a few
seconds, give up, reset all connectivity related to that connection,
basically give up.

Doing that is almost pointless and gives people false expectations.

Consider the simple network host-hub-hub-host.
If you disconnect the link between the two hubs then
only timeouts can cause disconnects.
Such a remote fault is much more likely than the local one.

If the kernel takes down connections when a local interface
goes down (as windows does) then people doing testing
fail to test the correct scenario.

You also don't want a power glitch in the wiring cupboard
to disconnect all your connections.

In the past I've also moved IP addresses between physical
interfaces - modern HA systems might do the same.

	David

On Thu, 2012-10-18 at 10:05 +0200, Mikael Abrahamsson wrote:

On Wed, 17 Oct 2012, Chris Friesen wrote:

quoted

1) create new IP address and assign to eth device
2) TCP server starts listening on that IP address
3) TCP client connects to server
4) remove new IP address

I'm a network engineer, as in I work primarily with IP routing. Ever since 
I started running Linux back in the mid 90ties I've had a love/hate 
relationship with how Linux handles disappearing network connectivity or 
IP addresses.

In my mind there are two ways to handle outage:

1. When a network connection (physical interface) goes down, keep 
everything as it is, it might come back up again shortly and then we can 
continue as if basically nothing happened. TCP was designed for this 
considering timeouts can be in hours.

2. When a network connection (physical interface) goes down, wait a few 
seconds, give up, reset all connectivity related to that connection, 
basically give up.

Now to my question for the netdev people:

Is there functionality in the kernel for a connection manager to easily 
accomplish 2, in that when it tries to deconfigure the IP address on the 
interface to also kill all TCP connections terminated at that IP? On my 
laptop, I regularily have to kill my ssh client after suspend/resume 
cycle, because it's been down for quite a while, and the ssh client 
doesn't know the TCP connection is now not functional anymore (TCP session 
is still up and retransmit won't happen for a while, so the TCP RST from 
the server (I use keepalives within SSH) isn't seen for a long time).

Without knowing what's in place right now, I see some behaviours that I'd 
like to have:

After resume (or otherwise network connectivity re-established), 
connection manager should be able to tell the kernel to:

a) kill all TCP/UDP/other sessions existing which doesn't currently have 
an active IP address on the machine. This is for the sake of local 
clients.

You do realize kernel has no idea that the loss of IP address is
temporary or not ?

Some links can be slow to setup after a resume.

b) the TCP/SCTP sessions that *do* have an IP, should have their 
retransmit timers "reset", so that whatever needs to be sent, should be 
sent immediately (or shortly, within a few seconds).

So they are going to force a close, even if the link becomes alive after
15 seconds. Too bad for some wireless setups, or tunnels.

c) tell the kernel to kill all TCP sessions bound to a certain IP, because 
the connection manager is going to remove it shortly. Send TCP RSTs or 
whatever and close the TCP session, so both ends know that network 
connectivity is going down.

Yes, why not. Why is it specific to linux I have no idea.

On Thu, 18 Oct 2012, Eric Dumazet wrote:

quoted

After resume (or otherwise network connectivity re-established),
connection manager should be able to tell the kernel to:

a) kill all TCP/UDP/other sessions existing which doesn't currently have
an active IP address on the machine. This is for the sake of local
clients.

You do realize kernel has no idea that the loss of IP address is
temporary or not ?

Indeed, that's why I wrote "connection manager tell the kernel to..", and didn't write "the kernel should do".

Some links can be slow to setup after a resume.

Absolutely.

quoted

b) the TCP/SCTP sessions that *do* have an IP, should have their
retransmit timers "reset", so that whatever needs to be sent, should be
sent immediately (or shortly, within a few seconds).

So they are going to force a close, even if the link becomes alive after
15 seconds. Too bad for some wireless setups, or tunnels.

I don't get what you're saying. I'm saying reset the retransmit timers, if there is no response, exponential backoff applies again. The reset of the timers is to avoid that the exponential backoff timer is now in "wait for several minutes to send the next packet" due to no connectivity during the first 10-20 seconds after coming out of resume.

On my Ubuntu 12.04 system, when I bring it out of resume and press enter in the terminal with an established ssh session, it takes minutes to discover that this session doesn't work and close it (it seems the TCP RST from the server isn't triggered for a long time). I usually end up killing the ssh process on the laptop and logging in again. What I want to happen is that within a few seconds of network connectivity being established, TCP should try to communicate and discover that the other end already closed the connection and close the local socket call so ssh quits and I can re-login again. One way of doing this would be for the connection manager to ask the kernel to reset (or severely lower) the retransmit timers on all established TCP connections.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

On Thu, 18 Oct 2012, David Laight wrote:

Doing that is almost pointless and gives people false expectations.

Consider the simple network host-hub-hub-host.
If you disconnect the link between the two hubs then
only timeouts can cause disconnects.
Such a remote fault is much more likely than the local one.

My doing suspect/resume on my laptop causes more network interruptions than any link going down in the network outside my home.

If the kernel takes down connections when a local interface
goes down (as windows does) then people doing testing
fail to test the correct scenario.

I never asked for the kernel to take down connections, I asked for the connection manager to be able to do it.

You also don't want a power glitch in the wiring cupboard to disconnect all your connections.

In a server environment, most likely you're right. In some other environment, the opposite probably applies.

In the past I've also moved IP addresses between physical
interfaces - modern HA systems might do the same.

Yes, in your described scenario you're right, in my scenario (laptop suspend/resume) I don't agree with your view.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

On Thu, Oct 18, 2012 at 11:13:29AM +0200, Eric Dumazet wrote:

quoted

c) tell the kernel to kill all TCP sessions bound to a certain IP, because 
the connection manager is going to remove it shortly. Send TCP RSTs or 
whatever and close the TCP session, so both ends know that network 
connectivity is going down.

Yes, why not.

FYI the idea was by Andi Kleen back in 2003. If was flag IFF_DYNAMIC
on device (apparently, it should be per-interface sysctl instead
or even a flag on specific address).

Andi suggested to hook netdev notifier and to reset tcp connections
bound to addresses on this interface. He did not go so far to send
resets before address is actually disabled (it was not a goal, normally
address is already dead to the time when it is deleted),
but techically it is the same.

The problem with this was purely technical, the code has to scan through
all the tcp hash table to search for connections to this address (grrr already :-))
and to take socket lock before making any actions. It is doable, but quite chumbersome
and nobody was interested enough to finish the job.

On Thu, 18 Oct 2012, Alexey Kuznetsov wrote:

Andi suggested to hook netdev notifier and to reset tcp connections bound to addresses on this interface. He did not go so far to send resets before address is actually disabled (it was not a goal, normally address is already dead to the time when it is deleted), but techically it is the same.

With IPv6 and temporary addresses etc, and DHCPv4/v6 leases expiring, addresses going away can be a quite foreseeable event. When I suspend my laptop, the IP going away is also quite forseeable.

The problem with this was purely technical, the code has to scan through all the tcp hash table to search for connections to this address (grrr already :-)) and to take socket lock before making any actions. It is doable, but quite chumbersome and nobody was interested enough to finish the job.

Are there hooks so that someone could write a userland application to do this job? I could imagine the connection manager doing this job as well (so it's configurable what connection to close or not).

What about the reset of the TCP retransmit timers, is that doable from userland? Or at least lowering them to 1s or something fairly sane (going to fast-retransmit doesn't make sense, neither does having a retransmit timer in the 10s of minutes).

Btw, what are the TCP retransmit timers when I come out of suspend? The machine has been "offline" for hours, what idea does TCP have of what's going on, will it keep the timers it had when machine went down or does TCP believe it hasn't seen a response for a long time and now the retransmit timers are very high?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Chris Friesen [off-list ref] writes:

7) client waits for response from server, and waits forever or until
keepalive expires

Please don't fix that. I sometimes move my laptop from one ethernet
socket to another. During the move I lose my DHCP-assigned IP address.

I do not want to lose my SSH sessions too.


/Benny

On Thu, 18 Oct 2012, Benny Amorsen wrote:

Chris Friesen [off-list ref] writes:

quoted

7) client waits for response from server, and waits forever or until
keepalive expires

Please don't fix that. I sometimes move my laptop from one ethernet
socket to another. During the move I lose my DHCP-assigned IP address.

Would you mind if it was user configurable?

I do not want to lose my SSH sessions too.

Would you mind if the TCP retransmit timers were severely lowered upon reconnection so your TCP sessions started working quicker? Or you don't see this problem at all?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Mikael Abrahamsson [off-list ref] writes:

Would you mind if it was user configurable?

No, I would not mind. But I am the wrong person to ask that question, I
do not have to deal with supporting configuration options.

Would you mind if the TCP retransmit timers were severely lowered upon
reconnection so your TCP sessions started working quicker? Or you
don't see this problem at all?

I would not mind that at all. I do not particularly recall seeing the
problem, but that sounds like a great idea.


/Benny

On 10/18/2012 01:29 AM, Eric Dumazet wrote:

On Thu, 2012-10-18 at 01:08 -0600, Chris Friesen wrote:

quoted

While I agree generally, it's a bit unfortunate that we can't (as a
quality of implementation thing) give an earlier notice of failure since
the kernel knows about both ends of the connection even though the IP
address is gone.  On the other hand, I imagine that would mean
special-casing things and presumably that would open a whole can of worms.

Really what is the difference between a cable cut and what you are
doing ?

Some frames are lost (Dropped), and sender doesnt 'know' that is
definitive or temporary failure.

If you want faster response, you need to send RST messages, not dropping
frames.

After thinking about it for a while, I think you're right.  Initially I was expecting that since we know the server side has been taken down we should be able to kill the client side, but then I considered the case where some sort of of high availability system may have moved the server to another host.

So change your strategy, and add an iptables rule for example ?

That's a good suggestion.  I'll pass it on.

Thanks,
Chris

Eric Dumazet [off-list ref] writes:

I see no real problem here.

Its like you cut the cable somewhere in the path.

I had a patch for this a long time ago to speed up recovery.
It was very useful with dial on demand ISDN links or DSL
connections that hang up occasionally to force a new IP.

It was in SUSE kernels for a long time, but never made it 
into mainline

http://www.firstfloor.org/pub/ak/quilt-before-arch-merge/patches/iff-dynamic

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only