[PATCH] tcp resets are misrouted

After commit e2446eaa.. tcp resets are always lost, when routing is asymmetric.
Yes, backing out that patch will result in misrouting of resets for dead connections
which used interface binding when were alive, but we actually cannot do anything here.
What's died that's died and correct handling normal unbound connections is obviously a priority.

Comment to comment:

This has few benefits:
  1. tcp_v6_send_reset already did that.

It was done to route resets for IPv6 link local addresses. It was a mistake to
do so for global addresses. The patch fixes this as well.

Actually, the problem appears to be even more serious than guaranteed loss of resets.
As reported by Sergey Soloviev [off-list ref], those misrouted resets create a lot of
arp traffic and huge amount of unresolved arp entires putting down to knees NAT firewalls
which use asymmetric routing.

Signed-off-by: Alexey Kuznetsov <redacted>
---
 net/ipv4/tcp_ipv4.c |    7 ++++---
 net/ipv6/tcp_ipv6.c |    3 ++-
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 75735c9..ef998b0 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c

@@ -708,10 +708,11 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
 	arg.csumoffset = offsetof(struct tcphdr, check) / 2;
 	arg.flags = (sk && inet_sk(sk)->transparent) ? IP_REPLY_ARG_NOSRCCHECK : 0;
 	/* When socket is gone, all binding information is lost.
-	 * routing might fail in this case. using iif for oif to
-	 * make sure we can deliver it
+	 * routing might fail in this case. No choice here, if we choose to force
+	 * input interface, we will misroute in case of asymmetric route.
 	 */
-	arg.bound_dev_if = sk ? sk->sk_bound_dev_if : inet_iif(skb);
+	if (sk)
+		arg.bound_dev_if = sk->sk_bound_dev_if;
 
 	net = dev_net(skb_dst(skb)->dev);
 	arg.tos = ip_hdr(skb)->tos;

diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 49c8903..26175bf 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c

@@ -877,7 +877,8 @@ static void tcp_v6_send_response(struct sk_buff *skb, u32 seq, u32 ack, u32 win,
 	__tcp_v6_send_check(buff, &fl6.saddr, &fl6.daddr);
 
 	fl6.flowi6_proto = IPPROTO_TCP;
-	fl6.flowi6_oif = inet6_iif(skb);
+	if (ipv6_addr_type(&fl6.daddr) & IPV6_ADDR_LINKLOCAL)
+		fl6.flowi6_oif = inet6_iif(skb);
 	fl6.fl6_dport = t1->dest;
 	fl6.fl6_sport = t1->source;
 	security_skb_classify_flow(skb, flowi6_to_flowi(&fl6));

-- 
1.7.2.3

On Fri, Oct 12, 2012 at 10:34 AM, Alexey Kuznetsov [off-list ref] wrote:

We ran into this as well and pulled that commit from our tree, it was
causing serious problems. Sending the RST back on the iif was
definitely the wrong thing to do. Patch looks good to me.

Thanks,
Debabrata

From: Alexey Kuznetsov <redacted>
Date: Fri, 12 Oct 2012 18:34:17 +0400

After commit e2446eaa.. tcp resets are always lost, when routing is asymmetric.
Yes, backing out that patch will result in misrouting of resets for dead connections
which used interface binding when were alive, but we actually cannot do anything here.
What's died that's died and correct handling normal unbound connections is obviously a priority.

Comment to comment:

quoted

This has few benefits:
  1. tcp_v6_send_reset already did that.

It was done to route resets for IPv6 link local addresses. It was a mistake to
do so for global addresses. The patch fixes this as well.

Actually, the problem appears to be even more serious than guaranteed loss of resets.
As reported by Sergey Soloviev [off-list ref], those misrouted resets create a lot of
arp traffic and huge amount of unresolved arp entires putting down to knees NAT firewalls
which use asymmetric routing.

Signed-off-by: Alexey Kuznetsov <redacted>

Applied and queued up for -stable, thanks!

I was trying to think of the scenario you were running into, since it
falls back to using a route lookup. You do not have routes on your machine
that would correctly send the packet, I assume? Then it simply comes down
to the kernel doesn't have enough information to know where to send the
packet.

Debabrata

On 10/12/12 1:31 PM, "Shawn Lu" [off-list ref] wrote:

I admit the commit e2446eaa did break the reset in asymmetric case.
There are also many use cases in interface binding, we don't want to
break it too... I am ok with your patch, if asymmetric route take higher
priority. Somehow, we need  come up with another one to resolve RST lost
in binding interface case, otherwise, people using interface binding will
find later on that RST break again...

thanks!

Shawn Lu [off-list ref]

quoted

-----Original Message-----
From: Debabrata Banerjee [mailto:dbavatar@gmail.com]
Sent: Friday, October 12, 2012 8:57 AM
To: Alexey Kuznetsov; Banerjee, Debabrata
Cc: netdev@vger.kernel.org; davem@davemloft.net; Shawn Lu;
eric.dumazet@gmail.com; sol@eqv.ru
Subject: Re: [PATCH] tcp resets are misrouted

On Fri, Oct 12, 2012 at 10:34 AM, Alexey Kuznetsov
[off-list ref] wrote:


We ran into this as well and pulled that commit from our
tree, it was causing serious problems. Sending the RST back
on the iif was definitely the wrong thing to do. Patch looks
good to me.

Thanks,
Debabrata

Yes. The scenario is  the kernel doesn't have enough information to routing the packet, so application depend on binding to interface to shortcut routing.  

thanks!

Shawn Lu [off-list ref] 

-----Original Message-----
From: Banerjee, Debabrata [mailto:dbanerje@akamai.com] 
Sent: Friday, October 12, 2012 10:48 AM
To: Shawn Lu; Debabrata Banerjee; Alexey Kuznetsov
Cc: netdev@vger.kernel.org; davem@davemloft.net; 
eric.dumazet@gmail.com; sol@eqv.ru
Subject: Re: [PATCH] tcp resets are misrouted

I was trying to think of the scenario you were running into, 
since it falls back to using a route lookup. You do not have 
routes on your machine that would correctly send the packet, 
I assume? Then it simply comes down to the kernel doesn't 
have enough information to know where to send the packet.

Debabrata

On 10/12/12 1:31 PM, "Shawn Lu" [off-list ref] wrote:

quoted

I admit the commit e2446eaa did break the reset in asymmetric case.
There are also many use cases in interface binding, we don't want to 
break it too... I am ok with your patch, if asymmetric route take 
higher priority. Somehow, we need  come up with another one 

to resolve 

quoted

RST lost in binding interface case, otherwise, people using 

interface 

quoted

binding will find later on that RST break again...

thanks!

Shawn Lu [off-list ref]

quoted

-----Original Message-----
From: Debabrata Banerjee [mailto:dbavatar@gmail.com]
Sent: Friday, October 12, 2012 8:57 AM
To: Alexey Kuznetsov; Banerjee, Debabrata
Cc: netdev@vger.kernel.org; davem@davemloft.net; Shawn Lu; 
eric.dumazet@gmail.com; sol@eqv.ru
Subject: Re: [PATCH] tcp resets are misrouted

On Fri, Oct 12, 2012 at 10:34 AM, Alexey Kuznetsov 
[off-list ref] wrote:


We ran into this as well and pulled that commit from our 

tree, it was 

quoted

quoted

causing serious problems. Sending the RST back on the iif was 
definitely the wrong thing to do. Patch looks good to me.

Thanks,
Debabrata

I admit the commit e2446eaa did break the reset in asymmetric case.  There are also many use cases in interface binding, we don't want to break it too... I am ok with your patch, if asymmetric route take higher priority. Somehow, we need  come up with another one to resolve RST lost in binding interface case, otherwise, people using interface binding will find later on that RST break again...    

thanks!

Shawn Lu [off-list ref] 

-----Original Message-----
From: Debabrata Banerjee [mailto:dbavatar@gmail.com] 
Sent: Friday, October 12, 2012 8:57 AM
To: Alexey Kuznetsov; Banerjee, Debabrata
Cc: netdev@vger.kernel.org; davem@davemloft.net; Shawn Lu; 
eric.dumazet@gmail.com; sol@eqv.ru
Subject: Re: [PATCH] tcp resets are misrouted

On Fri, Oct 12, 2012 at 10:34 AM, Alexey Kuznetsov 
[off-list ref] wrote:

quoted

After commit e2446eaa.. tcp resets are always lost, when 

routing is asymmetric.

quoted

Yes, backing out that patch will result in misrouting of resets for 
dead connections which used interface binding when were 

alive, but we actually cannot do anything here.

quoted

What's died that's died and correct handling normal unbound 

connections is obviously a priority.

quoted

Comment to comment:

quoted

This has few benefits:
  1. tcp_v6_send_reset already did that.

It was done to route resets for IPv6 link local addresses. It was a 
mistake to do so for global addresses. The patch fixes this as well.

Actually, the problem appears to be even more serious than 

guaranteed loss of resets.

quoted

As reported by Sergey Soloviev [off-list ref], those misrouted resets 
create a lot of arp traffic and huge amount of unresolved 

arp entires 

quoted

putting down to knees NAT firewalls which use asymmetric routing.

Signed-off-by: Alexey Kuznetsov <redacted>
---
 net/ipv4/tcp_ipv4.c |    7 ++++---
 net/ipv6/tcp_ipv6.c |    3 ++-
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 

75735c9..ef998b0 100644

--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c

@@ -708,10 +708,11 @@ static void tcp_v4_send_reset(struct 

sock *sk, struct sk_buff *skb)

quoted

        arg.csumoffset = offsetof(struct tcphdr, check) / 2;
        arg.flags = (sk && inet_sk(sk)->transparent) ? 

IP_REPLY_ARG_NOSRCCHECK : 0;

quoted

        /* When socket is gone, all binding information is lost.
-        * routing might fail in this case. using iif for oif to
-        * make sure we can deliver it
+        * routing might fail in this case. No choice here, 

if we choose to force

quoted

+        * input interface, we will misroute in case of 

asymmetric route.

quoted

         */
-       arg.bound_dev_if = sk ? sk->sk_bound_dev_if : inet_iif(skb);
+       if (sk)
+               arg.bound_dev_if = sk->sk_bound_dev_if;

        net = dev_net(skb_dst(skb)->dev);
        arg.tos = ip_hdr(skb)->tos;

diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 

49c8903..26175bf 100644

--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c

@@ -877,7 +877,8 @@ static void tcp_v6_send_response(struct 

sk_buff *skb, u32 seq, u32 ack, u32 win,

quoted

        __tcp_v6_send_check(buff, &fl6.saddr, &fl6.daddr);

        fl6.flowi6_proto = IPPROTO_TCP;
-       fl6.flowi6_oif = inet6_iif(skb);
+       if (ipv6_addr_type(&fl6.daddr) & IPV6_ADDR_LINKLOCAL)
+               fl6.flowi6_oif = inet6_iif(skb);
        fl6.fl6_dport = t1->dest;
        fl6.fl6_sport = t1->source;
        security_skb_classify_flow(skb, flowi6_to_flowi(&fl6));
--
1.7.2.3

We ran into this as well and pulled that commit from our 
tree, it was causing serious problems. Sending the RST back 
on the iif was definitely the wrong thing to do. Patch looks 
good to me.

Thanks,
Debabrata