[PATCH net] vxlan: update skb dst pmtu on tx path

Unlike ip tunnels, now vxlan doesn't do any pmtu update for
upper dst pmtu, even if it doesn't match the lower dst pmtu
any more.

The problem can be reproduced when reducing the vxlan lower
dev's pmtu when running netperf. In jianlin's testing, the
performance went to 1/7 of the previous.

This patch is to update the upper dst pmtu to match the lower
dst pmtu on tx path so that packets can be sent out even when
lower dev's pmtu has been changed.

It also works for metadata dst.

Note that this patch doesn't process any pmtu icmp packet.
But even in the future, the support for pmtu icmp packets
process of udp tunnels will also needs this.

The same thing will be done for geneve in another patch.

Signed-off-by: Xin Long <lucien.xin@gmail.com>
---
 drivers/net/vxlan.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index 19b9cc5..266665b 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c

@@ -2155,6 +2155,13 @@ static void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
 		}
 
 		ndst = &rt->dst;
+		if (skb_dst(skb)) {
+			int mtu = dst_mtu(ndst) - VXLAN_HEADROOM;
+
+			skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
+						       skb, mtu);
+		}
+
 		tos = ip_tunnel_ecn_encap(tos, old_iph, skb);
 		ttl = ttl ? : ip4_dst_hoplimit(&rt->dst);
 		err = vxlan_build_skb(skb, ndst, sizeof(struct iphdr),

@@ -2190,6 +2197,13 @@ static void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
 				goto out_unlock;
 		}
 
+		if (skb_dst(skb)) {
+			int mtu = dst_mtu(ndst) - VXLAN6_HEADROOM;
+
+			skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
+						       skb, mtu);
+		}
+
 		tos = ip_tunnel_ecn_encap(tos, old_iph, skb);
 		ttl = ttl ? : ip6_dst_hoplimit(ndst);
 		skb_scrub_packet(skb, xnet);

-- 
2.1.0

From: Xin Long <lucien.xin@gmail.com>
Date: Mon, 18 Dec 2017 14:20:56 +0800

Unlike ip tunnels, now vxlan doesn't do any pmtu update for
upper dst pmtu, even if it doesn't match the lower dst pmtu
any more.

The problem can be reproduced when reducing the vxlan lower
dev's pmtu when running netperf. In jianlin's testing, the
performance went to 1/7 of the previous.

This patch is to update the upper dst pmtu to match the lower
dst pmtu on tx path so that packets can be sent out even when
lower dev's pmtu has been changed.

It also works for metadata dst.

Note that this patch doesn't process any pmtu icmp packet.
But even in the future, the support for pmtu icmp packets
process of udp tunnels will also needs this.

The same thing will be done for geneve in another patch.

Signed-off-by: Xin Long <lucien.xin@gmail.com>

Yikes...

You're going to have to find a way to fix this without
invoking ->update_pmtu() on every single transmit.  That's
really excessive, especially for an operation which is
going to be a NOP %99.9999 of the time.

We need some way, instead, for the MTU change event to propagate
properly.  I know this might be hard, but doing this in the transmit
handler on every packet to deal with it is not the way to go.

Thanks.

On Wed, Dec 20, 2017 at 12:12 AM, David Miller [off-list ref] wrote:

From: Xin Long <lucien.xin@gmail.com>
Date: Mon, 18 Dec 2017 14:20:56 +0800

quoted

Unlike ip tunnels, now vxlan doesn't do any pmtu update for
upper dst pmtu, even if it doesn't match the lower dst pmtu
any more.

The problem can be reproduced when reducing the vxlan lower
dev's pmtu when running netperf. In jianlin's testing, the
performance went to 1/7 of the previous.

This patch is to update the upper dst pmtu to match the lower
dst pmtu on tx path so that packets can be sent out even when
lower dev's pmtu has been changed.

It also works for metadata dst.

Note that this patch doesn't process any pmtu icmp packet.
But even in the future, the support for pmtu icmp packets
process of udp tunnels will also needs this.

The same thing will be done for geneve in another patch.

Signed-off-by: Xin Long <lucien.xin@gmail.com>

Yikes...

You're going to have to find a way to fix this without
invoking ->update_pmtu() on every single transmit.  That's
really excessive, especially for an operation which is
going to be a NOP %99.9999 of the time.

understand, I couldn't find a better way,  and all iptunnels are
doing it in this way.

Or is it possible to go with an unlikely here ?

                if (unlikely(skb_dst(skb) && mtu < dst_mtu(skb_dst(skb))))
                        skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
                                                       skb, mtu);

We need some way, instead, for the MTU change event to propagate
properly.  I know this might be hard, but doing this in the transmit
handler on every packet to deal with it is not the way to go.

how about doing it in vxlan_get_route():

@@ -1896,6 +1896,13 @@ static struct rtable *vxlan_get_route(struct

vxlan_dev *vxlan, struct net_device
                *saddr = fl4.saddr;
                if (use_cache)
                        dst_cache_set_ip4(dst_cache, &rt->dst, fl4.saddr);
+
+               if (skb_dst(skb)) {
+                       int mtu = dst_mtu(ndst) - VXLAN_HEADROOM;
+
+                       skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
+                                                      skb, mtu);
+               }


This would do it only when no dst_cache and it has to do real route lookup.

Note that even when update_pmtu is hit, mostly it will do nothing and
just return
as usually new mtu >= skb_dst(skb)'s pmtu.

Thanks.

From: Xin Long <lucien.xin@gmail.com>
Date: Wed, 20 Dec 2017 01:05:32 +0800

On Wed, Dec 20, 2017 at 12:12 AM, David Miller [off-list ref] wrote:

quoted

You're going to have to find a way to fix this without
invoking ->update_pmtu() on every single transmit.  That's
really excessive, especially for an operation which is
going to be a NOP %99.9999 of the time.

understand, I couldn't find a better way,  and all iptunnels are
doing it in this way.

Or is it possible to go with an unlikely here ?

                if (unlikely(skb_dst(skb) && mtu < dst_mtu(skb_dst(skb))))
                        skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
                                                       skb, mtu);

 ...

Ok, yeah, this is really difficult.

I'll apply your patch for now, but generally speaking we have to handle this
issue better.

On Wed, Dec 20, 2017 at 2:38 AM, David Miller [off-list ref] wrote:

From: Xin Long <lucien.xin@gmail.com>
Date: Wed, 20 Dec 2017 01:05:32 +0800

quoted

On Wed, Dec 20, 2017 at 12:12 AM, David Miller [off-list ref] wrote:

quoted

You're going to have to find a way to fix this without
invoking ->update_pmtu() on every single transmit.  That's
really excessive, especially for an operation which is
going to be a NOP %99.9999 of the time.

understand, I couldn't find a better way,  and all iptunnels are
doing it in this way.

Or is it possible to go with an unlikely here ?

                if (unlikely(skb_dst(skb) && mtu < dst_mtu(skb_dst(skb))))
                        skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
                                                       skb, mtu);

 ...

quoted

how about doing it in vxlan_get_route():

@@ -1896,6 +1896,13 @@ static struct rtable *vxlan_get_route(struct

vxlan_dev *vxlan, struct net_device
                *saddr = fl4.saddr;
                if (use_cache)
                        dst_cache_set_ip4(dst_cache, &rt->dst, fl4.saddr);
+
+               if (skb_dst(skb)) {
+                       int mtu = dst_mtu(ndst) - VXLAN_HEADROOM;
+
+                       skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL,
+                                                      skb, mtu);
+               }


This would do it only when no dst_cache and it has to do real route lookup.

Note that even when update_pmtu is hit, mostly it will do nothing and
just return
as usually new mtu >= skb_dst(skb)'s pmtu.

4 more quoted lines

Thanks Dave.

I'm still thinking about how to support for all udp tunnels icmp packet process,
It's also difficult, as no udp sock for TX.

is it possible we do like the following patch, though I know it's pretty bad
to still update pmtu even if no sock is found. do you have any suggestion
about this ?

diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index e4ff25c..b3d2a50 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c

@@ -608,6 +608,36 @@ static inline bool __udp_is_mcast_sock(struct net

*net, struct sock *sk,
        return true;
 }

+static int udp4_encap_err_handler(struct sk_buff *skb, u32 info)
+{
+       struct iphdr *iph = (struct iphdr *)skb->data;
+       struct net *net = dev_net(skb->dev);
+       int type = icmp_hdr(skb)->type;
+       int code = icmp_hdr(skb)->code;
+
+       switch (type) {
+       case ICMP_DEST_UNREACH:
+               if (code == ICMP_FRAG_NEEDED) {
+                       ipv4_update_pmtu(skb, net, info, 0, 0,
+                                        iph->protocol, 0);
+                       break;
+               } else if (code == ICMP_SR_FAILED) {
+                       break;
+               }
+               return -ENOENT;
+       case ICMP_TIME_EXCEEDED:
+               if (code != ICMP_EXC_TTL)
+                       break;
+               return -ENOENT;
+       case ICMP_REDIRECT:
+               ipv4_redirect(skb, net, 0, 0, iph->protocol, 0);
+       default:
+               break;
+       }
+
 /*
  * This routine is called by the ICMP module when it gets some
  * sort of error condition.  If err < 0 then the socket should

@@ -635,7 +665,8 @@ void __udp4_lib_err(struct sk_buff *skb, u32 info,

struct udp_table *udptable)
                               iph->saddr, uh->source, skb->dev->ifindex, 0,
                               udptable, NULL);
        if (!sk) {
-               __ICMP_INC_STATS(net, ICMP_MIB_INERRORS);
+               if (udp4_encap_err_handler(skb, info))
+                       __ICMP_INC_STATS(net, ICMP_MIB_INERRORS);
                return; /* No socket for error */
        }

On 12/18/2017 07:20 AM, Xin Long wrote:

Unlike ip tunnels, now vxlan doesn't do any pmtu update for
upper dst pmtu, even if it doesn't match the lower dst pmtu
any more.

The problem can be reproduced when reducing the vxlan lower
dev's pmtu when running netperf. In jianlin's testing, the
performance went to 1/7 of the previous.

This patch is to update the upper dst pmtu to match the lower
dst pmtu on tx path so that packets can be sent out even when
lower dev's pmtu has been changed.

It also works for metadata dst.

Sorry for resurrecting the conversation, but what exactly was meant by this "works metadata dst" remark? I am currently having kernel crashes related to ops->update_pmtu == NULL on VXLAN xmit and I am using tunnel metadata set by act_tunnel_key. Before this patch, it was OK.

My first thought was to add `if (ops->update_pmtu != NULL)`, but it appears some thought was given to this. Or maybe md_dst_ops should have update pmtu?

Thanks, Roman Kapl

On Tue, Jan 23, 2018 at 6:35 AM, Roman Kapl [off-list ref] wrote:

On 12/18/2017 07:20 AM, Xin Long wrote:

quoted

Unlike ip tunnels, now vxlan doesn't do any pmtu update for
upper dst pmtu, even if it doesn't match the lower dst pmtu
any more.

The problem can be reproduced when reducing the vxlan lower
dev's pmtu when running netperf. In jianlin's testing, the
performance went to 1/7 of the previous.

This patch is to update the upper dst pmtu to match the lower
dst pmtu on tx path so that packets can be sent out even when
lower dev's pmtu has been changed.

It also works for metadata dst.

Sorry for resurrecting the conversation, but what exactly was meant by this
"works metadata dst" remark? I am currently having kernel crashes related to
ops->update_pmtu == NULL on VXLAN xmit and I am using tunnel metadata set by
act_tunnel_key. Before this patch, it was OK.

You are right, it is introudced by this patch.

Jiri Benc mentioned about this patch on md_dst, then I wrote a test
case with ip route
running over it. It worked well, now I realized I cleaned up the env too fast.

Pls also fix it on geneve_xmit. thanks.

My first thought was to add `if (ops->update_pmtu != NULL)`, but it appears
some thought was given to this. Or maybe md_dst_ops should have update pmtu?

md_dst_ops doesn't support pmtu, I had no plans for it yet by now.

Thanks, Roman Kapl