Re: [bug] cxgb4: vrf stopped working with cxgb4 card

On 19.06.2018 16:24, Ganesh Goudar wrote:

On Monday, June 06/11/18, 2018 at 14:47:55 +0530, Ganesh Goudar wrote:

quoted

On Saturday, June 06/09/18, 2018 at 18:47:55 -0600, David Ahern wrote:

quoted

Ganesh:

On 6/4/18 9:03 AM, AMG Zollner Robert wrote:

quoted

I have noticed that vrf is not working with kernel v4.15.0 but was
working with v4.13.0 when using cxgb4 Chelsio driver (T520-cr)

Setup:
Two metal servers with a T520-cr card each, directly connected without a
switch in between.

        SVR1  only ipfwd                 SVR2     with vrf
.----------------------------. .----------------------------------.
|                            |         |             |
|    192.168.8.1 [  ens2f4]--|---------|--[ens1f4] 192.168.8.2   |
|    192.168.9.1 [ens2f4d1]--|---------|--<ens1f4d1> 192.168.9.2 VRF=10   |
`----------------------------' `----------------------------------'

When vrf is not working there are no error messages (dmesg or iproute
commands), tcpdump on the interface (SVR2.ens1f4d1) enslaved in vrf 10
shows packets(arp req/reply) coming in and going out, but outgoing
packets(arp reply) do not reach the other server SVR1.ens2f4d1


Bisect:
Found this commit to be the problem after doing a git bisect between
v4.13..v4.15:

commit ba581f77df23c8ee70b372966e69cf10bc5453d8
Author: Ganesh Goudar [off-list ref]
Date:   Sat Sep 23 16:07:28 2017 +0530

     cxgb4: do DCB state reset in couple of places

     reset the driver's DCB state in couple of places
     where it was missing.

Are you working on a fix for this or should a revert of the above patch
be sent?

Will look into it and fix/revert it soon, Thanks for responding to Robert.

quoted

quoted

A bisect step was considered good when:
- successful ping from SVR1 to SVR2.ens1f4d1 vrf interface
- successful ping from SVR2 global to SVR2 vrf interface trough SVR1(l3
forwarding) (this check was redundant,both tests fail or pass simultaneous)

The problem is still present on recent kernels also, checked v4.16.0 and
v4.17.rc7

Disabling DCB for the card support fixes the problem ( Compiling kernel
with "CONFIG_CHELSIO_T4_DCB=n")



This is my first time reporting a bug to the linux kernel and hope I
have included the right amount of information. Please let me know if I
have missed something.



Thank you,
Zollner Robert


--------
Logs:

VRF configured using folowing commands:

#!/bin/sh

CHDEV=ens1f4
VRF=vrf-recv

sysctl -w net.ipv4.tcp_l3mdev_accept=1
sysctl -w net.ipv4.udp_l3mdev_accept=1
sysctl -w net.ipv4.conf.all.accept_local=1

ifconfig ${CHDEV}   192.168.8.2/24
ifconfig ${CHDEV}d1 192.168.9.2/24

ip link add ${VRF} type vrf table 10
ip link set dev ${VRF} up

ip rule add pref 32765 table local
ip rule del pref 0

ip route add table 10 unreachable default metric 4278198272

ip link set dev ${CHDEV}d1 master ${VRF}

ip route add table 10 default via 192.168.9.1
ip route add 192.168.9.0/24 via 192.168.8.1

-netdev, Please feel free to add if needed.

Hi Robert,

My knowledge of VRF is very limited, I am trying to bring
up VRF setup, I just wanted to check if you are doing anything
related DCB and also please let me know how did you setup SRV1.

Thanks

Hello Ganesh,

SRV1 is just forwarding(l3) between the two physical ports of the 
T520-CR card.

ifconfig ens1f4   192.168.8.1/24
ifconfig ens1f4d1 192.168.9.1/24
sysctl -w net.ipv4.ip_forward=1

- No VRF is configured on this box
- DCB is also not used


SVR2 is using VRF and is configured with the script inlined in the first 
email.