On 19/10/2021 13:49, Florian Westphal wrote:

David Ahern [off-list ref] wrote:

quoted

Thanks for the detailed summary and possible solutions.

NAT/MASQ rules with VRF were not really thought about during
development; it was not a use case (or use cases) Cumulus or other NOS
vendors cared about. Community users were popping up fairly early and
patches would get sent, but no real thought about how to handle both
sets of rules - VRF device and port devices.

What about adding an attribute on the VRF device to declare which side
to take -- rules against the port device or rules against the VRF device
and control the nf resets based on it?

This would need a way to suppress the NF_HOOK invocation from the
normal IP path.  Any idea on how to do that?  AFAICS there is no way to
get to the vrf device at that point, so no way to detect the toggle.

Or did you mean to only suppress the 2nd conntrack round?

For packets that get forwarded we'd always need to run those in the vrf
context, afaics, because doing an nf_reset() may create a new conntrack
entry (if flow has DNAT, then incoming address has been reversed
already, so it won't match existing REPLY entry in the conntrack table anymore).

For locally generated packets, we could skip conntrack for VRF context
via 'skb->_nfct = UNTRACKED' + nf_reset_ct before xmit to lower device,
and for lower device by eliding the reset entirely.

I think that I have SNAT (at least) working fine with VRFs, without the
commit. What I do is I set notrack at vrf prerouting callback. Could it
be the "proper" way to go? I don't know if I am breaking anything else
though. Here is my reproducer script. SNAT works on kernels without the
"reset conntrack" commit.

(Sorry my Thunderbird inserts extra newlines :( )

========
#!/bin/sudo /bin/bash



for i in 1 2; do

	ip li sh src$i >/dev/null 2>&1 && ip li set src$i nomaster \
		&& ip li del src$i

	ip li sh sink$i >/dev/null 2>&1 && ip li del sink$i

	ip li sh vrf$i >/dev/null 2>&1 && ip li del vrf$i

	ip r flush table 10$i

	ip netns list | grep -q ns$i && ip netns del ns$i

done

nft list table testnat >/dev/null 2>&1 && nft delete table testnat



case $1 in

        clean)  echo "cleaned up"; exit 0;;

esac



sysctl -w net.ipv4.ip_forward=1



for i in 1 2; do

	ip netns add ns$i

	ip netns exec ns$i ip li set lo up



	ip li add vrf$i type vrf table 10$i

	ip r add vrf vrf$i unreachable default metric 4278198272

	ip li add src$i type veth peer wayout netns ns$i

	ip li set src$i master vrf$i

	ip a add 172.31.$i.1/32 dev src$i

	ip li set src$i up

	ip li set vrf$i up

	#/sbin/sysctl -w net.ipv4.conf.src$i.accept_local=1

	ip netns exec ns$i ip a add 172.31.$i.2/24 dev wayout

	ip netns exec ns$i ip li set wayout up

	ip netns exec ns$i ip r add default via 172.31.$i.1



	ip li add sink$i type veth peer wayin netns ns$i

	ip netns exec ns$i ip li set wayin up

	ip netns exec ns$i /sbin/sysctl -w net.ipv4.conf.wayin.rp_filter=0

	ip li set sink$i up

done



ip r add 172.31.1.0/24 dev sink1 table 102

ip r add 172.31.2.0/24 dev sink2 table 101



ip r add 100.64.0.0/24 dev sink1 table 102



nft -f - <<__END__

table testnat {

        chain rawpre {

                type filter hook prerouting priority raw;

                #iif { src1, src2 } meta nftrace set 1

                iif { src1, src2 } ct zone set 1 return

                notrack

        }

        chain rawout {

                type filter hook output priority raw;

                notrack

        }

	chain natpost {

		type nat hook postrouting priority srcnat;

		oif sink2 snat ip to 100.64.0.2

	}

}

__END__



conntrack -F

ip netns exec ns2 tcpdump -lni wayin arp or icmp &

tdpid=$!

sleep 1

ip netns exec ns1 ping -W 1 -c 1 172.31.2.2

conntrack -L

sleep 1

kill $tdpid

wait