[BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks

Hi,

I'm not sure how it behaved in earlier kernels (can check later), but it 
is / looks bugged in at least recent 5.x+ ones (tests were done with 
5.11.8 and 5.10.25).

Consider following setup:

# ip -o ad sh
1: lo    inet 127.0.0.1/8 scope host lo
2: right1    inet 10.0.10.2/24 scope global
3: right2    inet 10.0.20.2/24 scope global

# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2

# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2

And routing rules:

0:      from all lookup local
9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10:     from all ipproto udp sport 1194 lookup 123
32766:  from all lookup main
32767:  from all lookup default

This - without any mangling via ipt/nft or by other means - works 
correctly, for example:

nc -u -p 1194 1.2.3.4 12345

will be routed out correctly via 'right2' using 10.0.20.2

But if we add mark to locally outgoing packets:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

Then *both* rule 9 and rule 10 will be ignored during reroute check. 
tcpdump on interface 'right1' will show:

# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot 
length 262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], 
proto UDP (17), length 33)
     10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5

Initial routing decision in rule 10 will set the address correctly, but 
the packet goes out via interface right1, ignoring both 9 and 10.

If I add another routing roule:

8:      from all fwmark 0x1 lookup 123

Then the packects will flow correctly - but I *cannot* use (from the 
ones I tested): sport, dport, ipproto, uidrange - as they will cause the 
rule to be ignored. For example, this setup of routing rules will fail, 
if there is any mark set on a packet (nc had uid 1120):

# ip ru sh
0:      from all lookup local
10:     from all ipproto udp lookup 123
10:     from all sport 1194 lookup 123
10:     from all dport 12345 lookup 123
10:     from all uidrange 1120-1120 lookup 123
32766:  from all lookup main
32767:  from all lookup default

Adding correct fwmark to the above rules will have *no* effect either. 
Only fwmark *alone* will work (or in combination with: iif, from, to - 
from the ones I tested).

I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't 
look like there is anything wrong there. I initially suspected lack of 
'rule->mark &&' in mark related line - but considering that rules such 
as 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look 
like it's the culprit (and mark_mask covers that test either way).

OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the 
culprit here - but I haven't analyzed them yet. Perhaps it's just an 
issue of changing output interface incorrectly after ip_route_me_harder() ?

Is this a bug ? Or am I misinterpreting how 'reroute check' works after 
initial routing decision ? One would expect routing rules during 
post-mangle check to not be ignored out of the blue, only because packet 
mark changed on the packet. Not mentioning both marks and routing rules 
can be used for separate purposes (e.g. marks for shaping).

On 3/28/21 4:05 PM, Michal Soltys wrote:

Hi,

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

After some extra checks, the exact same issue happens if we mangle tos 
instead of mark, so ...

iptables -t mangle -A OUTPUT -j TOS --set-tos 0x02

... will cause same routing issues with incorrect output interface being 
set.

Anyway, I've attached 2 scripts setting up namespace with all those cases.

- run setup-host.sh
- then in the namespace: run setup-namespace.sh
- fire up another bash in ns with tcpdump on right1

nc -u -p 1194 1.2.3.4 12345

This will work fine, as the packet will be routed correctly via right2, 
tcpdump will show nothing

Now add in the namespace:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

or

iptables -t mangle -A OUTPUT -j TOS --set-tos 0x02

Same nc as above - now the packet will go out via right1, using right2's 
address (initial routing decision).


Unrelated issue - while doing the tests I noticed that routing rules 
based on tos have no effect at all for locally generated packets. Will 
make another post about it though.

On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:

Hi,

I'm not sure how it behaved in earlier kernels (can check later), but it is
/ looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
5.10.25).

Consider following setup:

# ip -o ad sh
1: lo    inet 127.0.0.1/8 scope host lo
2: right1    inet 10.0.10.2/24 scope global
3: right2    inet 10.0.20.2/24 scope global

# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2

# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2

And routing rules:

0:      from all lookup local
9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10:     from all ipproto udp sport 1194 lookup 123
32766:  from all lookup main
32767:  from all lookup default

This - without any mangling via ipt/nft or by other means - works correctly,
for example:

nc -u -p 1194 1.2.3.4 12345

will be routed out correctly via 'right2' using 10.0.20.2

But if we add mark to locally outgoing packets:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
on interface 'right1' will show:

# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
UDP (17), length 33)
    10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5

Initial routing decision in rule 10 will set the address correctly, but the
packet goes out via interface right1, ignoring both 9 and 10.

If I add another routing roule:

8:      from all fwmark 0x1 lookup 123

Then the packects will flow correctly - but I *cannot* use (from the ones I
tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
ignored. For example, this setup of routing rules will fail, if there is any
mark set on a packet (nc had uid 1120):

# ip ru sh
0:      from all lookup local
10:     from all ipproto udp lookup 123
10:     from all sport 1194 lookup 123
10:     from all dport 12345 lookup 123
10:     from all uidrange 1120-1120 lookup 123
32766:  from all lookup main
32767:  from all lookup default

Adding correct fwmark to the above rules will have *no* effect either. Only
fwmark *alone* will work (or in combination with: iif, from, to - from the
ones I tested).

I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
like there is anything wrong there. I initially suspected lack of
'rule->mark &&' in mark related line - but considering that rules such as
'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
it's the culprit (and mark_mask covers that test either way).

OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
here - but I haven't analyzed them yet. Perhaps it's just an issue of
changing output interface incorrectly after ip_route_me_harder() ?

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

Is this a bug ? Or am I misinterpreting how 'reroute check' works after
initial routing decision ? One would expect routing rules during post-mangle
check to not be ignored out of the blue, only because packet mark changed on
the packet. Not mentioning both marks and routing rules can be used for
separate purposes (e.g. marks for shaping).

On Mon, Mar 29, 2021 at 11:52:10PM +0300, Ido Schimmel wrote:

On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:

quoted

Hi,

I'm not sure how it behaved in earlier kernels (can check later), but it is
/ looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
5.10.25).

Consider following setup:

# ip -o ad sh
1: lo    inet 127.0.0.1/8 scope host lo
2: right1    inet 10.0.10.2/24 scope global
3: right2    inet 10.0.20.2/24 scope global

# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2

# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2

And routing rules:

0:      from all lookup local
9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10:     from all ipproto udp sport 1194 lookup 123
32766:  from all lookup main
32767:  from all lookup default

This - without any mangling via ipt/nft or by other means - works correctly,
for example:

nc -u -p 1194 1.2.3.4 12345

will be routed out correctly via 'right2' using 10.0.20.2

But if we add mark to locally outgoing packets:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
on interface 'right1' will show:

# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
UDP (17), length 33)
    10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5

Initial routing decision in rule 10 will set the address correctly, but the
packet goes out via interface right1, ignoring both 9 and 10.

If I add another routing roule:

8:      from all fwmark 0x1 lookup 123

Then the packects will flow correctly - but I *cannot* use (from the ones I
tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
ignored. For example, this setup of routing rules will fail, if there is any
mark set on a packet (nc had uid 1120):

# ip ru sh
0:      from all lookup local
10:     from all ipproto udp lookup 123
10:     from all sport 1194 lookup 123
10:     from all dport 12345 lookup 123
10:     from all uidrange 1120-1120 lookup 123
32766:  from all lookup main
32767:  from all lookup default

Adding correct fwmark to the above rules will have *no* effect either. Only
fwmark *alone* will work (or in combination with: iif, from, to - from the
ones I tested).

I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
like there is anything wrong there. I initially suspected lack of
'rule->mark &&' in mark related line - but considering that rules such as
'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
it's the culprit (and mark_mask covers that test either way).

OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
here - but I haven't analyzed them yet. Perhaps it's just an issue of
changing output interface incorrectly after ip_route_me_harder() ?

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

It's already tomorrow here, but I think that if you record the
'fib:fib_table_lookup' tracepoint before and after adding the mangling
rules you will see that there is a second lookup for the packet with
zero source / destination port. Something like:

# perf record -a -e fib:fib_table_lookup -- sleep 5
# perf script --stdio

quoted

Is this a bug ? Or am I misinterpreting how 'reroute check' works after
initial routing decision ? One would expect routing rules during post-mangle
check to not be ignored out of the blue, only because packet mark changed on
the packet. Not mentioning both marks and routing rules can be used for
separate purposes (e.g. marks for shaping).

On 3/29/21 10:52 PM, Ido Schimmel wrote:

On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:

quoted

Hi,

I'm not sure how it behaved in earlier kernels (can check later), but it is
/ looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
5.10.25).

Consider following setup:

# ip -o ad sh
1: lo    inet 127.0.0.1/8 scope host lo
2: right1    inet 10.0.10.2/24 scope global
3: right2    inet 10.0.20.2/24 scope global

# ip ro sh tab main
default via 10.0.10.1 dev right1
10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2

# ip ro sh tab 123
default via 10.0.20.1 dev right2 src 10.0.20.2

And routing rules:

0:      from all lookup local
9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
10:     from all ipproto udp sport 1194 lookup 123
32766:  from all lookup main
32767:  from all lookup default

This - without any mangling via ipt/nft or by other means - works correctly,
for example:

nc -u -p 1194 1.2.3.4 12345

will be routed out correctly via 'right2' using 10.0.20.2

But if we add mark to locally outgoing packets:

iptables -t mangle -A OUTPUT -j MARK --set-mark 1

Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
on interface 'right1' will show:

# tcpdump -nvi right1 udp
tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
262144 bytes
13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
UDP (17), length 33)
     10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5

Initial routing decision in rule 10 will set the address correctly, but the
packet goes out via interface right1, ignoring both 9 and 10.

If I add another routing roule:

8:      from all fwmark 0x1 lookup 123

Then the packects will flow correctly - but I *cannot* use (from the ones I
tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
ignored. For example, this setup of routing rules will fail, if there is any
mark set on a packet (nc had uid 1120):

# ip ru sh
0:      from all lookup local
10:     from all ipproto udp lookup 123
10:     from all sport 1194 lookup 123
10:     from all dport 12345 lookup 123
10:     from all uidrange 1120-1120 lookup 123
32766:  from all lookup main
32767:  from all lookup default

Adding correct fwmark to the above rules will have *no* effect either. Only
fwmark *alone* will work (or in combination with: iif, from, to - from the
ones I tested).

I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
like there is anything wrong there. I initially suspected lack of
'rule->mark &&' in mark related line - but considering that rules such as
'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
it's the culprit (and mark_mask covers that test either way).

OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
here - but I haven't analyzed them yet. Perhaps it's just an issue of
changing output interface incorrectly after ip_route_me_harder() ?

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

Nah, started using ipproto/sport just recently - so only tested with 
5.10+ kernels.

But - I did quick check just a moment ago with 4.19.132 kernel - it's as 
you say, not working there either.

quoted

Is this a bug ? Or am I misinterpreting how 'reroute check' works after
initial routing decision ? One would expect routing rules during post-mangle
check to not be ignored out of the blue, only because packet mark changed on
the packet. Not mentioning both marks and routing rules can be used for
separate purposes (e.g. marks for shaping).

On 3/29/21 10:52 PM, Ido Schimmel wrote:

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

So all the more recent additions to routing rules - src port, dst port, 
uid range and ipproto - are not functioning correctly with the second 
routing check.

Are there plans to eventually fix that ?

While I just adjusted/rearranged my stuff to not rely on those, it 
should probably be at least documented otherwise (presumably in ip-rule 
manpage and perhaps in `ip rule help` as well).

Michal Soltys [off-list ref] wrote:

On 3/29/21 10:52 PM, Ido Schimmel wrote:

quoted

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

So all the more recent additions to routing rules - src port, dst port, uid
range and ipproto - are not functioning correctly with the second routing
check.

Are there plans to eventually fix that ?

While I just adjusted/rearranged my stuff to not rely on those, it should
probably be at least documented otherwise (presumably in ip-rule manpage and
perhaps in `ip rule help` as well).

Fixing this would be better. As Ido implies it should be enough to fully
populate the flow keys in ip(6)_route_me_harder.

On Fri, Apr 09, 2021 at 03:02:41PM +0200, Florian Westphal wrote:

Michal Soltys [off-list ref] wrote:

quoted

On 3/29/21 10:52 PM, Ido Schimmel wrote:

quoted

ip_route_me_harder() does not set source / destination port in the
flow key, so it explains why fib rules that use them are not hit after
mangling the packet. These keys were added in 4.17, but I
don't think this use case every worked. You have a different experience?

So all the more recent additions to routing rules - src port, dst port, uid
range and ipproto - are not functioning correctly with the second routing
check.

Are there plans to eventually fix that ?

While I just adjusted/rearranged my stuff to not rely on those, it should
probably be at least documented otherwise (presumably in ip-rule manpage and
perhaps in `ip rule help` as well).

Fixing this would be better.

Yep.

As Ido implies it should be enough to fully populate the flow keys in
ip(6)_route_me_harder.

Will try to patch this.