I think that it is worth doing, as the original kernel change broke my user space program and could do the same to others as well.
On another setup, even with the DIVERT rule in place, I'm still seeing the RST after the ACK. I'm not sure how it is behaving differently than the other setup so I need to look into that. But it definitely worked before the changes to the kernel.
From: Florian Westphal <fw@strlen.de>
Sent: Tuesday, August 2, 2016 3:11 PM
To: Brandon Cazander
Cc: Florian Westphal
Subject: Re: PROBLEM: TPROXY and DNAT broken (bisected to 079096f103fa)
Brandon Cazander [off-list ref] wrote:
quoted
Please try this patch, it makes it work for me again.
I decided to extend the existing snat support in xt_socket.c instead
of changing TPROXY target:
This fixes my example (with the DIVERT chain), but does not fix the two-line example you gave below. Another setup I have is also still broken as of this diff (similarly, there is a rule in nat PREROUTING that goes to a chain with the TPROXY rule).
Yes, I did not touch TPROXY target, we would need something similar
(take tuple addresses from the conntrack entry) there as well if we
need to make it work without the -m socket rule.
Is there anything I can provide or do to help get this issue fixed? Even with the patch provided, our application is still broken.
-Brandon
From: Brandon Cazander
Sent: Wednesday, August 3, 2016 8:47 AM
To: Florian Westphal
Cc: netdev@vger.kernel.org
Subject: Re: PROBLEM: TPROXY and DNAT broken (bisected to 079096f103fa)
I think that it is worth doing, as the original kernel change broke my user space program and could do the same to others as well.
On another setup, even with the DIVERT rule in place, I'm still seeing the RST after the ACK. I'm not sure how it is behaving differently than the other setup so I need to look into that. But it definitely worked before the changes to the kernel.
From: Florian Westphal <fw@strlen.de>
Sent: Tuesday, August 2, 2016 3:11 PM
To: Brandon Cazander
Cc: Florian Westphal
Subject: Re: PROBLEM: TPROXY and DNAT broken (bisected to 079096f103fa)
Brandon Cazander [off-list ref] wrote:
quoted
Please try this patch, it makes it work for me again.
I decided to extend the existing snat support in xt_socket.c instead
of changing TPROXY target:
This fixes my example (with the DIVERT chain), but does not fix the two-line example you gave below. Another setup I have is also still broken as of this diff (similarly, there is a rule in nat PREROUTING that goes to a chain with the TPROXY rule).
Yes, I did not touch TPROXY target, we would need something similar
(take tuple addresses from the conntrack entry) there as well if we
need to make it work without the -m socket rule.
Is there anything I can provide or do to help get this issue fixed? Even with the patch provided, our application is still broken.
[..]
I think that it is worth doing, as the original kernel change broke my user space program and could do the same to others as well.
On another setup, even with the DIVERT rule in place, I'm still seeing the RST after the ACK. I'm not sure how it is behaving differently than the other setup so I need to look into that. But it definitely worked before the changes to the kernel.
Well, what is different in that setup?
(e.g. rules, application binding different ip address, etc)
Once I can re-create the problem chances for a fix will be a bit
higher...
I can recreate the issue with these rules:
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 8080 -j TPROXY --on-port 9876 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
iptables -t nat -A PREROUTING -d 192.168.7.20/32 -i eth0 -j DNAT --to-destination 192.168.8.1
If I add in the DIVERT chain it works:
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 8080 -j TPROXY --on-port 9876 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
But that's still a regression in my opinion.
Sorry to resurrect this so much later—I just got back from holidays and this was still on my desk.
Will anyone have another chance to look at this? It appears that the DIVERT rule is not working in our case, and I wonder if it is possible to fix the TPROXY target as well as the socket target fix that Florian provided.
It appears as though nobody else has encountered this regression, so I can appreciate that it comes up pretty low on the priority list. If it is not realistic that this will be looked at further, then we will have to look at replacing TPROXY.
Thanks for your time.
From: Brandon Cazander
Sent: Monday, August 15, 2016 9:28 AM
To: Florian Westphal
Cc: netdev@vger.kernel.org; Eric Dumazet
Subject: Re: PROBLEM: TPROXY and DNAT broken (bisected to 079096f103fa)
I can recreate the issue with these rules:
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 8080 -j TPROXY --on-port 9876 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
iptables -t nat -A PREROUTING -d 192.168.7.20/32 -i eth0 -j DNAT --to-destination 192.168.8.1
If I add in the DIVERT chain it works:
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 8080 -j TPROXY --on-port 9876 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
But that's still a regression in my opinion.
Brandon Cazander [off-list ref] wrote:
[ cc netfilter-devel ]
Sorry to resurrect this so much later—I just got back from holidays and this was still on my desk.
Will anyone have another chance to look at this? It appears that the DIVERT rule is not working in our case, and I wonder if it is possible to fix the TPROXY target as well as the socket target fix that Florian provided.
Are there reproducer instructions available for this?
I don't see how TPROXY can be 'fixed' because when skb (tcp syn) is in
mangle PREROUTING nat transformation(s) have not been set up (yet).
So ip header addresses are all we have.
Only the ack (that finishes 3whs) or retransmitted syns will
have the post-nat address info available.
Ack should work fine with (changed) -m socket since the
socket should already be in the main ehash table.
Syn should also work just fine because Erics changes
should not affect initial listener lookup done by TPROXY.
It appears as though nobody else has encountered this regression, so I can appreciate that it comes up pretty low on the priority list. If it is not realistic that this will be looked at further, then we will have to look at replacing TPROXY.
If you already need NAT anyway you can also use -j REDIRECT (or exclude
tproxied packets from nat).