After debugging a bit into how nf_tproxy_get_sock_v4() looks up a
listener for foreign connections using __inet_lookup_listener() and
compute_score() I found that the problem was happening because of the
way I had the TPROXY rules setup. Those rules work if the proxy has a
single listening socket for INADDR_ANY. If multiple listening sockets
must be used, one for each local address, which is the case for using
SO_BINDTODEVICE then the TPROXY rules must also be "fully qualified"
(with interface name [-i] and IP address thereof [--on-ip]). So the
rules in my example change as follows...

Instead of using:

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 4002

Use:

iptables -t mangle -A PREROUTING *-i eth0* -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 *--on-ip 192.168.0.65* --on-port 4002
iptables -t mangle -A PREROUTING *-i eth1* -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 *--on-ip 192.168.1.65* --on-port 4002

With these rules the listening socket was looked up correctly and the
sk_bound_dev_if of the socket was honored.

- Ashwani

PS: Sorry to anyone (Jamal) who already spent any time on this.
PPS: Copying the tproxy list in case someone else is looking for this
stuff in the future.


On Mon, Nov 30, 2009 at 3:15 PM, Ashwani Wason [off-list ref] wrote:

It appears that SO_BINDTODEVICE is not working when used along with
IP_TRANSPARENT of TPROXY. The use case is for a transparent proxy that
has to handle clients with overlapping IP addresses coming in on
different interfaces (VLANs or physical). Each such interface has a
unique next hop gateway.

Setup: Kernel 2.6.30.9 built with required TPROXY configuration. A
system with three interfaces, eth0 (192.168.0.65/24), eth0.1
(10.0.7.65/16), and eth1 (192.168.1.65/24), each with its own default
router and a fairly standard set of TPROXY rules:

default via 192.168.1.62 dev eth1
default via 192.168.0.62 dev eth0
default via 10.0.0.9 dev eth0.1

iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 --on-port 4002

ip rule add fwmark 1 iif eth0 lookup 101
ip rule add fwmark 1 iif eth1 lookup 101

ip route add local 0.0.0.0/0 dev lo table 101

Additional information:

* The proxy has a separate listening socket for each interface.

* On each listening socket the proxy sets the IP_TRANSPARENT option.
It also sets the SO_BINDTODEVICE option (on each socket) to the
associated interface. (I have experimented with the ordering of these
options and setting one before the other has no change in behavior.)

* The proxy works perfectly fine in "transparent mode" when
SO_BINDTODEVICE is not used.

* The proxy works perfectly fine in "explicit mode" when
SO_BINDTODEVICE *is used*.

Now my test is very simple with the expectation that when a SYN comes
in on, say, interface eth0 then the SYN/ACK must go out back on that
interface to the MAC address of the designated next hop router. That
does not seem to happen however. The SYN/ACK is not being sent out at
all, from any interface. It seems that either the SYN or the SYN/ACK
is being eaten up somewhere. No one 'netstat -s' seems to be updated
either. (Since even 'netstat -ant' does not show the TCP connection
state, I suspect that the SYN is being eaten up.)

When I reconfigure the proxy to be in explicit mode, i.e. not set the
IP_TRANSPARENT option, and have the clients connect directly to the IP
address and port of the proxy then the system works as expected, i.e.
the SYN/ACK (and all subsequent packets) go out on the associated
interface.

Thank you,
Ashwani