Kris Op de Beeck wrote:

quoted

What does "grep <srcport from above> /proc/net/nf_conntrack" show
when the problem occurs?

[ 1976.495472] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.29 DST=10.9.9.28 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58096 DF PROTO=TCP SPT=41675 DPT=80 SEQ=3967333855 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00065A1E0000000001030305) UID=1000

 sudo grep 41675 /proc/net/nf_conntrack
ipv4     2 tcp      6 43 SYN_RECV src=192.168.1.29 dst=10.9.9.28 sport=41675 dport=80 packets=1 bytes=60 src=192.168.1.1 dst=192.168.1.29 sport=80 dport=41675 packets=3 bytes=180 mark=0 secmark=0 use=1

That looks like the client send a SYN, the server sent three
SYN/ACKs that never reached the client and the client retransmits
its SYN. The SYN should still be NATed, but conntrack thinks
its out of sync because its already in SYN_RECV state, while the
client is apparently still in SYN_SENT state.

Looking back at your first mail:

    print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
    print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
    print "ip ro add table $vlan default dev eth2.$vlan\n";
    print "ip ru add fwmark $vlan table $vlan\n";

This looks like a chicken-and-egg problem. You mark packets based
on the output device, but use the mark to direct them to the output
device.

I guess if you use the source IP for routing table selection it
will work. Not sure why it works at all currently.