Lennert Buytenhek wrote:

In my situation, there are three different sites in the same city
(Amsterdam), interconnected using a shared L2 vlan, and six routers
(A1, A2, B1, B2, C1, C2) on that vlan, two per site for redundancy
reasons.  Each router runs ospf.

The vlan is provided to us by a telco that we do not necessarily trust.
So ideally, we'd like all traffic that goes over the vlan (modulo ARPs
and STP and stuff) to be encrypted.  ("-o eth3 -j ENCRYPT")

The problem I kept running into with tunnel mode is that tunnel
mode SPD rules appear to dictate routing policy in a way that's not
compatible with dynamic routing.

I.e., a line like:

	spdadd 10.10.1.0/24 10.0.1.0/24 any -P out ipsec esp/tunnel/1.2.3.4-5.6.7.8/require;

effectively says "All traffic from 10.10.1.0/24 to 10.0.1.0/24 will
be sent over a tunnel with local endpoint 1.2.3.4 and remote endpoint
5.6.7.8", but:
- I have no idea beforehand what address ranges are going to be routed
  over this vlan.  (Customers might send traffic with source addresses
  in address ranges that they don't announce to us (asymmetric routing),
  and I don't want those packets to remain unencrypted just because they
  don't match the SPD rule.)  A 0.0.0.0/0 0.0.0.0/0 rule would not be
  appropriate either since that'd suck _all_ traffic into this tunnel

You can specify an ifindex for oif in the selector, but you need
to use the xfrm_user interface.

- I have no idea beforehand what the remote nexthop is going to be.  A1
  might ordinarily send its traffic for site B to B1, but if B1 fails
  it'll want to start using B2 instead, which would be prevented by the
  SPD rule hardcoding the remote tunnel endpoint to B1.

The workaround we tried at first was to create GRE tunnels between each
pair of routers on the vlan, and to run ospf over the tunnels instead of
directly over the vlan interface.  That gave MTU problems, though, which
made us just forget about ipsec altogether and use vtund instead, hardly
better than nothing.  (Now that Herbert has submitted a number of fixes
for ipsec MTU issues with tunnels, I guess I should go and give the
GRE-over-ipsec setup a go again.)

Hmm .. sounds like using the routing realm in the selector would
solve this while avoiding the GRE overhead.

Regards
Patrick