I have put a v2 here which addresses that by retaining which bridge
the port was added to and comparing that against the bridge net_device
we want to join:

https://github.com/ffainelli/linux/tree/dsa-hw-bridge-v2

Hi Florian

This looks O.K.

If we ever get a switch which does not allow disjoint sets of ports,
it will need some changes, but lets not over engineer it now.

   Andrew