Paul Moore wrote:

OK I misunderstood. Sorry

You are saying that the port number should be dropped by the pfkey /
xfrm interface - OK
  

Yes. I think thats also why it includes the unique" option.

This is actually what happens. (BTW this is fortunate - in a few cases
racoon accidentally passes down 500)

I meant that the consensus was that the wire behavior is wrong. 

Yes, if the selectors would actually differ, it would be wrong.