Paul Moore wrote:

So how do I get an SA with a port set on it

You don't, using pfkey. Use reqid or ip xfrm.