We already do this and set the skb->packet_type field to
PACKET_HOST, PACKET_BROADCAST, PACKET_MULTICAST etc.
And the input paths test these values.

We still have to pass the packet into netif_rx() to deal
with network taps that want to see packets that are not
destined for us in any way.