On Thu, 21 Jan 2016 14:49:25 +0200 Or Gerlitz [off-list ref] wrote:

On Thu, Jan 21, 2016 at 1:27 PM, Jesper Dangaard Brouer
[off-list ref] wrote:

quoted

On Wed, 20 Jan 2016 15:27:38 -0800 Tom Herbert [off-list ref] wrote:
 

quoted

eth_type_trans touches headers  

True, the eth_type_trans() call in the driver is a major bottleneck,
because it touch the packet header and happens very early in the driver.

In my experiments, where I extract several packet before calling
napi_gro_receive(), and I also delay calling eth_type_trans().  Most of
my speedup comes from this trick, as the prefetch() now that enough
time.

 while ((skb = __skb_dequeue(&rx_skb_list)) != NULL) {
        skb->protocol = eth_type_trans(skb, rq->netdev);
        napi_gro_receive(cq->napi, skb);
 }

What is the HW could provide the info we need in the descriptor?!?


eth_type_trans() does two things:

1) determine skb->protocol
2) setup skb->pkt_type = PACKET_{BROADCAST,MULTICAST,OTHERHOST}

Could the HW descriptor deliver the "proto", or perhaps just some bits
on the most common proto's?

The skb->pkt_type don't need many bits.  And I bet the HW already have
the information.  The BROADCAST and MULTICAST indication are easy.  The
PACKET_OTHERHOST, can be turned around, by instead set a PACKET_HOST
indication, if the eth->h_dest match the devices dev->dev_addr (else a
SW compare is required).

Is that doable in hardware?  

As I wrote earlier, for determination of the eth-type HWs can do what
you ask here and more.

That is great! Is this already being delivered in the descriptor?

Protocol being IP or not (and only then you look in the data) you
could get I guess from many NICs, e.g if the NIC sets PKT_HASH_TYPE_L4
or PKT_HASH_TYPE_L3 then we know it's an IP packets and only if
we don't see this indication we look into the data.

It is a good trick. But at this very early stage we only need the
eth-proto/type.  Once we get to processing the IP layer, then
packet-data will have been pulled / prefetched into L1 cache, thus cost
of determining that should be almost free.

As for pkt_type we can use NIC steering HW to provide us a tag saying
if it was our broadcast, other multicast or "our" unicast.

That would be good. Does that conflict with other programming of the
NIC HW, or can we always have it turned on?

If we can pull this off, then we can do some very interesting cache
latency hiding! :-)  (In my perf top eth_type_trans() is one of the top
contenders, especially for your mlx5 driver).

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer