ixgbe patch to provide NIC's tx/rx counters via ethtool

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire.  The Intel 82599 NIC has
registers that keep count of the physical packets.  Add these counters to
the ethtool stats.  The byte counters are 36-bit, but the high 4 bits were
being ignored in the 2.6.31 ixgbe driver:  Read those as well to allow
longer time between polling the stats to detect wraps.

Signed-off-by: Ben Greear <redacted>


Please do not apply this until the ixgbe authors ACK it.  There may
have been reasons for not reading the high 4 bits, or they may dislike
this approach entirely.


Here is ethtool stats output with LRO enabled, with patch applied:

#ethtool -S eth20
NIC statistics:
      rx_packets: 15944000
      tx_packets: 12339293
      rx_bytes: 272306022656
      tx_bytes: 940244184
      rx_pkts_nic: 187747191
      tx_pkts_nic: 12340822
      rx_bytes_nic: 284695533402
      tx_bytes_nic: 989725050
      lsc_int: 3
...

Thanks,
Ben

-- 
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

Ben Greear wrote:

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire. 

When LRO is enabled, are all the bytes on the wire actually transferred into the 
host?

rick jones

Rick Jones wrote:

Ben Greear wrote:

quoted

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire. 

When LRO is enabled, are all the bytes on the wire actually 
transferred into the host?

No...the ethernet, IP and TCP headers and such are not, for packets that 
are combined into a single
large SKB.

That is why the driver counts them wrong.  The bytes are off by a few 
percentage points, but the
packet count is off by an order of magnitude.

Thanks,
Ben

-- 
Ben Greear [off-list ref] 
Candela Technologies Inc  http://www.candelatech.com

Ben Greear wrote:

Rick Jones wrote:

quoted

Ben Greear wrote:

quoted

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire. 

When LRO is enabled, are all the bytes on the wire actually 
transferred into the host?

No...the ethernet, IP and TCP headers and such are not, for packets that 
are combined into a single large SKB.

That is why the driver counts them wrong.  The bytes are off by a few 
percentage points, but the packet count is off by an order of magnitude.

An overly philosphical question perhaps, but are ethtool stats supposed to 
represent what was on the wire, or what entered the host?

rick

On 09/24/2009 09:30 AM, Rick Jones wrote:

Ben Greear wrote:

quoted

Rick Jones wrote:

quoted

Ben Greear wrote:

quoted

When LRO is enabled, the received packet and byte counters represent
the
LRO'd packets, not the packets/bytes on the wire.

When LRO is enabled, are all the bytes on the wire actually
transferred into the host?

No...the ethernet, IP and TCP headers and such are not, for packets
that are combined into a single large SKB.

That is why the driver counts them wrong. The bytes are off by a few
percentage points, but the packet count is off by an order of magnitude.

An overly philosphical question perhaps, but are ethtool stats supposed
to represent what was on the wire, or what entered the host?

They report whatever they report, you get to set custom labels for the values,
and every NIC/driver may be different, so only humans and crazy code like mine that does
specific things based on the driver reported by ethtool should use it.

A more interesting question to me is what netdev-stats tx/rx byte counters should report?

My opinions:
ethernet header (yes)
ethernet CRC  (yes)
ethernet preamble (no)
ethernet frame gap (no)

I think many don't count the CRC, but I haven't looked recently.

Some didn't even report the ethernet header properly a few years ago, but
I think most do now.

When LRO is enabled, it's hard to say if we should report the LRO pkt
stats or the stats on the wire for the netdev-stats.  At least in my case,
I want to report the stats on the wire, but it's also good to see the
LRO stats because you can easily tell that LRO is actually working if you
see low pkts-per-second counters v/s high-bits-per-sec.


Thanks,
Ben

-- 
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

On Wed, 2009-09-23 at 15:36 -0700, Ben Greear wrote:

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire.  The Intel 82599 NIC has
registers that keep count of the physical packets.  Add these counters to
the ethtool stats.  The byte counters are 36-bit, but the high 4 bits were
being ignored in the 2.6.31 ixgbe driver:  Read those as well to allow
longer time between polling the stats to detect wraps.

Signed-off-by: Ben Greear <redacted>


Please do not apply this until the ixgbe authors ACK it.  There may
have been reasons for not reading the high 4 bits, or they may dislike
this approach entirely.

Aside from the trivial line-wrap on the comments, I'm fine with this
patch.  There is no issue I could find with the hardware that would
limit you from reading the high 4 bits.  And since we're reading it
already to clear the register, we might as well use the value we get
from it.

Acked-by: Peter P Waskiewicz Jr <redacted>

On Thu, Sep 24, 2009 at 11:28, Peter P Waskiewicz Jr
[off-list ref] wrote:

On Wed, 2009-09-23 at 15:36 -0700, Ben Greear wrote:

quoted

When LRO is enabled, the received packet and byte counters represent the
LRO'd packets, not the packets/bytes on the wire.  The Intel 82599 NIC has
registers that keep count of the physical packets.  Add these counters to
the ethtool stats.  The byte counters are 36-bit, but the high 4 bits were
being ignored in the 2.6.31 ixgbe driver:  Read those as well to allow
longer time between polling the stats to detect wraps.

Signed-off-by: Ben Greear <redacted>


Please do not apply this until the ixgbe authors ACK it.  There may
have been reasons for not reading the high 4 bits, or they may dislike
this approach entirely.

Aside from the trivial line-wrap on the comments, I'm fine with this
patch.  There is no issue I could find with the hardware that would
limit you from reading the high 4 bits.  And since we're reading it
already to clear the register, we might as well use the value we get
from it.

Acked-by: Peter P Waskiewicz Jr <redacted>

I have added this patch to my tree and will push along with my other
ixgbe patches to dave/netdev.  Thanks.

-- 
Cheers,
Jeff