Re: Future Direction for rte_eth_stats_get()

Hi Maryam,

I'm not dictating they be re-added (although adding would be nice) but more important I'm trying to express an application view point rather than a driver view point.  

I completely understand how a driver wants to be able to advertise all the stats they want to advertise to help them debug their issues (i.e. xstats).   Yet, I'm very interested in DPDK providing a driver agnostic method of advertising well-defined stats.

For example, what if there was a kind of "stats registry" composed of ID and name.  It would work similar to xtats except instead of advertising strings only the "get" API would return ID/count pairs.  If the application wishes to use the DPDK provided string they can call another API to get the stat string via the ID.  These IDs would be well-defined clearly explaining what the count represent.  This way the strings for counts will be uniform across drivers and also make it clear to the users what the counts truly represent and the application could obtain stats from any driver in a driver agnostic manner.

Just an idea,
Dave

-----Original Message-----
From: Tahhan, Maryam [mailto:maryam.tahhan@intel.com]
Sent: Friday, January 22, 2016 9:18 AM
To: David Harton (dharton) <redacted>; dev@dpdk.org
Cc: Olivier MATZ <redacted>; Van Haaren, Harry
[off-list ref]
Subject: RE: Future Direction for rte_eth_stats_get()

Hi David

+ Olivier and Harry as contributors and advisors in the past on the stats
and stats API.

So I pulled the rtnl_link_stats64 from http://lxr.free-
electrons.com/source/include/uapi/linux/if_link.h#L41 and crossed them
with http://dpdk.org/dev/patchwork/patch/5842/ and
http://dpdk.org/browse/dpdk/tree/lib/librte_ether/rte_ethdev.h

40 /* The main device statistics structure */
 41 struct rtnl_link_stats64 {
 42         __u64   rx_packets;             /* total packets received
*/             <MT> is in struct rte_eth_stats
 43         __u64   tx_packets;             /* total packets transmitted
*/          <MT> is in struct rte_eth_stats
 44         __u64   rx_bytes;               /* total bytes received
*/                  <MT> is in struct rte_eth_stats
 45         __u64   tx_bytes;               /* total bytes transmitted
*/               <MT> is in struct rte_eth_stats
 46         __u64   rx_errors;              /* bad packets received
*/                <MT> I'm working on a patch to distinguish between error
and drops for struct rte_eth_stats at the moment only drops are reported
through ierrors and oerrors and we are tied to those field name for
backward compatibility.
 47         __u64   tx_errors;              /* packet transmit problems
*/           <MT> Same comment as rx_errors.
 48         __u64   rx_dropped;             /* no space in linux buffers
*/          <MT> is in struct rte_eth_stats
 49         __u64   tx_dropped;             /* no space available in linux
*/         <MT> is in struct rte_eth_stats
 50         __u64   multicast;              /* multicast packets received
*/           <MT> was deprecated in struct rte_eth_stats - but we can
remove the notice and expose again from drivers we modified
 51         __u64   collisions; <MT> Not in struct rte_eth_stats - not
sure it's in xstats either...
 52
 53         /* detailed rx_errors: */
 54         __u64   rx_length_errors;
<MT> was deprecated in struct rte_eth_stats, is in xstats - but we can
remove the notice and expose again from drivers we modified
 55         __u64   rx_over_errors;         /* receiver ring buff overflow
*/    <MT> was never exposed to struct rte_eth_stats  - but is in xstats.
 56         __u64   rx_crc_errors;          /* recved pkt with crc error
*/         <MT> was deprecated in struct rte_eth_stats, is in xstats - but
we can remove the notice and expose again from drivers we modified
 57         __u64   rx_frame_errors;        /* recv'd frame alignment
error */ <MT> was never exposed to struct rte_eth_stats - but is in
xstats.
 58         __u64   rx_fifo_errors;         /* recv'r fifo overrun
*/               <MT> was never in struct rte_eth_stats. Not exposed by
all drivers
 59         __u64   rx_missed_errors;       /* receiver missed packet
*/   <MT> was deprecated in struct rte_eth_stats, is in xstats - but we
can remove the notice
 60
 61         /* detailed tx_errors */
 62         __u64   tx_aborted_errors;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 63         __u64   tx_carrier_errors;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 64         __u64   tx_fifo_errors;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 65         __u64   tx_heartbeat_errors;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 66         __u64   tx_window_errors;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 67
 68         /* for cslip etc */
 69         __u64   rx_compressed;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 70         __u64   tx_compressed;
<MT> was never in struct rte_eth_stats. Not exposed by all drivers
 71 };

So what can be enabled again in struct rte_eth_stats  from what was
already there is the equivalent of:
* rx_length_errors
* rx_crc_errors
* rx_missed_errors - the deprecation notice was removed for this field.
* multicast

What should be added in to distinguish between errors and drops. struct
rte_eth_stats :
 * rx_errors
 * tx_errors

As for the detailed rx errors and tx errors I'm open to feedback from you
folks as to what should go in and what is too detailed. These weren't in
struct rte_eth_stats previously, they are available through xstats and are
uniformly named across the drivers. Oliver + Harry any thoughts?

David I assume you are looking for all the missing fields to be added?

Best Regards,
Maryam

quoted

-----Original Message-----
From: David Harton (dharton) [mailto:dharton@cisco.com]
Sent: Friday, January 22, 2016 1:41 PM
To: Tahhan, Maryam <redacted>; dev@dpdk.org
Subject: RE: Future Direction for rte_eth_stats_get()

Hi Maryam,

Thanks for the pointer.  I'll review the convo's.

Consider I have an application that uses many(all?) of the DPDK drivers
and their netmap counterparts depending on configuration.  The user
interface provides a set of I/O stats to help debug I/O issues and that
set is the same regardless of driver type.  The set of stats provided
matches what linux provides today since netmap existed before dpdk.

What I want to avoid is having an application that is driver independent
having to become driver dependent interpreting a bunch of strings
(from xstats) or worse the layer running at the data plane core that is
advertising the API needed by the application parsing those strings
because the application cannot change the upper layer.

What if instead of passing strings and values a set of stat ids and

value

quoted

are returned.  At least this way the application can remain driver
agnostic versus having to parse a free form string that likely isn't the
same across driver types.

Also, why wouldn't rte_eth_stats_get() align with linux which is the
defacto standard?  I understand that not every driver may not support
every stat but that's ok.  Just return 0 (pause frames, crc errors,

etc).

quoted

It's not like the list of linux stats is ever growing or advertising an
absurd number of stats that aren't applicable to all drivers.

Thanks again,
Dave

quoted

-----Original Message-----
From: Tahhan, Maryam [mailto:maryam.tahhan@intel.com]
Sent: Friday, January 22, 2016 6:08 AM
To: David Harton (dharton) <redacted>; dev@dpdk.org
Subject: RE: Future Direction for rte_eth_stats_get()

Hi David
Some of the stats were HW specific rather than generic stats that
should be exposed through rte_eth_stats and were migrated to the

xstats API.

quoted

http://dpdk.org/ml/archives/dev/2015-June/019915.html. The

naming was

quoted

also not generic enough to cover some of the drivers and in some

cases

quoted

what was exposed was only a partial view of the relevant stats.

They were marked as deprecated to deter people from using them

in the

quoted

future, but haven't been removed from all the driver

implementations yet.

quoted

The Registers that remain undeprecated are those considered to be

generic.

quoted

Which registers are you particularly interested in that have been
deprecated? Can you elaborate on what you mean by " scenarios

where a

quoted

static view is expected "

Thanks in advance.

Best Regards,
Maryam

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David

Harton

quoted

quoted

(dharton)
Sent: Wednesday, January 20, 2016 5:19 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Future Direction for rte_eth_stats_get()

I see that some of the rte_eth_stats have been marked

deprecated in

quoted

quoted

2.2 that are returned by rte_eth_stats_get().  Applications that
utilize any number of device types rely on functions like this one
to debug I/O issues.

Is there a reason the stats have been deprecated?  Why not keep

the

quoted

quoted

stats in line with the standard linux practices such as

rtnl_link_stats64?

quoted

Note, using rte_eth_xstats_get() does not help for this particular

scenario

quoted

because a common binary API is needed to communicate through

various

quoted

quoted

layers and also provide a consistent view/meaning to users.  The
xstats is excellent for debugging device specific scenarios but
can't

help

quoted

in scenarios where a static view is expected.

Thanks,
Dave