Re: patch for common networking error messages

On Mon, 16 Jun 2003 21:12:50 CDT, Janice Girouard said:

Okay.  That solves the issue of events generated in a plethora of formats
for the same event.  Any suggestions on what should be included in this new
family?  I can present a patch to suggest a starting point. However, it
would be great to hear from everyone that has any initial thoughts.

Well, at the risk of torquing off any SCO supporters, I'd suggest a quick
peek over at the general design of the AIX errpt/trace systems - in both
cases, data comes out of the kernel in a formatted binary stream, and then
a 'template' file is used to drive the parsing of the formatted data.

Quite slick overall, and nicely extensible - you add a new kernel subsystem
that has more trace points, you just tack its templates onto the end of
the format file and you're good to go....

(And SCO can't even claim that's their code - it's pretty obvious the
parentage of the AIX errpt/trace logging is the OS/VS1 and MVS SMF logging
features :)

Binary interface's will never cut it.  Read the hotplug thread to see how Linus
said, he will never add a binary event daemon interface.

That said, there is an oppurtunity for to provide a ascii interface (similar to
/sbin/hotplug) decodes the data from the rtnetlink interface in a standardized format.

Then it would be easy to write things like perl monitoring scripts that do things
like:
	perl phone-me-if-network-dies.pl < /proc/net/events

Don't flame me about the choce of name. /proc/net/events is not the right name 
to use for such an interface since adding more to /proc is probably not desired.

   From: Stephen Hemminger [off-list ref]
   Date: Tue, 17 Jun 2003 09:08:59 -0700

   Read the hotplug thread to see how Linus
   said, he will never add a binary event daemon interface.

Funny, rtnetlink is exactly this and it is in the tree :-)

Every networking configuration event is transmitted over
rtnetlink sockets to all listeners, in a fixed binary
format.

What Linus doesn't want is this for configuration events,
ie. things like "device appears" not "my ethernet burped"

David S. Miller wrote:

   From: Stephen Hemminger [off-list ref]
   Date: Tue, 17 Jun 2003 09:08:59 -0700

   Read the hotplug thread to see how Linus
   said, he will never add a binary event daemon interface.

Funny, rtnetlink is exactly this and it is in the tree :-)

...and it's been in the tree for quite a while too.  It's a shame people aren't taking advantage of it's obvious utility...

	Jeff

   From: Jeff Gazik [off-list ref]
   Date: Tue, 17 Jun 2003 01:46

   ...and it's been in the tree for quite a while too.  It's a shame
   people aren't taking advantage of it's obvious utility...


I'd like to hear what others believe belong in this new netlink family. The two events that come to mind for this family (or netdev notified if that's more appropriate) are:
1) device initialization failures,
2) events that drive load balancing software.  Right now if we need to throttle the card, we don't send events up to indicate we have reached capacity. 

Possibly the first might belong in the netdev notified family, and the second in the netlink family, since you might want to present more than a two state (success/failure.. or just failure in this case) result.
-

Janice M Girouard wrote:

2) events that drive load balancing software.  Right now if we need to throttle the card, we don't send events up to indicate we have reached capacity.

Question related to this item specifically :)

Do you want to individually send 4000 - 16000 (or more) TX stop / start events per second to userspace?  :)  At some point Heisenburg defeats low latency :)

If not (and I hope not), perhaps also look into the net stack statistics already kept (or add more sampling stats if necessary), and instead trigger events based on sampling those statistics.

	Jeff

Jeff Garzik wrote:

   Do you want to individually send 4000 - 16000 (or more) TX stop /
   start events per second to userspace?  :)   At some point Heisenburg
   defeats low latency :)

How about looking at 1000 byte packet transmit example.  A gigabit adapter would send 125,000 packets per second.  I'm thinking that most of the time, you will have enough available buffers in the adapter that you don't start to see the adapter buffers completely fill up.  Are you saying that 3.2% - 12.8% of the time in this case you're disabling the tcp/ip stack because the transmit buffers on your card are completely full?   Perhaps with zero copy enabled, but the tcp/ip cpu load alone will throttle your ability to fill the adapter buffers up. 

What does your own experience indicate for gigabit adapter cards?

I could see the buffers backing up for 10/100 cards. So that case favors your point.  I'm still thinking that it's a sign someone should be buying a 2nd card and ramping up their network capability.  But I can see your point.

   From: Janice M Girouard [off-list ref]
   Date: Tue, 17 Jun 2003 14:46:56 -0500

    I could see the buffers backing up for 10/100 cards. So that case 
   favors your point.  I'm still thinking that it's a sign someone should 
   be buying a 2nd card and ramping up their network capability.  But I can 
   see your point.

And when we have 1GHZ memory busses and 10GHz cpus tomorrow,
what does this say for 1gbit and 10gbit cards?

You want to define a machine as having too much "work" or not, yet you
only want to consider one metric to do so.  Such schemes are
fundamentally flawed.

From David S. Miller:

And when we have 1GHZ memory busses and 10GHz cpus tomorrow,
what does this say for 1gbit and 10gbit cards?

Such schemes are fundamentally flawed.

Bottom line..   I was asking for input, and I received it.  It's valid to say... look at the statistics.  I really like the concept of driving events through netlink, but querying statistics works. 


p.s.  It's been my experience that the memory system is the main bottleneck when trying to support a heavy network load.  When the 10 Gigabit card emerges, and it's here today, the memory system will be pressed to support it, especially if you're not using zerocopy and you're thinking of using more than one card.  Perhaps if RDMA is capabilities are added to Linux, then things might be different.

So.. when do you think RDMA will show up on Linx?

   From: Janice M Girouard [off-list ref]
   Date: Tue, 17 Jun 2003 15:24:20 -0500
   
   Perhaps if RDMA is  capabilities are added to Linux, then things
   might be different.
   
   So.. when do you think RDMA will show up on Linx?

RDMA is total junk.

On RX, clever RX buffer management is what we need.

On TX zerocopy + TSO is more than sufficient and we have
that today.