RE: [PATCH] abysmal e1000 performance (DITR)

Thayne,

I can't speak to what happened previously, only what's going on now.
You need to be careful about basing any tuning on any one single test.
For each one of these test you find that have performance problems I can
show you where it works just fine and most cases works better.  So let
be careful about making judgements like this.

We are going to look at this and make some changes based on what we
find.  I already said that we probably won't be ripping out the DITR as
some people really make a lot of use out of it.  It may be off by
default or have some default setting which doesn't hurt test cases.
Since Netpipe is a relatively unused test for performance (not really
it's purpose, it was developed to find "holes" in internet packets
lengths), it's not part of our normal testing.

Like I said, we are looking at it and will come up with a more robust
solution.

Cheers,
John

-----Original Message-----
From: Thayne Harbaugh [mailto:tharbaugh@lnxi.com] 
Sent: Friday, August 27, 2004 3:05 PM
To: Ronciak, John
Cc: Jeff Garzik; hadi@cyberus.ca; Venkatesan, Ganesh; 
netdev@oss.sgi.com; Feldman, Scott; Brandeburg, Jesse
Subject: RE: [PATCH] abysmal e1000 performance (DITR)


On Fri, 2004-08-27 at 14:49 -0700, Ronciak, John wrote:

quoted

Jamal, Thayne,

I've asked Jeff to go ahead and apply this patch as a way 

around this

quoted

for now.  We would liketo see the DITR stay but now have this
performacne problem so we don't want to rip it out.  We do 

however need

quoted

a test case to replicate this as we have not been seeing it in our
testing.  Please get us those case that break things.  We'll have a
better solution longer term based on the test cases (as 

well as the ones

quoted

we normally use of course).

Attached is an email that I sent out 2004 May 25.  The email is very
detailed about what the problem is, how to test it, and an initial
patch.  The email was sent to several addresses at Intel.  I 
later sent
it to lkml and netdev.

There was almost no response at the time.  The best reply that I
received from Intel was that the DITR was put in and calibrated
according to a marketing benchmark program and that it wouldn't be
changed even though tests (and customers) showed that the performance
was *abysmal*.  It's also a big question what role DITR plays when it
seems deprecated by NAPI.

The response was disappointing.  I'm now wondering why this has now
become interesting and the thread has resumed.  What's 
changed?  How did
this catch someone's attention?  What can I do next time 
(hopefully that
won't happen) so that people *do* take interest.

quoted

quoted

-----Original Message-----
From: netdev-bounce@oss.sgi.com 
[mailto:netdev-bounce@oss.sgi.com] On Behalf Of Thayne Harbaugh
Sent: Thursday, August 26, 2004 2:29 PM
To: Jeff Garzik
Cc: hadi@cyberus.ca; Venkatesan, Ganesh; netdev@oss.sgi.com; 
Feldman, Scott; Brandeburg, Jesse
Subject: Re: [PATCH] abysmal e1000 performance (DITR)


On Thu, 2004-08-26 at 16:26 -0400, Jeff Garzik wrote:

quoted

Thayne Harbaugh wrote:

quoted

On Thu, 2004-08-26 at 13:55 -0400, jamal wrote:

quoted

Ganesh,

Can you please make this feature off by default and perhaps
accesible via ethtool for peopel who want to turn it on.
I just wasted a few hours and was bitten by this 

performance-wise.

quoted

quoted

quoted

quoted

quoted

Please consider disabling it.

This is a *horrible* problem.  Even though it's fixable 

by passing a

quoted

quoted

module parameter, the default bites those that *know* 

about it.  We have

quoted

quoted

had customers bitten by this and customers that have 

insisted in

quoted

quoted

quoted

quoted

swapping all the NICs in a cluster to Broadcom TG3 NICs.

It's a black eye for Intel and a loss of business - 

that's the opinion

quoted

quoted

of our customers.

If it's so bad we should disable it by default, either via 

the module 

quoted

parameter or via a kernel CONFIG_xxx option.

Yes, it is so bad.  The dynamic interrupt setting should 

be deprecated

quoted

quoted

by the use of NAPI.

This is a simple way to disable it, yet still keep the 

code so that

quoted

quoted

someone can enable it if they really wanted it.  I, however, 
would just
as soon see all of the DITR code ripped out.

There are other ways that might be better for dealing with 
it, yet still
keeping the DITR code viable.

--- drivers/net/e1000/e1000_param.c.broken_ditr 2004-08-26 

15:40:34.436456736 -0600

+++ drivers/net/e1000/e1000_param.c     2004-08-26 

15:49:07.186506880 -0600

@@ -212,7 +212,7 @@
 #define MAX_TXABSDELAY            0xFFFF
 #define MIN_TXABSDELAY                 0
  
-#define DEFAULT_ITR                    1
+#define DEFAULT_ITR                 8000
 #define MAX_ITR                   100000
 #define MIN_ITR                      100
  

On Fri, 2004-08-27 at 15:41 -0700, Ronciak, John wrote:

Thayne,

I can't speak to what happened previously, only what's going on now.
You need to be careful about basing any tuning on any one single test.
For each one of these test you find that have performance problems I can
show you where it works just fine and most cases works better.  So let
be careful about making judgements like this.

I would agree with you if the problem was only apparent with a
benchmark.  The incident that started the investigation was due to real-
world workloads on a cluster - a newer cluster that should have been
significantly faster, but was many times slower than an older cluster.
That is why customers can't use the e1000 with the stock 5.x e1000
driver (or without changing the DITR value as a module option).  When
the default setup is anywhere from 10 to 30 percent of the expected
performance then it becomes very difficult to justify shipping a NIC
that will likely be incorrectly configured when the driver or kernel is
changed - too many support problems.

I see that as a wise judgment based on the real usage patterns of
customers - not a knee-jerk reaction based on a simplistic benchmark.
At the time that I investigated this problem I looked at other
benchmarks that *didn't* exhibit this problem.  The question was why our
customers had this problem but it didn't show up when we tested the
cluster.

It's a perfect example of the classic problem: benchmarks aren't always
representative of real-world loads.  This is a case where a real-world
load exposed a deficiency of the driver - a deficiency that wasn't
exposed by benchmarks but becomes quite obvious when the driver is
scrutinized and understood.

We are going to look at this and make some changes based on what we
find.

Wonderful.

  I already said that we probably won't be ripping out the DITR as
some people really make a lot of use out of it.

That's what I'm hoping you can help me with: why is DITR so useful?  I
just can't see why:

NAPI does the same thing and can work with *any* card (provided the
driver is written correctly).  I'd much rather deal with something
universal than have to deal with something unique to a card.

The DITR algorithm doesn't consider packet backlog nor system load.  Why
reduce the interrupt rate when the load is low?  When the interrupt rate
is reduced when the load is low then unnecessary latency is added.  The
DITR may work fine with an old system with an e1000, but a dual Opteron
with an e1000 has plenty of horsepower to perform calculations *and*
push packets.  For DITR to correctly work, it needs to consider much
more than the number of packets received and transmitted - the current
algorithm is overly simplistic (it also doesn't correctly consider
symmetric vs asymmetric receive/transmit loads).

Please provide details so that I can become one of those that can ". . .
make a lot of use out of [DITR]."

It may be off by
default or have some default setting which doesn't hurt test cases.

What about settings that don't hurt real-world fluid dynamics
calculations?  I couldn't care less about benchmarks that aren't
representative of the real-world load.  The Netpipe program was used
because it reasonably resembles the traffic flow of real-world cases
that didn't work with DITR.  Netpipe is much easier to provide and
configure than a fluid dynamics cluster program that would necessitate
licensing and other infrastructure for use as a testing tool.

Since Netpipe is a relatively unused test for performance (not really
it's purpose, it was developed to find "holes" in internet packets
lengths), it's not part of our normal testing.

Well, maybe I can interest you in buying a small cluster with some fluid
dynamics software that has packet flow similar to what Netpipe produces?

Like I said, we are looking at it and will come up with a more robust
solution.

Wonderful.  I hope you can find a good solution.  I do prefer being able
to use all those e1000 NICs rather than replace them.  Yes, really.  I
significantly prefer using good products that perform as expected than
spending my time whining, complaining and carrying on.  Yes, really, I
prefer to have happy customers, happy vendors and be happy myself.  I
only bring this up because I care and I hope someone else cares, and I
want everyone to get back to being happy.