I might have mis-spoken about HPET.

The 4.6Mpps is with 2.6.32.4 vanilla, HPET on.

Either way, I'm happy now ;-P

-A

On Thu, Jan 28, 2010 at 10:06 PM, Andrew Dickinson [off-list ref] wrote:

Short response: CONFIG_HPET was the dirty little bastard!

Answering your questions below in case somebody else stumbles across
this thread...

On Thu, Jan 28, 2010 at 4:18 PM, Brandeburg, Jesse
[off-list ref] wrote:

quoted

On Thu, 28 Jan 2010, Andrew Dickinson wrote:

quoted

I'm running into some unexpected performance issues.  I say
"unexpected" because I was running the same tests on this same box 5
months ago and getting very different (and much better) results.

can you try turning off cpuspeed service, C-States in BIOS, and GV3 (aka
speedstep) support in BIOS?

Yup, everything's on "maximum performance" in my BIOS's vernacular (HP
GL360g6) no C-states, etc.

quoted

Have you upgraded your BIOS since before?

Not that I'm aware of, but our provisioning folks might have done
something crazy.

quoted

I agree you should be able to see better numbers, I suspect that you are
getting cross-cpu traffic that is limiting your throughput.

That's what I would have suspected as well.

quoted

How many flows are you pushing?

I'm pushing two streams of traffic, one in each direction.  Each
stream is defined as follows:
   North-bound:
       L2: a0a0a0a0a0a0 -> b0b0b0b0b0b0
       L3: RAND(10.0.0.0/16) -> RAND(100.0.0.0/16)
       L4: UDP with random data
   South-bound is the reverse.

   where "RAND(CIDR)" is a random address within that CIDR (I'm using
an hardware traffic generator).

quoted

Another idea is to compile the "perf" tool in the tools/perf directory of
the kernel and run "perf record -a -- sleep 10" while running at steady
state.  then show output of perf report to get an idea of which functions
are eating all the cpu time.

did you change to the "tickless" kernel?  We've also found that routing
performance improves dramatically by disabling tickless, preemptive kernel
and setting HZ=100.  What about CONFIG_HPET?

yes, yes, yes, and no...

changed CONFIG_HPET to n, rebooted and retested....

ta-da!

quoted

You should try the kernel that the scheduler fixes went into (maybe 31?)
or at least try 2.6.32.6 so you've tried something fully up to date.

I'll give it a whirl :D

quoted

quoted

=== Background ===

The box is a dual Core i7 box with a pair of Intel 82598EB's.  I'm
running 2.6.30 with the in-kernel ixgbe driver.  My tests 5 months ago
were using 2.6.30-rc3 (with a tiny patch from David Miller as seen
here: http://kerneltrap.org/mailarchive/linux-netdev/2009/4/30/5605924).
 The box is configured with both NICs in a bridge; normally I'm doing
some packet processing using ebtables, but for the sake of keeping
things simple, I'm not doing anything special.. just straight bridging
(no ebtables rules, etc).  I'm not running irqbalance and instead
pinning my interrupts, one per core.  I've re-read and double checked
various settings based on Intel's README (i.e. gso off, tso off, etc).

In my previous tests, i was able to pass 3+Mpps regardless of how that
was divided across the two NICS (i.e. 3Mpps all in one direction,
1.5Mpps in each direction simultaneously, etc).  Now, I'm hardly able
to exceed about 750kpps x 2 (i.e. 750k in both directions), and I
can't do more than 750kpps in one direction even with the other
direction having no traffic).

Unfortunately, I didn't take very good notes when I did this last time
so I don't have my previous .config and I'm not 100% positive I've got
identical ethtool settings, etc.  That being said, I've worked through
seemingly every combination of factors that I can think of and I'm
still unable to see the old performance (NUMA on/off, Hyperthreading
on/off, various irq coelescing settings, etc).

I have two identical boxes, they both see the same thing; so a
hardware issue seems unlikely.  My next step is to grab 2.6.30-rc3 and
see if I can repro the good performance with that kernel again and
determine if there was a regression between 2.6.30-rc3 and 2.6.30...
but I'm skeptical that that's the issue since I'm sure other people
would have noticed this as well.


=== What I'm seeing ===

CPU% (almost entirely softirq time, which is expected) ramps extremely
quickly as packet rate increases.  The following table show the packet
rate ("150 x 2" means 150kpps in each direction simultaneously), the
right side is the cpu utilization (as measured by %si in top).

150 x 2:   4%
300 x 2:   8%
450 x 2:  18%
483 x 2:  50%
525 x 2:  66%
600 x 2:  85%
750 x 2: 100% (and dropping frames)

I _am_ seeing interrupts getting spread nicely across cores, so in the
"150 x 2" case, that's about 4% soft-interrupt time per each of the 16
cores.   The CPUs are otherwise idle bar a small amount of hardware
interrupt time (less than 1%).


=== Where it gets weird... ===

Trying to isolate the problem, I added an ebtables rule to drop
everything on the forward chain.  I was expecting to see the CPU
utilization drop since I'd no longer be dealing with the TX-side... no
change.

I then decided to switch from a bridge to a route-based solution.  I
tore down the bridge, enabled ip_forward, setup some IPs and route
entries, etc.  Nothing changes.  CPU performance is identical to
what's shown above.  Additionally, if I add an iptables drop on
FORWARD, the CPU utilization remains unchanged (just like in the
bridging case above).

The point that [I think] I'm driving to is that there's something
fishy going on with the receive-side of the packets.  I wish I could
point to something more specific or a section of code, but I haven't
been able to par this down to anything more granular in my testing.


=== Questions ===

Has anybody seen this before?  If so, what was wrong?
Do you have any recommendations on things to try (either as guesses
or, even better, to help eliminate possibilities)
And along those lines... can anybody think of any possible reasons for this?

hope the above helped.

quoted

This is so frustrating since I _know_ this hardware is capable of so
much more.  It's relatively painless for me to re-run tests in my lab,
so feel free to throw something at me that you think will stick :D

last I checked, I recall with 82599 I was pushing ~4.5 million 64 byte
packets a second (bidirectional, no drop), after disabling irqbalance and
16 tx/rx queues set with set_irq_affinity.sh script (available in our
ixgbe-foo.tar.gz from sourceforge).  82598 should be a bit lower, but
probably can get close to that number.

I haven't run the test lately though, but at that point I was likely on
2.6.30 ish

Jesse

Thank you so much... I wish I'd sent this email out a week ago ;-P

-A