From: Marc Leeman <hidden> Date: 2007-06-21 14:15:27
Ok, I guess it's comparing apples to lemons here, but I'll have a go at
it anyway.
I'm trying to figure out why partial network decoding on an 8347E is
disappointingly slow wrt an older 8245 processor.
When simply receiving (cf. att) a multicast stream of 12 Mbps, an
8245/uclibc 0.9.28 @350 MHz, ppc arch , the system runs smoothly at a
load of around 4% on a e100 based MAC (pci: 8086:1209 ).
When doing the same thing on 8347e/0.9.28 @400 Mhz, powerpc arch, the
system is loaded at around 34%.
any clues?
This program just takes in data and does nothing with it, to limit the
search area :)
--
greetz, marc
You know until today, I never really realized how much I love my feet.
Chiana - Vitas Mortis
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
From: Kumar Gala <hidden> Date: 2007-06-21 15:32:31
On Jun 21, 2007, at 9:15 AM, Marc Leeman wrote:
Ok, I guess it's comparing apples to lemons here, but I'll have a
go at
it anyway.
I'm trying to figure out why partial network decoding on an 8347E is
disappointingly slow wrt an older 8245 processor.
When simply receiving (cf. att) a multicast stream of 12 Mbps, an
8245/uclibc 0.9.28 @350 MHz, ppc arch , the system runs smoothly at a
load of around 4% on a e100 based MAC (pci: 8086:1209 ).
When doing the same thing on 8347e/0.9.28 @400 Mhz, powerpc arch, the
system is loaded at around 34%.
any clues?
How are you measuring load? I'm assuming the 8245 and 8347 are using
the same kernel.
- k
From: Marc Leeman <hidden> Date: 2007-06-21 15:56:30
How are you measuring load? I'm assuming the 8245 and 8347 are using
the same kernel.
Simply the load of the processor. The 8245 is running ppc/2.6.17, the
8347e is running powerpc/2.6.21.1.
The 8245 kernel is not being upgraded anymore since
1. we're not using them anymore on the new designs
2. I didn't bother fixing the board support after the interrupt
handling changed in 2.6.18 because of 1.
Disabling NAPI seems to improve the situation a bit, but there's still a
load difference 25% on a marginally faster processor.
Mem: 10788K used, 116940K free, 0K shrd, 0K buff, 4212K cached
Load average: 0.39 0.43 0.29
PID USER STATUS VSZ PPID %CPU %MEM COMMAND
361 barco SW 152 270 30.8 0.1 recv
2222 barco RW 1124 270 0.3 0.8 top
119 root SW 1216 1 0.0 0.9 dropbear
270 barco SW 1132 1 0.0 0.8 sh
94 root SW 1132 1 0.0 0.8 syslogd
1 root SW 1128 0 0.0 0.8 init
95 root SW 1112 1 0.0 0.8 klogd
Mem: 8144K used, 20944K free, 0K shrd, 888K buff, 2384K cached
Load average: 0.00 0.00 0.00 (Status: S=sleeping R=running, W=waiting)
PID USER STATUS RSS PPID %CPU %MEM COMMAND
407 barco R 124 1 6.4 0.4 recv
510 root S 668 130 0.7 2.2 dropbear
13889 root R 368 7377 0.3 1.2 top
511 barco S 476 510 0.0 1.6 sh
52 root S 376 1 0.0 1.2 syslogd
1 root S 352 0 0.0 1.2 init
59 root S 340 1 0.0 1.1 klogd
--
greetz, marc
Oh no, no, no, no I don't boogie.
Crichton - Won't Get Fooled Again
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
From: Marc Leeman <hidden> Date: 2007-06-21 17:13:50
Simply the load of the processor. The 8245 is running ppc/2.6.17, the
8347e is running powerpc/2.6.21.1.
Sorry, I'm confusing my boards: this particular board with the 8245
processor is running 2.4.34 [1].
so
8245: 2.4.34
8237e: 2.6.21.1
[1] the 2.4 line balanced the load better of multiple streams being
taken in wrt to the 2.6 kernels; this is the reason we stuck with 2.4
for this platform.
--
greetz, marc
He claims to be a human from a planet called Erp.
Aeryn - Premiere
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
From: Marc Leeman <hidden> Date: 2007-06-28 18:06:44
a small update:
8245: 2.4.34
8237e: 2.6.21.1
I've tried the following setup:
multicast stream @8192 kbps, one process taking in and dumping the data
on each board [1].
a) 8245/2.4.34/e100: 2.3.43-k1, @400 MHz
b) 8245/2.6.17/e100: 2.3.43-k1 [2] @350 MHz
c) 8347e/2.6.21.1/gianfar @400 Mhz
c) XScale-IXP42x/2.6.18-4/ixp4xx @266 MHz (NSLU2)
(2.3.43-k1 is the e100 driver version).
The process load for taking in the data is:
a) 4-5% [3]
b) 10-11%
c) 13-14%
d) 4-5%
While the current 8347/gianfar platform is the worst performer, the
2.6 kernel with the 2.4 e100 (before the rewrite) seems to perform
poorly too [4].
So the 834x preforms worse wrt the 8245 based configuration even though
it is slightly higher clocked.
It seems as if I bumped into the problem that lead me stick with the 2.4
in the first place for this 8245 platform; but never got round to
investigating. I find these results especially intriguing when
considering an ARM platform (NSLU2 device) that I had around, clocked at
only 66% of the 8347 and at 80% of the 8245 performs certainly in par
with the last one...
Even though I will need to recheck this (results to follow), a quick
test didn't reveal any significant difference between a ppc and powerpc
arch in the kernel.
It does look like, on our 8245/83xx platforms, the 2.6.x kernel performs
worse wrt the 2.4 ppc kernels and the 83xx configuration is worse wrt
the 8245 based configuration [5]. In retrospect, we had signals that
there was a problem with the 8245/83xx performance over the network last
year when investigating gstreamer, but due to time pressure but assumed
it was due to gstreamer and not the processor. This came as a suprise to
some of the ppl on the gstreamer mailing list that reported performant
ports to ARM architectures.
The results with the NSLU2 will certainly put heat on us from management
when redesigning or for follow up designs :(
Anyhow, I'm currently extending my test setups since this is an
important problem and set back.
If anyone has a hint to explaining what is going on here, please do
since solving this will certainly beat redesigning (esp. considering the
timeframe we've been assigned).
I've only found one relevant reference to 2.4/2.6 network performance
decrease at this point [6].
[1] sources attached
mcrecv -p 225.1.2.3 -a 12345
mcsend -p 225.1.2.3 -a 12345 -b 8192
I'm preparing more tests in the next days, in trying to figure out
what really is going on here.
[2] 2.4 driver ported to 2.6 kernel.
[3] This figure is read from top and not from the app since it seems to
be an underestimate (./fs/proc/array.c).
[4] I believe I ported the 2.4 e100 to the 2.6 2 years ago because it
performed much better, but I'll verify that in the next days.
[5] Obviously, testing 834x against the 2.4 kernel is not really an
option :)
[6] http://www.mail-archive.com/linux-net@vger.kernel.org/msg01283.html
--
greetz, marc
Don't think I'm going to miss you, any of you. I'm not. Well, maybe
a little bit.
Rygel - Into the Lion's Den - Wolf in Sheep's Clothing
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
Platform h is just an old server as reference to see if a 2.6.x scales
as bad with an e100 on a different architecture.
The process load for taking in the data is:
a) 4-5% [3]
b) 10-11%
c) 13-14%
d) 4-5%
e) 10-11%
f) 2-3%
g) 5%
h) 0%
This situation is even (a lot) worse when increasing the bitrate. When
a bitrate of 12 Mbps is used, we get the following results:
a) 4-5%
b) 18%
c) 35%
d) 4-7%
e) 18%
f) -
g) -
h) 1-2 %
While the current 8347/gianfar platform is the worst performer, the
2.6 kernel with the 2.4 e100 (before the rewrite) seems to perform
poorly too [4].
So the 834x preforms worse wrt the 8245 based configuration even though
it is slightly higher clocked.
It seems as if I bumped into the problem that lead me stick with the 2.4
in the first place for this 8245 platform; but never got round to
investigating. I find these results especially intriguing when
considering an ARM platform (NSLU2 device) that I had around, clocked at
only 66% of the 8347 and at 80% of the 8245 performs certainly in par
with the last one...
The load is even worsening in a non linearly as the bitrate goes up (I
coult not test all the platforms for this since not all the embedded
platforms are located in our network and I've rallied some collegues
from over the company to get some other platforms tested, probably I
will get more data next week).
Even though I will need to recheck this (results to follow), a quick
test didn't reveal any significant difference between a ppc and powerpc
arch in the kernel.
It does look like, on our 8245/83xx platforms, the 2.6.x kernel performs
worse wrt the 2.4 ppc kernels and the 83xx configuration is worse wrt
the 8245 based configuration [5]. In retrospect, we had signals that
there was a problem with the 8245/83xx performance over the network last
year when investigating gstreamer, but due to time pressure but assumed
it was due to gstreamer and not the processor. This came as a suprise to
some of the ppl on the gstreamer mailing list that reported performant
ports to ARM architectures.
If I look at platform (f), 405/2.6.22-rc6, it doesn't seem to be a
general powerpc problem, but just a 824x/83xx or platform issue.
The results with the NSLU2 will certainly put heat on us from management
when redesigning or for follow up designs :(
Anyhow, I'm currently extending my test setups since this is an
important problem and set back.
If anyone has a hint to explaining what is going on here, please do
since solving this will certainly beat redesigning (esp. considering the
timeframe we've been assigned).
I've only found one relevant reference to 2.4/2.6 network performance
decrease at this point [6].
[1] sources attached
mcrecv -a 225.1.2.3 -p 12345
mcsend -a 225.1.2.3 -p 12345 -b 8192
--
greetz, marc
Aeryn, did I say or do anything to piss you off? I mean other than
caving in the side of your head?
Crichton - Die Me, Dichotomy
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
From: Marc Leeman <hidden> Date: 2007-07-09 15:47:32
More platforms and higher bitrate tests (I've left the previous post in
comment):
I finally was able to figure out the culprid:
CONFIG_SLOB=y
instead of
CONFIG_SLAB=y
--------
CONFIG_SLAB:
Disabling this replaces the advanced SLAB allocator and
kmalloc support with the drastically simpler SLOB allocator.
SLOB is more space efficient but does not scale well and is
more susceptible to fragmentation.
--------
I was expecting a lower DMM performance but wasn't expecting such a
drain on kernel/network load.
The original reason for this change was a fixed flashmap and a increased
2.6 kernel that didn't fit in this region (backwards compatible).
--
greetz, marc
I feel like I had a spiritual enema.
Jool - Losing Time
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux
On Mon, Jul 09, 2007 at 05:47:23PM +0200, Marc Leeman wrote:
Disabling this replaces the advanced SLAB allocator and
kmalloc support with the drastically simpler SLOB allocator.
SLOB is more space efficient but does not scale well and is
more susceptible to fragmentation.
--------
I was expecting a lower DMM performance but wasn't expecting such a
drain on kernel/network load.
OK, to be clear: you seem to be saying that using the SLOB instead
of the SLAB allocator results in such terrible memory fragmentation
that network performance is degraded by large factors (2x or 5x or
something like that, if I remember your earlier emails). Is that right?
I thought I heard about some memory-defrag patches being posted.
What happens if these are used together with SLOB? Does one regain the
lost performance? Perhaps maybe one gets even better performance?
--linas
From: Marc Leeman <hidden> Date: 2007-07-10 07:55:40
quoted
I was expecting a lower DMM performance but wasn't expecting such a
drain on kernel/network load.
OK, to be clear: you seem to be saying that using the SLOB instead
of the SLAB allocator results in such terrible memory fragmentation
that network performance is degraded by large factors (2x or 5x or
something like that, if I remember your earlier emails). Is that right?
Yep, I thought I would at least post my findings after hurracing the
list with my posts.
Well, I don't really know if it is the fragmentation that comes into
play, or if it is simply the implementation of the slob allocator that
much more inefficient in allocating free blocks of memory; but that's
about right.
I thought I heard about some memory-defrag patches being posted.
What happens if these are used together with SLOB? Does one regain the
lost performance? Perhaps maybe one gets even better performance?
In the ChangeLog of the 2.6.22, I saw something about a slub allocator
that I want to test; I'll give your suggestion a go too, though I would
not expect significant improvements: I suspect it's the slob
implementation that is slower.
But I had a small problem with my flash not being detected anymore when
quickly booting the 2.6.22, I'll look into it today, there was a note in
the ChangeLog for powerpc about this IIRC.
--
greetz, marc
Better wed than dead.
Crichton - Look at the Princess - A Kiss is Just a Kiss
chiana 2.6.18-4-ixp4xx #1 Tue Mar 27 18:01:56 BST 2007 GNU/Linux