Did some additional testing and have an update:
1) compiled 2.6.32.27 with CGROUP and NAMESPACES
disabled as much as 'make menuconfig' will allow.
Made no difference on performance--same exact
result.
2) did observe that the IRQ rate is 100k on
2.6.32.27 where it is 33k on 2.6.18(rhel).
3) compiled 2.6.39.4 with same config used
in (1) above, allowing 'make menuconfig'
to fill in differences. Tried 'make defconfig'
but it left out too many modules and the kernel
would not even install. The config used to
build this kernel is attached.
.39 Runs 7% better than .32 but still 27.5% worse
than 2.6.18(rhel) on total reported CPU and 97%
worse on system CPU. The IRQ rate was 50k here.
4) Ran the full 30 minute test again with
perf record -a
running and generated a report (attached).
This was done in packet socket mode because
all the newer kernels have some serious bug
where UDP data is not delivered to about
half of the sockets even though it arrives
to the interface. [I've been ignoring
this since packet socket performance is
close to UDP socket performance and I'm more
worried about network overhead than the
UDP bug. Comparisons are with same mode
test on the 2.6.18(rhel) kernel.]
The application '_raw_spin_lock' number
stands out to me--makes me think that
2.6.39 has greater bias toward spinning
futexes than 2.6.18(rhel) as the user
CPU was 6.5% higher. The .32(rhel) kernel
is exactly the same on user CPU. In UDP
mode there is little or none of this lock-
contention CPU--it appears here due to the
need for queuing messages to worker
threads in packet-socket mode.
Beyond that it looks to me like the kernel paths
have no notable hot-spots, which makes me think
that the code path has gotten longer everywhere
or that subtle changes have interacted badly
with cache behavior to cause the performance
loss. However someone who knows the kernel
code may see things here that I cannot.
-----
This popped into my head. About two years ago
I tried benchmarking SLES RT with our application.
The results were horrifically bad. Don't know
if anything from the RT work was merged into
the kernel, but my overall impression was that
RT traded CPU for latency to the extreme point
where any application that used more than
10% of the much higher CPU consumption would
not work. Haven't looked at latency during
these tests, but I suppose if there are
improvements it might be worth the extra CPU
it's costing. Any thoughts on this?
Le dimanche 02 octobre 2011 à 01:33 -0400, starlight@binnacle.cx a
écrit :
Did some additional testing and have an update:
1) compiled 2.6.32.27 with CGROUP and NAMESPACES
disabled as much as 'make menuconfig' will allow.
Made no difference on performance--same exact
result.
2) did observe that the IRQ rate is 100k on
2.6.32.27 where it is 33k on 2.6.18(rhel).
3) compiled 2.6.39.4 with same config used
in (1) above, allowing 'make menuconfig'
to fill in differences. Tried 'make defconfig'
but it left out too many modules and the kernel
would not even install. The config used to
build this kernel is attached.
.39 Runs 7% better than .32 but still 27.5% worse
than 2.6.18(rhel) on total reported CPU and 97%
worse on system CPU. The IRQ rate was 50k here.
4) Ran the full 30 minute test again with
perf record -a
running and generated a report (attached).
This was done in packet socket mode because
all the newer kernels have some serious bug
where UDP data is not delivered to about
half of the sockets even though it arrives
to the interface. [I've been ignoring
this since packet socket performance is
close to UDP socket performance and I'm more
worried about network overhead than the
UDP bug. Comparisons are with same mode
test on the 2.6.18(rhel) kernel.]
The application '_raw_spin_lock' number
stands out to me--makes me think that
2.6.39 has greater bias toward spinning
futexes than 2.6.18(rhel) as the user
CPU was 6.5% higher. The .32(rhel) kernel
is exactly the same on user CPU. In UDP
mode there is little or none of this lock-
contention CPU--it appears here due to the
need for queuing messages to worker
threads in packet-socket mode.
Beyond that it looks to me like the kernel paths
have no notable hot-spots, which makes me think
that the code path has gotten longer everywhere
or that subtle changes have interacted badly
with cache behavior to cause the performance
loss. However someone who knows the kernel
code may see things here that I cannot.
-----
This popped into my head. About two years ago
I tried benchmarking SLES RT with our application.
The results were horrifically bad. Don't know
if anything from the RT work was merged into
the kernel, but my overall impression was that
RT traded CPU for latency to the extreme point
where any application that used more than
10% of the much higher CPU consumption would
not work. Haven't looked at latency during
these tests, but I suppose if there are
improvements it might be worth the extra CPU
it's costing. Any thoughts on this?
You might try to disable any fancy power saving mode in your machine.
Maybe on your machine, cost to enter/exit deep sleep state is too high.
I see nothing obvious in the profile but userland processing, futex
calls.
Network processing seems to account less than 10% of total cpu...
All this sounds more a process scheduling regression than a network
stack one..
On new kernels, you can check if your udp sockets drops frames because
of rcvbuffer being full (cat /proc/net/udp, check last column 'drops')
To check if softirq processing hit some limits :
cat /proc/net/softnet_stat
Please send full "dmesg" output
Le dimanche 02 octobre 2011 à 09:21 +0200, Eric Dumazet a écrit :
You might try to disable any fancy power saving mode in your machine.
Maybe on your machine, cost to enter/exit deep sleep state is too high.
I see nothing obvious in the profile but userland processing, futex
calls.
Just to clarify a bit :
Sometimes, optimizing one part of the kernel can have a negative impact
on some workloads because we end up doing more sleep/wakeup of
consumers : Several threads might try to acquire a lock at the same
time, while previously they got no contention.
In 2.6.35, commit c377411f2494a (net: sk_add_backlog() take rmem_alloc
into account) changed backlog limit, avoid taking socket lock on flood,
allowing to receive 200.000 pps on a test machine instead of 100pps.
But the receiver was doing a plain
while (1)
recv(...);
And maximum throughput was reached because task never called
scheduler...
At 09:21 AM 10/2/2011 +0200, Eric Dumazet wrote:
Le dimanche 02 octobre 2011 Ã 01:33 -0400,
You might try to disable any fancy power saving
mode in your machine. Maybe on your machine, cost
to enter/exit deep sleep state is too high.
I'll check this out. It's an Opteron 6174 and
so recent and with all the aggressive power
saving AMD can dish out. Note that 'cpuspeed'
is turned off. Only thing I can think of
off the top of my head is to boot with
'nohalt' (which used to be idle=poll). If
anyone knows further Magney Cours tweaks
please let me know.
Just to clarify a bit :
Sometimes, optimizing one part of the kernel can
have a negative impact on some workloads because
we end up doing more sleep/wakeup of consumers :
Several threads might try to acquire a lock at the
same time, while previously they got no
contention.
In 2.6.35, commit c377411f2494a (net:
sk_add_backlog() take rmem_alloc into account)
changed backlog limit, avoid taking socket lock on
flood, allowing to receive 200.000 pps on a test
machine instead of 100pps. But the receiver was
doing a plain
while (1)
recv(...);
And maximum throughput was reached because task
never called scheduler...
Sometimes I think it might be nice to run without
an OS at all :-)
We are looking at various kernel bypass approaches
for receiving packets and it's in my mind that
one of these may get us to running on newer
kernels without giving up performance.
I see nothing obvious in the profile but userland
processing, futex calls.
Indeed, this is from the app-internal queuing
I mentioned in the last post. Because UDP is
presently unusable in any .32 or .39 kernel
I'm stuck with using packet sockets as a proxy
for them. As mentioned, the two produce very
similar results in all past tests.
Network processing seems to account less than 10%
of total cpu... All this sounds more a process
scheduling regression than a network stack one..
Interesting. When I start testing kernel bypass
the degree to which this is the case should become
clear.
On new kernels, you can check if your udp sockets
drops frames because of rcvbuffer being full (cat
/proc/net/udp, check last column 'drops')
As I've stated several times, all these tests are
run with verifiable zero data loss. The max-load
test is the highest rate were zero data loss can
be maintained. UDP socket buffers are set to 64MB.
To check if softirq processing hit some limits :
cat /proc/net/softnet_stat
Ok.
Please send full "dmesg" output
Ok.
At 07:47 AM 10/2/2011 -0700, Stephen Hemminger wrote:
Try disabling PCI DMA remapping. The additional
overhead of setting up IO mapping can be a
performance buzz kill. Try CONGIG_DMAR=n
Definitely. Thanks for the tip!
-----
Probably won't get to these until Monday or
Tuesday.
Sometimes I think it might be nice to run without
an OS at all :-)
We are looking at various kernel bypass approaches
for receiving packets and it's in my mind that
one of these may get us to running on newer
kernels without giving up performance.
If permissible in your environment, try running PACKET:tpacket_v3.
Also, I always wondered about the following in the output of:
net/ptype
dada edsa_rcv+0x0/0x246
001b dsa_rcv+0x0/0x225
and so when I'm fine tuning apps in virtual environments I nuke the
above during kernel compilation.
Chetan Loke