On APM82181, "vmstat" (/proc/stat) doesn't show correct idle percent, if
kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic Tick).
When I run wireless throughput test with heavy traffic, "vmstat" shows very
high idle percent while "oprofile" shows very low idle percent. During the
test, the system is idle, but network traffic uses a lot of hard IRQ and
soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more idle
time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ" in kernel,
idle percent in "vmstat" and "oprofile" would match.
My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
report correct stats.
Thanks,
Fushen
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-12-13 23:24:32
On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
On APM82181, "vmstat" (/proc/stat) doesn't show correct idle
percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
Tick).
When I run wireless throughput test with heavy traffic, "vmstat" shows
very high idle percent while "oprofile" shows very low idle percent.
During the test, the system is idle, but network traffic uses a lot of
hard IRQ and soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
in kernel, idle percent in "vmstat" and "oprofile" would match.
My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
report correct stats.
Hi Thomas ! Any idea what we're doing wrong ? :-)
Cheers,
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-12-13 23:34:24
On Wed, 2011-12-14 at 00:28 +0100, Thomas Gleixner wrote:
On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:
quoted
On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
quoted
On APM82181, "vmstat" (/proc/stat) doesn't show correct idle
percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
Tick).
When I run wireless throughput test with heavy traffic, "vmstat" shows
very high idle percent while "oprofile" shows very low idle percent.
During the test, the system is idle, but network traffic uses a lot of
hard IRQ and soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
in kernel, idle percent in "vmstat" and "oprofile" would match.
My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
report correct stats.
Hi Thomas ! Any idea what we're doing wrong ? :-)
Not really, that had been an issue before and had been fixed. Peter ????
From: Thomas Gleixner <hidden> Date: 2011-12-13 23:44:25
On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:
On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
quoted
On APM82181, "vmstat" (/proc/stat) doesn't show correct idle
percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
Tick).
When I run wireless throughput test with heavy traffic, "vmstat" shows
very high idle percent while "oprofile" shows very low idle percent.
During the test, the system is idle, but network traffic uses a lot of
hard IRQ and soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
in kernel, idle percent in "vmstat" and "oprofile" would match.
My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
report correct stats.
Hi Thomas ! Any idea what we're doing wrong ? :-)
Not really, that had been an issue before and had been fixed. Peter ????
Thanks,
tglx
This is 2.6.32, but I think 2.6.36 is the same.
Thanks,
Fushen
On Tue, Dec 13, 2011 at 3:34 PM, Benjamin Herrenschmidt <
benh@kernel.crashing.org> wrote:
On Wed, 2011-12-14 at 00:28 +0100, Thomas Gleixner wrote:
quoted
On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:
quoted
On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
quoted
On APM82181, "vmstat" (/proc/stat) doesn't show correct idle
percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
Tick).
When I run wireless throughput test with heavy traffic, "vmstat"
shows
quoted
quoted
quoted
very high idle percent while "oprofile" shows very low idle percent.
During the test, the system is idle, but network traffic uses a lot
of
quoted
quoted
quoted
hard IRQ and soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add
more
quoted
quoted
quoted
idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
in kernel, idle percent in "vmstat" and "oprofile" would match.
My APM82181 kernel configuration is "CONFIG_NO_HZ",
"CONFIG_HZ_250=y",
quoted
quoted
quoted
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
My question is that if kernel enables "CONFIG_NO_HZ", how would
kernel
quoted
quoted
quoted
report correct stats.
Hi Thomas ! Any idea what we're doing wrong ? :-)
Not really, that had been an issue before and had been fixed. Peter ????
From: Anton Blanchard <hidden> Date: 2011-12-14 03:17:55
Hi,
This is 2.6.32, but I think 2.6.36 is the same.
Sounds a bit like this, merged in 2.6.39.
Anton
--
commit ad5d1c888e556bc00c4e86f452cad4a3a87d22c1
Author: Anton Blanchard [off-list ref]
Date: Sun Mar 20 15:28:03 2011 +0000
powerpc: Fix accounting of softirq time when idle
commit cf9efce0ce31 (powerpc: Account time using timebase rather
than PURR) used in_irq() to detect if the time was spent in
interrupt processing. This only catches hardirq context so if we
are in softirq context and in the idle loop we end up accounting it
as idle time. If we instead use in_interrupt() we catch both softirq
and hardirq time.
The issue was found when running a network intensive workload. top
showed the following:
0.0%us, 1.1%sy, 0.0%ni, 85.7%id, 0.0%wa, 9.9%hi, 3.3%si, 0.0%st
85.7% idle. But this was wildly different to the perf events data.
To confirm the suspicion I ran something to keep the core busy:
# yes > /dev/null &
8.2%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 10.3%hi, 81.4%si, 0.0%st
We only got 8.2% of the CPU for the userspace task and softirq has
shot up to 81.4%.
With the patch below top shows the correct stats:
0.0%us, 0.0%sy, 0.0%ni, 5.3%id, 0.0%wa, 13.3%hi, 81.3%si, 0.0%st
Signed-off-by: Anton Blanchard [off-list ref]
Cc: stable@kernel.org
Signed-off-by: Benjamin Herrenschmidt [off-list ref]