On 12/17/2012 08:36 PM, Russ Anderson wrote:

The 3.7 kernel grinds to a halt on boot of a system with
2048 cpus.  NMI showed most of the cpus in
_raw_spin_lock in cpuidle_get_cpu_driver().  (backtrace below)

A quick look at cpuidle_get_cpu_driver() shows the hot lock.

In drivers/cpuidle/driver.c:
--------------------------------------------------------
/**
 * cpuidle_get_cpu_driver - return the driver tied with a cpu
 */
struct cpuidle_driver *cpuidle_get_cpu_driver(struct cpuidle_device *dev)
{
        struct cpuidle_driver *drv;

        if (!dev)
                return NULL;

        spin_lock(&cpuidle_driver_lock);
        drv = __cpuidle_get_cpu_driver(dev->cpu);
        spin_unlock(&cpuidle_driver_lock);

        return drv;
}
--------------------------------------------------------

Hi Russ,

thanks for investigating the problem. You are right, there is a
bottleneck here.

Regarding how is used the cpuidle code, I think it is safe to remove the
locks.

This change was added in on Nov 14th, 2012.
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=bf4d1b5ddb78f86078ac6ae0415802d5f0c68f92

The patch says it adds support for cpus with different characteristics,
but adds a big global lock.  The comment claims "no impact for the other
platforms if the option is disabled", which leads me to believe the
spin_lock was added inadvertently.  CPU_IDLE_MULTIPLE_DRIVERS is off
in my config file.

linux$ grep CPU_IDLE_MULTIPLE_DRIVERS .config
# CONFIG_CPU_IDLE_MULTIPLE_DRIVERS is not set

As more cpus become idle, more cpus fight over the lock until
the system livelocks on the crushing weight of idle.

The fix may be to move the spin_lock into __cpuidle_get_cpu_driver,
which has different versions for CONFIG_CPU_IDLE_MULTIPLE_DRIVERS,
to avoid impacting the disabled case, or get rid of the spin_lock
all together.


--------------------------------------------------------
== UV NMI process trace cpu 12: ==
CPU 12
Pid: 0, comm: swapper/12 Tainted: G           O 3.7.0.rja-sgi+ #38
RIP: 0010:[<ffffffff81614e45>]  [<ffffffff81614e45>] _raw_spin_lock+0x25/0x30
[...]
Call Trace:
 [<ffffffff814c891c>] cpuidle_get_cpu_driver+0x1c/0x30
 [<ffffffff814c871d>] cpuidle_idle_call+0x7d/0x1b0
 [<ffffffff8101d08d>] cpu_idle+0xdd/0x130
 [<ffffffff8160a3ea>] start_secondary+0xc6/0xcc
--------------------------------------------------------

-- 
 <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog