Re: smp: Start up non-boot CPUs asynchronously

[Small note, it appears as if the last 2 of your replies to this thread
 didn't reach LKML.]

On 02/14/2012 08:01 PM, Arjan van de Ven wrote:

one coments; will comment more when I get to work

On Tue, Feb 14, 2012 at 1:48 AM, Srivatsa S. Bhat 

7. And whichever code between smp_init() and async_synchronize_full() didn't

    care about CPU hotplug till today but depended on all cpus being
    online must
    suddenly start worrying about CPU Hotplug. They must register a cpu
    notifier
    and handle callbacks etc etc.. Or if they are not worth that
    complexity, they
    should atleast be redesigned or moved around - like the print
    statements that
    tell how many cpus came up, for example.


frankly, such code HAS to worry about cpus going online and offline even
today; the firmware, at least on X86, can start taking cores
offline/online once ACPI is initialized....
(as controlled by a data center manager from outside the box, usually
done based on thermal or power conditions on a datacenter level).
Now, no doubt that we have bugs in this space, since this only happened
very rarely before.

Question is what to do from a longer term strategy:
Either we declare the number of online CPUs invariant during a certain
phase of the boot (and make ACPI and co honor this as well somehow)
or
We decide to go about fixing these (maybe with the help of lockdep?)

In addition to this, the reality is that the whole "bring cpus up"
sequence needs to be changed; the current one is very messy and requires
the hotplug lock for the whole bring up of each individual cpu... which
is a very unfortunate design; a much better design would be to only take
the lock for the actual registration of the newly brought up CPU to the
kernel, while running the physical bringup without the global lock.
If/when that change gets made, we can do the physical bring up in
parallel (with each other, but also with the rest of the kernel boot),
and do the registration en-mass at some convenient time in the boot,
potentially late.

Sounds like a good idea, but how will we take care of CPU_UP_PREPARE and
CPU_STARTING callbacks then? Because, CPU_UP_PREPARE callbacks are run
before bringing up the cpu and CPU_STARTING is called from the cpu that is
coming up. Also, CPU_UP_PREPARE callbacks can be failed, which can lead
to that particular cpu boot getting aborted. With the "late commissioning
of CPUs" idea you proposed above, retaining such semantics could become
very challenging.

Regards,
Srivatsa S. Bhat