From: Joel Fernandes <redacted>
Sent: Monday, February 27, 2023 9:22 PM
To: Zhuo, Qiuxu <qiuxu.zhuo@intel.com>
Cc: linux-kernel@vger.kernel.org; Frederic Weisbecker <frederic@kernel.org>;
Lai Jiangshan [off-list ref]; linux-doc@vger.kernel.org; Paul E.
McKenney [off-list ref]; rcu@vger.kernel.org
Subject: Re: [PATCH RFC v2] rcu: Add a minimum time for marking boot as
completed

quoted

On Feb 27, 2023, at 2:53 AM, Zhuo, Qiuxu [off-list ref] wrote:

quoted

From: Joel Fernandes (Google) <redacted>
Sent: Saturday, February 25, 2023 11:34 AM
To: linux-kernel@vger.kernel.org
Cc: Joel Fernandes (Google) <redacted>; Frederic
Weisbecker [off-list ref]; Lai Jiangshan
[off-list ref]; linux- doc@vger.kernel.org; Paul E.
McKenney [off-list ref]; rcu@vger.kernel.org
Subject: [PATCH RFC v2] rcu: Add a minimum time for marking boot as
completed

On many systems, a great deal of boot happens after the kernel thinks
the boot has completed. It is difficult to determine if the system
has really booted from the kernel side. Some features like lazy-RCU
can risk slowing down boot time if, say, a callback has been added
that the boot synchronously depends on.

Further, it is better to boot systems which pass
'rcu_normal_after_boot' to stay expedited for as long as the system is still

booting.

quoted

quoted

For these reasons, this commit adds a config option
'CONFIG_RCU_BOOT_END_DELAY' and a boot parameter
rcupdate.boot_end_delay.

By default, this value is 20s. A system designer can choose to
specify a value here to keep RCU from marking boot completion.  The
boot sequence will not be marked ended until at least boot_end_delay

milliseconds have passed.

quoted

Hi Joel,

Just some thoughts on the default value of 20s, correct me if I'm wrong :-).

Does the OS with CONFIG_PREEMPT_RT=y kernel concern more about the
real-time latency than the overall OS boot time?

But every system has to boot, even an RT system.

   Yes, this is true.

quoted

If so, we might make rcupdate.boot_end_delay = 0 as the default value
(NOT the default 20s) for CONFIG_PREEMPT_RT=y kernels?

Could you measure how much time your RT system takes to boot before the
application runs?

I don't have a real-time OS environment to measure the OS boot time. 
I tried to measure the OS boot time of my "CentOS Stream 8" w/o and
w/ Joel’s patch. 

My testing showed the positive result that the OS boot time was 
reduced by ~4.6% on my side after applying Joel’s patch.

For testing details, please see the below:

1) Testing environment:
    OS            : CentOS Stream 8 (non-RT OS)
    Kernel     : v6.2
    Machine : Intel Cascade Lake server (2 sockets, each with 44 logical threads)
    Qemu  args  : -cpu host -enable-kvm, -smp 88,threads=2,sockets=2, … 

2) My OS boot time definition: 
    The time from the start of the kernel boot to the shell command line 
    prompt is shown from the console. [ Different people may have
    different OS boot time definitions. ]
  
3) My measurement method (very rough method): 
    A timer in the kernel periodically prints the boot time every 100ms. 
    As soon as the shell command line prompt is shown from the console,
    we record the boot time printed by the timer, then the printed boot 
    time is the OS boot time. 
   
    The console log (mixed userspace and kernel logs) looked like this:

           [  OK  ] Started Permit User Sessions.
                        Starting Terminate Plymouth Boot Screen...
                        Starting Hold until boot process finishes up...
           [  OK  ] Started Command Scheduler.
           [    6.824466] input: ImExPS/2 Generic Explorer ...
           [    6.884685] Boot ms 6863
		...
           [    7.170920] Spectre V2 : WARNING: Unprivileged eBPF ...
           [    7.173140] Spectre V2 : WARNING: Unprivileged eBPF ...
           [    7.196741] Boot ms 7175
		...
           [    8.236757] Boot ms 8215

           CentOS Stream 8
           Kernel 6.2.0-rcu+ on an x86_64

           login: [    8.340751] Boot ms 8319
           [    8.444756] Boot ms 8423
		...

     Then the log "login: [    8.340751] Boot ms 8319" roughly showed the OS boot time was ~8.3s.

4) Measured OS boot time (in seconds)
   a) Measured 10 times w/o Joel's patch:
        8.7s, 8.4s, 8.6s, 8.2s, 9.0s, 8.7s, 8.8s, 9.3s, 8.8s, 8.3s
        The average OS boot time was: ~8.7s
   
   b) Measure 10 times w/ Joel's patch: 
        8.5s, 8.2s, 7.6s, 8.2s, 8.7s, 8.2s, 7.8s, 8.2s, 9.3s, 8.4s
        The average OS boot time was: ~8.3s.
		 
The OS boot time was reduced by : 8.7 – 8.3 = 0.4 second
The reduction percentage was       : 0.4/8.7 * 100% = 4.6%

If the testing above makes sense to you, please feel free to 
add

      Tested-by: Qiuxu Zhuo [off-list ref]

Thanks!
-Qiuxu

I can change it to default 0 essentially NOOPing it, but I would rather have a
saner default (10 seconds even), than having someone forget to tune this for
their system.

Thanks,

 - Joel