On Tue, Aug 18, 2020 at 8:06 PM Greg KH [off-list ref] wrote:

On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:

quoted

Hi all,

I’ve two identical Linux systems with only kernel differences.

What are the differences in the kernels?

quoted

While doing kernel profiling with perf, I got the below mentioned
metrics for Scheduler benchmarks.

1st system (older kernel version compared to the other system) benchmark result:

$ perf bench sched messaging -g 64
# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 64 groups == 2560 processes run

     Total time: 2.936 [sec]


2nd system benchmark result:

$ perf bench sched messaging -g 64
# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 64 groups == 2560 processes run

     Total time: 10.074 [sec]


So as per scheduler benchmark results, clearly a huge difference
between two systems.
Can anyone suggest to me how to dive deeper to know the root cause for
it.

Look a the differences between your different kernels, that would be a
great start :)

I created the difference between two kernel config files and then
tried to spot the CONFIG*SCHED* differences.
Interestingly I see the difference in I/O scheduler config, 1st system
is set to “deadline” and other one is set to “cfq”. So I made it equal
by echoing to “/sys/block/<disk device>/queue/scheduler" but still no
change in scheduler benchmark metrics.

Is it the correct way to find the differences between kernels? If so,
what other important CONFIG_* variables need to consider?


$ cat config.patch | grep -i sched

 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
 CONFIG_CGROUP_SCHED=y
 CONFIG_FAIR_GROUP_SCHED=y
 # CONFIG_RT_GROUP_SCHED is not set
 # IO Schedulers

@@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y
 CONFIG_IOSCHED_DEADLINE=y
 CONFIG_IOSCHED_CFQ=y
 CONFIG_CFQ_GROUP_IOSCHED=y
-CONFIG_DEFAULT_IOSCHED="deadline"
+CONFIG_DEFAULT_IOSCHED="cfq"
+CONFIG_MQ_IOSCHED_DEADLINE=m
+CONFIG_MQ_IOSCHED_KYBER=m
+CONFIG_IOSCHED_BFQ=m
+CONFIG_BFQ_GROUP_IOSCHED=y
 CONFIG_SCHED_SMT=y
 CONFIG_SCHED_MC=y
+CONFIG_SCHED_MC_PRIO=y
+# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y

good luck!

greg k-h

-- 
Thanks,
Sekhar