From: Andy Tang <hidden> Date: 2020-02-07 05:09:23
Hi Viresh,
Cpufreq failed to work on our board. CPU frequency keeps at the lowest frequency whatever governor I choose.
Hardware platform: ls1043ardb board (4 cores arm64 arch)
Software: linux kernel v5.4.3 + ubuntu 1804.
Test log:
root@localhost:~# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
All the parameters you may need:
root@localhost:~# for entry in `ls /sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry; cat /sys/devices/system/cpu/cpufreq/policy0/$entry; done
affected_cpus
0 1 2 3
cpuinfo_cur_freq
500000
cpuinfo_max_freq
1600000
cpuinfo_min_freq
500000
cpuinfo_transition_latency
31
related_cpus
0 1 2 3
scaling_available_frequencies
1600000 1000000 800000 500000
scaling_available_governors
userspace ondemand performance schedutil
scaling_cur_freq
500000
scaling_driver
qoriq_cpufreq
scaling_governor
performance
scaling_max_freq
500000
scaling_min_freq
500000
scaling_setspeed
<unsupported>
stats
cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory
Could you please instruct me how to investigate this issue?
BR,
Andy
Hi Viresh,
Cpufreq failed to work on our board. CPU frequency keeps at the lowest frequency whatever governor I choose.
Have you tried performance governor ? Freq should move to highest freq
there for sure.
How do you check the frequency ? Without significant load you won't
see the freq movement.
Also have a look at the policy0/stats/ directory. It will show the
details of freq switching properly, include the transitions that have
happened.
Hardware platform: ls1043ardb board (4 cores arm64 arch)
Software: linux kernel v5.4.3 + ubuntu 1804.
Test log:
root@localhost:~# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
All the parameters you may need:
root@localhost:~# for entry in `ls /sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry; cat /sys/devices/system/cpu/cpufreq/policy0/$entry; done
affected_cpus
0 1 2 3
cpuinfo_cur_freq
500000
cpuinfo_max_freq
1600000
cpuinfo_min_freq
500000
cpuinfo_transition_latency
31
related_cpus
0 1 2 3
scaling_available_frequencies
1600000 1000000 800000 500000
scaling_available_governors
userspace ondemand performance schedutil
scaling_cur_freq
500000
scaling_driver
qoriq_cpufreq
scaling_governor
performance
scaling_max_freq
500000
scaling_min_freq
500000
scaling_setspeed
<unsupported>
stats
cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory
Could you please instruct me how to investigate this issue?
BR,
Andy
From: Andy Tang <hidden> Date: 2020-02-07 06:03:18
Hi Viresh,
I have tried performance governor, cpu frequency is at the lowest frequency.
I also tried ondemand governor with high cpu load, cpu frequency doesn't increase too.
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
root@localhost:~#
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/*
cat: /sys/devices/system/cpu/cpufreq/policy0/stats/reset: Permission denied
1600000 214
1000000 91
800000 82
500000 480088
240
From : To
: 1600000 1000000 800000 500000
1600000: 0 15 10 10
1000000: 21 0 22 18
800000: 6 23 0 44
500000: 7 23 41 0
BR,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月7日 13:23
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 07-02-20, 05:09, Andy Tang wrote:
quoted
Hi Viresh,
Cpufreq failed to work on our board. CPU frequency keeps at the lowest
frequency whatever governor I choose.
Have you tried performance governor ? Freq should move to highest freq
there for sure.
How do you check the frequency ? Without significant load you won't see the
freq movement.
Also have a look at the policy0/stats/ directory. It will show the details of freq
switching properly, include the transitions that have happened.
quoted
Hardware platform: ls1043ardb board (4 cores arm64 arch)
Software: linux kernel v5.4.3 + ubuntu 1804.
Test log:
root@localhost:~# echo performance >
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
root@localhost:~# cat
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
All the parameters you may need:
root@localhost:~# for entry in `ls
/sys/devices/system/cpu/cpufreq/policy0/` ; do echo $entry; cat
/sys/devices/system/cpu/cpufreq/policy0/$entry; done affected_cpus
0 1 2 3
cpuinfo_cur_freq
500000
cpuinfo_max_freq
1600000
cpuinfo_min_freq
500000
cpuinfo_transition_latency
31
related_cpus
0 1 2 3
scaling_available_frequencies
1600000 1000000 800000 500000
scaling_available_governors
userspace ondemand performance schedutil scaling_cur_freq
500000
scaling_driver
qoriq_cpufreq
scaling_governor
performance
scaling_max_freq
500000
scaling_min_freq
500000
scaling_setspeed
<unsupported>
stats
cat: /sys/devices/system/cpu/cpufreq/policy0/stats: Is a directory
Could you please instruct me how to investigate this issue?
BR,
Andy
Hi Viresh,
I have tried performance governor, cpu frequency is at the lowest frequency.
I also tried ondemand governor with high cpu load, cpu frequency doesn't increase too.
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
1600000 1000000 800000 500000
root@localhost:~#
root@localhost:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
500000
root@localhost:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/*
cat: /sys/devices/system/cpu/cpufreq/policy0/stats/reset: Permission denied
1600000 214
1000000 91
800000 82
500000 480088
240
From : To
: 1600000 1000000 800000 500000
1600000: 0 15 10 10
1000000: 21 0 22 18
800000: 6 23 0 44
500000: 7 23 41 0
See, frequencies are getting changed according to this table (at least
from cpufreq's point of view), which means that cpufreq transitions
are passing just fine.
I just had a look at the driver and may have some idea on what's going
on. The cpufreq driver doesn't program the frequencies, but instead it
just switches the parent clk for changing the frequency. And when you
try to read the frequency, all we do is read the frequency from the
clk pointer using clk_get_rate().
So, here are the things which can go wrong, in order of how I would
like to debug them.
- The frequency of all parent clocks is set to 500 MHz by someone and
so clk_get_rate() always report that despite changes to the stats
table.
- clk_set_parent() isn't setting the parent correctly.
Try putting some print messages to the driver to see if target_index()
routine is able to set the parent clk properly or not.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-11 04:20:19
Hi Viresh,
I checked the clk part, no issue was found.
I did found a odd thing in cpufreq directory.
# cat scaling_available_frequencies
1600000 1000000 800000 500000
# cat scaling_max_freq
500000
Why there is a 1600Mhz frequency available, but the max freq is only 500M?
How the max freq is set?
Thanks,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月7日 15:06
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 07-02-20, 06:03, Andy Tang wrote:
quoted
Hi Viresh,
I have tried performance governor, cpu frequency is at the lowest
frequency.
quoted
I also tried ondemand governor with high cpu load, cpu frequency doesn't
See, frequencies are getting changed according to this table (at least from
cpufreq's point of view), which means that cpufreq transitions are passing
just fine.
I just had a look at the driver and may have some idea on what's going on.
The cpufreq driver doesn't program the frequencies, but instead it just
switches the parent clk for changing the frequency. And when you try to read
the frequency, all we do is read the frequency from the clk pointer using
clk_get_rate().
So, here are the things which can go wrong, in order of how I would like to
debug them.
- The frequency of all parent clocks is set to 500 MHz by someone and
so clk_get_rate() always report that despite changes to the stats
table.
- clk_set_parent() isn't setting the parent correctly.
Try putting some print messages to the driver to see if target_index() routine
is able to set the parent clk properly or not.
--
viresh
Hi Viresh,
I checked the clk part, no issue was found.
I did found a odd thing in cpufreq directory.
# cat scaling_available_frequencies
1600000 1000000 800000 500000
# cat scaling_max_freq
500000
Ahh, sorry about that as I must have noticed it earlier.
Why there is a 1600Mhz frequency available, but the max freq is only 500M?
How the max freq is set?
Hmm, I am not sure why this happened. Can you try to write 1600000 to
scaling_max_freq? This will fix it if the userspace is responsible for
this limitation and you need to find which part of your userspace is
causing it.
Else this is coming from within kernel and you need to track calls to
dev_pm_qos_add_request() and check all calls related to freq qos
constraints.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-11 06:23:07
Thanks for your instruction. I guess it was caused by QOS.
I try to change scaling_max_freq. it can be written, but have no effect.
I am going to track to dev_pm_qos_add_request() as you suggested.
Before that, I am also wondering what the factor it could be to limit the cpu max frequency?
BR,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月11日 13:53
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 11-02-20, 04:20, Andy Tang wrote:
quoted
Hi Viresh,
I checked the clk part, no issue was found.
I did found a odd thing in cpufreq directory.
# cat scaling_available_frequencies
1600000 1000000 800000 500000
# cat scaling_max_freq
500000
Ahh, sorry about that as I must have noticed it earlier.
quoted
Why there is a 1600Mhz frequency available, but the max freq is only
500M?
quoted
How the max freq is set?
Hmm, I am not sure why this happened. Can you try to write 1600000 to
scaling_max_freq? This will fix it if the userspace is responsible for this
limitation and you need to find which part of your userspace is causing it.
Else this is coming from within kernel and you need to track calls to
dev_pm_qos_add_request() and check all calls related to freq qos
constraints.
--
viresh
I am going to track to dev_pm_qos_add_request() as you suggested.
Before that, I am also wondering what the factor it could be to limit the cpu max frequency?
One of them is thermal, but there can be others as well. Just do
following search in your kernel source code:
git grep freq_qos_add_request
This will list all the sources of this constraint.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-11 07:20:07
Thank you very much. I confirmed it was caused by QOS.
To be specific, it was caused by TMU.
BR,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月11日 14:35
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 11-02-20, 06:23, Andy Tang wrote:
quoted
I am going to track to dev_pm_qos_add_request() as you suggested.
Before that, I am also wondering what the factor it could be to limit the cpu
max frequency?
One of them is thermal, but there can be others as well. Just do following
search in your kernel source code:
git grep freq_qos_add_request
This will list all the sources of this constraint.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-12 10:11:48
Hi Viresh,
I have another cpufreq issue on our lx2160a platform which has 16 cores and 8 clusters.
Cpufreq works well for ondemand, performance governor.
But for conservative governor, even though cpu usage is almost zero, but some cpus are not
work on the lowest frequency sometimes. It keeps bouncing from the lowest frequency
to the other higher frequencies.
The defaut values are used for the tunable for conservative governor.
In summary, why cpu doesn't work at the lowest frequency even the cpu load is almost zero?
root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/
down_threshold freq_step ignore_nice_load sampling_down_factor sampling_rate up_threshold
root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/*
20
5
0
1
8000
80
root@lx2160ardb:/sys/devices/system/cpu# cat cpu[0-9]/cpufreq/scaling_governor
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative
conservative
root@lx2160ardb:/sys/devices/system/cpu# cat cpu[0-9]/cpufreq/scaling_cur_freq
1000000
1000000
1000000
1000000
1000000
1000000
2000000 /// this is not working at the lowest frequency
2000000
700000
700000
Could you please instruct me how to debug this issue?
Thanks,
BR,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月11日 14:35
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 11-02-20, 06:23, Andy Tang wrote:
quoted
I am going to track to dev_pm_qos_add_request() as you suggested.
Before that, I am also wondering what the factor it could be to limit the cpu
max frequency?
One of them is thermal, but there can be others as well. Just do following
search in your kernel source code:
git grep freq_qos_add_request
This will list all the sources of this constraint.
--
viresh
Hi Viresh,
I have another cpufreq issue on our lx2160a platform which has 16 cores and 8 clusters.
Cpufreq works well for ondemand, performance governor.
But for conservative governor, even though cpu usage is almost zero, but some cpus are not
work on the lowest frequency sometimes. It keeps bouncing from the lowest frequency
to the other higher frequencies.
The defaut values are used for the tunable for conservative governor.
In summary, why cpu doesn't work at the lowest frequency even the cpu load is almost zero?
root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/
down_threshold freq_step ignore_nice_load sampling_down_factor sampling_rate up_threshold
root@lx2160ardb:/sys/devices/system/cpu# cat cpufreq/conservative/*
20
Try increasing down_threshold to a value of 50 (i.e. we reduce
frequency if the load is below 50%) and see if it makes a difference.
I believe something is running on the CPU, that you are unaware of.
Try doing tracing to see what's going on all CPUs.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-13 08:19:01
Hi Viresh,
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月12日 19:51
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 12-02-20, 10:11, Andy Tang wrote:
quoted
Hi Viresh,
I have another cpufreq issue on our lx2160a platform which has 16 cores
and 8 clusters.
quoted
Cpufreq works well for ondemand, performance governor.
But for conservative governor, even though cpu usage is almost zero,
but some cpus are not work on the lowest frequency sometimes. It keeps
bouncing from the lowest frequency to the other higher frequencies.
The defaut values are used for the tunable for conservative governor.
In summary, why cpu doesn't work at the lowest frequency even the cpu
Try increasing down_threshold to a value of 50 (i.e. we reduce frequency if
the load is below 50%) and see if it makes a difference.
I tried down_threshold = 50, it is better. But some cpus still don't work at the lowest frequency sometimes.
I believe something is running on the CPU, that you are unaware of.
Try doing tracing to see what's going on all CPUs.
I keep monitoring cpu usage for several minutes, by top and mpstat tool, and found
most of the cpus are 100% idle, once in a while one of the cpus increases the load to about 18%.
Technically if the cpu usage is not more than up_threshold(80), cpu won't increase its frequency and keep
at the lowest frequency when cpu is at the lowest frequency and conservative governor is used.
So I can't understand why cpu frequency increased?
Thanks,
Andy
I keep monitoring cpu usage for several minutes, by top and mpstat tool, and found
most of the cpus are 100% idle, once in a while one of the cpus increases the load to about 18%.
I don't think top will be a good way for doing this test as we are
talking about load on a CPU in interval in milliseconds. Please check
with tracing and see what's keeping the CPU busy..
Technically if the cpu usage is not more than up_threshold(80), cpu won't increase its frequency and keep
at the lowest frequency when cpu is at the lowest frequency and conservative governor is used.
I agree. Which means that there are some spikes of work getting
scheduled on CPUs. The load will increase slowly and will also
decrease slowly (step-by-step) with conservative governor.
So I can't understand why cpu frequency increased?
Tracing or debugging cs_dbs_update() in conservative governor is the
only way out I would suggest.
--
viresh
From: Andy Tang <hidden> Date: 2020-02-13 09:12:37
Hi Viresh,
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月13日 16:48
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
I agree. Which means that there are some spikes of work getting scheduled
on CPUs. The load will increase slowly and will also decrease slowly
(step-by-step) with conservative governor.
quoted
So I can't understand why cpu frequency increased?
Tracing or debugging cs_dbs_update() in conservative governor is the only
way out I would suggest.
What's the *TRACING" here referring to? Could you please explain more about it?
I have no idea what it is.
Thanks,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月13日 16:48
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
I agree. Which means that there are some spikes of work getting scheduled
on CPUs. The load will increase slowly and will also decrease slowly
(step-by-step) with conservative governor.
quoted
So I can't understand why cpu frequency increased?
Tracing or debugging cs_dbs_update() in conservative governor is the only
way out I would suggest.
What's the *TRACING" here referring to? Could you please explain more about it?
I have no idea what it is.
From: Andy Tang <hidden> Date: 2020-02-13 09:22:03
Appreciate your help.
Thanks,
Andy
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月13日 17:16
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
Caution: EXT Email
On 13-02-20, 09:12, Andy Tang wrote:
quoted
Hi Viresh,
quoted
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月13日 16:48
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
I agree. Which means that there are some spikes of work getting
scheduled on CPUs. The load will increase slowly and will also
decrease slowly
(step-by-step) with conservative governor.
quoted
So I can't understand why cpu frequency increased?
Tracing or debugging cs_dbs_update() in conservative governor is the
only way out I would suggest.
What's the *TRACING" here referring to? Could you please explain more
about it?
quoted
I have no idea what it is.
Sorry about that, I was talking about the in-kernel function tracer here.
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
kernel.org%2Fdoc%2FDocumentation%2Ftrace%2Fftrace.txt&data=02%
7C01%7Candy.tang%40nxp.com%7Ce9d217e1e493426ff8a708d7b0654c77%
7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63717182142741422
1&sdata=HXTASVHLewwjZhX3W9dna9wD87arEgK95snQP7UNZgc%3D
&reserved=0
--
viresh
From: Andy Tang <hidden> Date: 2020-02-14 10:00:25
Hi Viresh,
-----Original Message-----
From: Viresh Kumar <viresh.kumar@linaro.org>
Sent: 2020年2月13日 16:48
To: Andy Tang <redacted>
Cc: rjw@rjwysocki.net; linux-pm@vger.kernel.org
Subject: Re: [EXT] Re: Ask for help about cpufreq issue
On 13-02-20, 08:18, Andy Tang wrote:
quoted
I keep monitoring cpu usage for several minutes, by top and mpstat
tool, and found most of the cpus are 100% idle, once in a while one of the
cpus increases the load to about 18%.
I don't think top will be a good way for doing this test as we are talking about
load on a CPU in interval in milliseconds. Please check with tracing and see
what's keeping the CPU busy..
quoted
Technically if the cpu usage is not more than up_threshold(80), cpu
won't increase its frequency and keep at the lowest frequency when cpu is
at the lowest frequency and conservative governor is used.
I agree. Which means that there are some spikes of work getting scheduled
on CPUs. The load will increase slowly and will also decrease slowly
(step-by-step) with conservative governor.
quoted
So I can't understand why cpu frequency increased?
Tracing or debugging cs_dbs_update() in conservative governor is the only
way out I would suggest.
From cs_dbs_update() function, I did see that sometimes the LOAD is at 100% usage.
There is almost no application running. That must be some spikes of working running.
Unfortunately I can't track them by cs_dbs_update() or ftrace tool.
I doubt the ftrace can be used this way. Do you have any other suggestions? Maybe another tools?
BR,
Andy
From cs_dbs_update() function, I did see that sometimes the LOAD is at 100% usage.
There is almost no application running. That must be some spikes of working running.
Unfortunately I can't track them by cs_dbs_update() or ftrace tool.
I doubt the ftrace can be used this way. Do you have any other suggestions? Maybe another tools?
Not sure if you used ftrace correctly, do following steps:
Enable following options in kernel config and rebuild your image.
- CONFIG_FTRACE
- CONFIG_FUNCTION_TRACER
and then run this script at runtime.
-------------------------8<-------------------------
#!/bin/bash
# Locals
BUFFER=40000
FILE="dft"
# What to trace ??
#EVENTS=" -e sched -e power -e workqueue -e irq"
EVENTS="all"
echo "Starting trace"
/usr/bin/trace-cmd start -b $BUFFER $EVENTS
#You can run some test here that you want to trace.
sleep 5
/usr/bin/trace-cmd stop
cat /sys/kernel/debug/tracing/trace > $FILE.txt
/usr/bin/trace-cmd extract -o $FILE.dat
echo "Saved trace in: "$FILE
sync
-------------------------8<-------------------------
The output file here (dft.dat) can be opened with kernel-shark tool.
Your testing platform's userspace must have trace-cmd command for this to work.
--
viresh