From: Sachin Sant <hidden> Date: 2021-06-22 15:59:52
quoted
On Tue, 22 Jun 2021 at 09:39, Sachin Sant [off-list ref] wrote:
quoted
While booting 5.13.0-rc7-next-20210621 on a PowerVM LPAR following warning
is seen
[ 30.922154] ------------[ cut here ]------------
[ 30.922201] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
[ 30.922219] WARNING: CPU: 6 PID: 762 at kernel/sched/fair.c:3277 update_blocked_averages+0x758/0x780
Yes. That was exactly the purpose of the patch. There is one last
remaining part which could generate this. I'm going to prepare a patch
Could you try the patch below ? I have been able to reproduce the problem locally and this
fix it on my system:
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 07:19:45
Hi Sachin,
Le mardi 22 juin 2021 à 21:29:36 (+0530), Sachin Sant a écrit :
quoted
quoted
On Tue, 22 Jun 2021 at 09:39, Sachin Sant [off-list ref] wrote:
quoted
While booting 5.13.0-rc7-next-20210621 on a PowerVM LPAR following warning
is seen
[ 30.922154] ------------[ cut here ]------------
[ 30.922201] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
[ 30.922219] WARNING: CPU: 6 PID: 762 at kernel/sched/fair.c:3277 update_blocked_averages+0x758/0x780
Yes. That was exactly the purpose of the patch. There is one last
remaining part which could generate this. I'm going to prepare a patch
Could you try the patch below ? I have been able to reproduce the problem locally and this
fix it on my system:
I can recreate the issue with this patch.
ok, so your problem seem to be different from my assumption. Could you try
the patch below on top of the previous one ?
This will help us to confirm that the problem comes from load_avg and that
it's linked to the cfs load_avg and it's not a problem happening earlier in
the update of PELT.
From: Sachin Sant <hidden> Date: 2021-06-23 07:58:27
quoted
quoted
Could you try the patch below ? I have been able to reproduce the problem locally and this
fix it on my system:
I can recreate the issue with this patch.
ok, so your problem seem to be different from my assumption. Could you try
the patch below on top of the previous one ?
This will help us to confirm that the problem comes from load_avg and that
it's linked to the cfs load_avg and it's not a problem happening earlier in
the update of PELT.
From: Sachin Sant <hidden> Date: 2021-06-23 10:23:21
On 23-Jun-2021, at 1:28 PM, Sachin Sant [off-list ref] wrote:
quoted
quoted
quoted
Could you try the patch below ? I have been able to reproduce the problem locally and this
fix it on my system:
I can recreate the issue with this patch.
ok, so your problem seem to be different from my assumption. Could you try
the patch below on top of the previous one ?
This will help us to confirm that the problem comes from load_avg and that
it's linked to the cfs load_avg and it's not a problem happening earlier in
the update of PELT.
Indeed. With both the patches applied I see following warning related to load_avg
I left the machine running for sometime. Then attempted a kernel compile.
I subsequently saw warnings triggered for util_avg as well as runnable_avg
[ 8371.964935] ------------[ cut here ]------------
[ 8371.964958] cfs_rq->avg.util_avg
[ 8371.964969] WARNING: CPU: 16 PID: 479551 at kernel/sched/fair.c:3283 update_blocked_averages+0x700/0x830
……..
……..
[ 8664.754506] ------------[ cut here ]------------
[ 8664.754569] cfs_rq->avg.runnable_avg
[ 8664.754583] WARNING: CPU: 23 PID: 125 at kernel/sched/fair.c:3284 update_blocked_averages+0x730/0x830
…….
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 12:08:43
Le mercredi 23 juin 2021 à 15:52:59 (+0530), Sachin Sant a écrit :
quoted
On 23-Jun-2021, at 1:28 PM, Sachin Sant [off-list ref] wrote:
quoted
quoted
quoted
Could you try the patch below ? I have been able to reproduce the problem locally and this
fix it on my system:
I can recreate the issue with this patch.
ok, so your problem seem to be different from my assumption. Could you try
the patch below on top of the previous one ?
This will help us to confirm that the problem comes from load_avg and that
it's linked to the cfs load_avg and it's not a problem happening earlier in
the update of PELT.
Indeed. With both the patches applied I see following warning related to load_avg
I left the machine running for sometime. Then attempted a kernel compile.
I subsequently saw warnings triggered for util_avg as well as runnable_avg
[ 8371.964935] ------------[ cut here ]------------
[ 8371.964958] cfs_rq->avg.util_avg
[ 8371.964969] WARNING: CPU: 16 PID: 479551 at kernel/sched/fair.c:3283 update_blocked_averages+0x700/0x830
……..
……..
[ 8664.754506] ------------[ cut here ]------------
[ 8664.754569] cfs_rq->avg.runnable_avg
[ 8664.754583] WARNING: CPU: 23 PID: 125 at kernel/sched/fair.c:3284 update_blocked_averages+0x730/0x830
…….
Ok. This becomes even more weird. Could you share your config file and more details about
you setup ?
Have you applied the patch below ?
https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
Regarding the load_avg warning, I can see possible problem during attach. Could you add
the patch below. The load_avg warning seems to happen during boot and sched_entity
creation.
---
kernel/sched/fair.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
Hi,
Wouldn't the attached diff below also help when load is removed,
Vincent? Isn't there a theoretical chance that x_sum ends up at zero
while x_load ends up as a positive value (without this patch)? Can
post as a separate patch if it works for Sachin.
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 12:22:35
On Wed, 23 Jun 2021 at 14:18, Odin Ugedal [off-list ref] wrote:
Hi,
Wouldn't the attached diff below also help when load is removed,
Vincent? Isn't there a theoretical chance that x_sum ends up at zero
while x_load ends up as a positive value (without this patch)? Can
post as a separate patch if it works for Sachin.
In theory it should not because _sum should be always larger or equal
to _avg * divider. Otherwise, it means that we have something wrong
somewhere else
ons. 23. jun. 2021 kl. 14:22 skrev Vincent Guittot [off-list ref]:
In theory it should not because _sum should be always larger or equal
to _avg * divider. Otherwise, it means that we have something wrong
somewhere else
Yeah, that might be the case. Still trying to wrap my head around
this. I might be wrong, but isn't there a possibility
that avg->period_contrib is increasing in PELTs accumulate_sum,
without _sum is increasing. This makes the pelt divider increase,
making the statement "_sum should be always larger or equal to _avg *"
false? Or am I missing something obvious here?
Still unable to reproduce what Sachin is reporting tho.
Odin
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 13:56:03
On Wed, 23 Jun 2021 at 14:37, Odin Ugedal [off-list ref] wrote:
ons. 23. jun. 2021 kl. 14:22 skrev Vincent Guittot [off-list ref]:
quoted
In theory it should not because _sum should be always larger or equal
to _avg * divider. Otherwise, it means that we have something wrong
somewhere else
Yeah, that might be the case. Still trying to wrap my head around
this. I might be wrong, but isn't there a possibility
that avg->period_contrib is increasing in PELTs accumulate_sum,
without _sum is increasing. This makes the pelt divider increase,
making the statement "_sum should be always larger or equal to _avg *"
false? Or am I missing something obvious here?
The pelt value of sched_entity is synced with cfs and its contrib
before being removed.
Then, we start to remove this load in update_cfs_rq_load_avg() before
calling __update_load_avg_cfs_rq so contrib should not have change and
we should be safe
Still unable to reproduce what Sachin is reporting tho.
Odin
ons. 23. jun. 2021 kl. 15:56 skrev Vincent Guittot [off-list ref]:
The pelt value of sched_entity is synced with cfs and its contrib
before being removed.
Hmm. Not sure what you mean by sched_entity here, since this is only
taking the "removed" load_avg
and removing it from cfs_rq, together with (removed.load_avg *
divider) from load_sum. (Although. ".removed" comes from
a sched entity)
Then, we start to remove this load in update_cfs_rq_load_avg() before
calling __update_load_avg_cfs_rq so contrib should not have change and
we should be safe
For what it is worth, I am now able to reproduce it (maybe
CONFIG_HZ=300/250 is the thing) as reported by Sachin,
and my patch makes it disappear. Without my patch I see situations
where _sum is zero while _avg is eg. 1 or 2 or 14 (in that range).
This happens for both load, runnable and util.
Lets see what Sachin reports back.
Thanks
Odin
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 15:25:24
On Wed, 23 Jun 2021 at 17:13, Odin Ugedal [off-list ref] wrote:
ons. 23. jun. 2021 kl. 15:56 skrev Vincent Guittot [off-list ref]:
quoted
The pelt value of sched_entity is synced with cfs and its contrib
before being removed.
Hmm. Not sure what you mean by sched_entity here, since this is only
taking the "removed" load_avg
and removing it from cfs_rq, together with (removed.load_avg *
divider) from load_sum. (Although. ".removed" comes from
a sched entity)
The sched_entity's load_avg that is put in removed.load, is sync with
the cfs_rq PELT signal, which includes contrib, before being added to
removed.load.
quoted
Then, we start to remove this load in update_cfs_rq_load_avg() before
calling __update_load_avg_cfs_rq so contrib should not have change and
we should be safe
For what it is worth, I am now able to reproduce it (maybe
CONFIG_HZ=300/250 is the thing) as reported by Sachin,
and my patch makes it disappear. Without my patch I see situations
where _sum is zero while _avg is eg. 1 or 2 or 14 (in that range).
hmm, so there is something wrong in the propagation
This happens for both load, runnable and util.
Lets see what Sachin reports back.
Thanks
Odin
From: Sachin Sant <hidden> Date: 2021-06-23 16:46:20
Ok. This becomes even more weird. Could you share your config file and more details about
you setup ?
Have you applied the patch below ?
https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
Regarding the load_avg warning, I can see possible problem during attach. Could you add
the patch below. The load_avg warning seems to happen during boot and sched_entity
creation.
Here is a summary of my testing.
I have a POWER box with PowerVM hypervisor. On this box I have a logical partition(LPAR) or guest
(allocated with 32 cpus 90G memory) running linux-next.
I started with a clean slate.
Moved to linux-next 5.13.0-rc7-next-20210622 as base code.
Applied patch #1 from Vincent which contains changes to dequeue_load_avg()
Applied patch #2 from Vincent which contains changes to enqueue_load_avg()
Applied patch #3 from Vincent which contains changes to attach_entity_load_avg()
Applied patch #4 from https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
With these changes applied I was still able to recreate the issue. I could see kernel warning
during boot.
I then applied patch #5 from Odin which contains changes to update_cfs_rq_load_avg()
With all the 5 patches applied I was able to boot the kernel without any warning messages.
I also ran scheduler related tests from ltp (./runltp -f sched) . All tests including cfs_bandwidth01
ran successfully. No kernel warnings were observed.
Have also attached .config in case it is useful. config has CONFIG_HZ_100=y
Thanks
-Sachin
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 16:55:24
On Wed, 23 Jun 2021 at 18:46, Sachin Sant [off-list ref] wrote:
quoted
Ok. This becomes even more weird. Could you share your config file and more details about
you setup ?
Have you applied the patch below ?
https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
Regarding the load_avg warning, I can see possible problem during attach. Could you add
the patch below. The load_avg warning seems to happen during boot and sched_entity
creation.
Here is a summary of my testing.
I have a POWER box with PowerVM hypervisor. On this box I have a logical partition(LPAR) or guest
(allocated with 32 cpus 90G memory) running linux-next.
I started with a clean slate.
Moved to linux-next 5.13.0-rc7-next-20210622 as base code.
Applied patch #1 from Vincent which contains changes to dequeue_load_avg()
Applied patch #2 from Vincent which contains changes to enqueue_load_avg()
Applied patch #3 from Vincent which contains changes to attach_entity_load_avg()
Applied patch #4 from https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
With these changes applied I was still able to recreate the issue. I could see kernel warning
during boot.
I then applied patch #5 from Odin which contains changes to update_cfs_rq_load_avg()
With all the 5 patches applied I was able to boot the kernel without any warning messages.
I also ran scheduler related tests from ltp (./runltp -f sched) . All tests including cfs_bandwidth01
ran successfully. No kernel warnings were observed.
ok so Odin's patch fixes the problem which highlights that we
overestimate _sum or don't sync _avg and _sum correctly
I'm going to look at this further
Have also attached .config in case it is useful. config has CONFIG_HZ_100=y
From: Vincent Guittot <vincent.guittot@linaro.org> Date: 2021-06-23 17:27:23
On Wed, 23 Jun 2021 at 18:55, Vincent Guittot
[off-list ref] wrote:
On Wed, 23 Jun 2021 at 18:46, Sachin Sant [off-list ref] wrote:
quoted
quoted
Ok. This becomes even more weird. Could you share your config file and more details about
you setup ?
Have you applied the patch below ?
https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
Regarding the load_avg warning, I can see possible problem during attach. Could you add
the patch below. The load_avg warning seems to happen during boot and sched_entity
creation.
Here is a summary of my testing.
I have a POWER box with PowerVM hypervisor. On this box I have a logical partition(LPAR) or guest
(allocated with 32 cpus 90G memory) running linux-next.
I started with a clean slate.
Moved to linux-next 5.13.0-rc7-next-20210622 as base code.
Applied patch #1 from Vincent which contains changes to dequeue_load_avg()
Applied patch #2 from Vincent which contains changes to enqueue_load_avg()
Applied patch #3 from Vincent which contains changes to attach_entity_load_avg()
Applied patch #4 from https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
With these changes applied I was still able to recreate the issue. I could see kernel warning
during boot.
I then applied patch #5 from Odin which contains changes to update_cfs_rq_load_avg()
With all the 5 patches applied I was able to boot the kernel without any warning messages.
I also ran scheduler related tests from ltp (./runltp -f sched) . All tests including cfs_bandwidth01
ran successfully. No kernel warnings were observed.
ok so Odin's patch fixes the problem which highlights that we
overestimate _sum or don't sync _avg and _sum correctly
I'm going to look at this further
The problem is "_avg * divider" makes the assumption that all pending
contrib are not null contributions whereas they can be null.
Odin patch is the right way to fix this. Other patches should not be
useful for your problem
quoted
Have also attached .config in case it is useful. config has CONFIG_HZ_100=y
ons. 23. jun. 2021 kl. 19:27 skrev Vincent Guittot [off-list ref]:
On Wed, 23 Jun 2021 at 18:55, Vincent Guittot
[off-list ref] wrote:
quoted
On Wed, 23 Jun 2021 at 18:46, Sachin Sant [off-list ref] wrote:
quoted
quoted
Ok. This becomes even more weird. Could you share your config file and more details about
you setup ?
Have you applied the patch below ?
https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
Regarding the load_avg warning, I can see possible problem during attach. Could you add
the patch below. The load_avg warning seems to happen during boot and sched_entity
creation.
Here is a summary of my testing.
I have a POWER box with PowerVM hypervisor. On this box I have a logical partition(LPAR) or guest
(allocated with 32 cpus 90G memory) running linux-next.
I started with a clean slate.
Moved to linux-next 5.13.0-rc7-next-20210622 as base code.
Applied patch #1 from Vincent which contains changes to dequeue_load_avg()
Applied patch #2 from Vincent which contains changes to enqueue_load_avg()
Applied patch #3 from Vincent which contains changes to attach_entity_load_avg()
Applied patch #4 from https://lore.kernel.org/lkml/20210621174330.11258-1-vincent.guittot@linaro.org/
With these changes applied I was still able to recreate the issue. I could see kernel warning
during boot.
I then applied patch #5 from Odin which contains changes to update_cfs_rq_load_avg()
With all the 5 patches applied I was able to boot the kernel without any warning messages.
I also ran scheduler related tests from ltp (./runltp -f sched) . All tests including cfs_bandwidth01
ran successfully. No kernel warnings were observed.
ok so Odin's patch fixes the problem which highlights that we
overestimate _sum or don't sync _avg and _sum correctly
I'm going to look at this further
The problem is "_avg * divider" makes the assumption that all pending
contrib are not null contributions whereas they can be null.
Yeah.
Odin patch is the right way to fix this. Other patches should not be
useful for your problem
Ack. As I see it, given how PELT works now, it is the only way to
mitigate it (without doing a lot of extra PELT stuff).
Will post it as a patch together with a proper message later today or tomorrow.
quoted
quoted
Have also attached .config in case it is useful. config has CONFIG_HZ_100=y