From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Fix this by not calling trace_tlbie() from native_hpte_clear().
Signed-off-by: Mahesh Salgaonkar <redacted>
Reported-by: Aneesh Kumar K.V <redacted>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/mm/hash_native_64.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
@@ -100,7 +101,15 @@ static inline void __tlbie(unsigned long vpn, int psize, int apsize, int ssize):"memory");break;}-trace_tlbie(0,0,va,0,0,0,0);+returnva;+}++staticinlinevoid__tlbie(unsignedlongvpn,intpsize,intapsize,intssize)+{+unsignedlongrb;++rb=___tlbie(vpn,psize,apsize,ssize);+trace_tlbie(0,0,rb,0,0,0,0);}staticinlinevoid__tlbiel(unsignedlongvpn,intpsize,intapsize,intssize)
From: Naveen N. Rao <hidden> Date: 2017-11-22 19:07:12
Mahesh J Salgaonkar wrote:
quoted hunk
From: Mahesh Salgaonkar <redacted>
=20
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
=20
Fix this by not calling trace_tlbie() from native_hpte_clear().
=20
Signed-off-by: Mahesh Salgaonkar <redacted>
Reported-by: Aneesh Kumar K.V <redacted>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/mm/hash_native_64.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
=20
Does it help if you use the _rcuidle variant instead, to turn off all=20
rcu checks for tracing __tlbie()?
trace_tlbie_rcuidle(0, 0, va, 0, 0, 0, 0);
- Naveen
=
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
I think the right thing in the longer run might be to do a TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is good.
Balbir Singh.
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Fix this by not calling trace_tlbie() from native_hpte_clear().
Signed-off-by: Mahesh Salgaonkar <redacted>
Reported-by: Aneesh Kumar K.V <redacted>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/mm/hash_native_64.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/mm/hash_native_64.c
b/arch/powerpc/mm/hash_native_64.c
index 3848af1..640cf56 100644
DEFINE_RAW_SPINLOCK(native_tlbie_lock);
-static inline void __tlbie(unsigned long vpn, int psize, int apsize,
int ssize)
+static inline unsigned long ___tlbie(unsigned long vpn, int psize,
+ int apsize, int ssize)
{
unsigned long va;
unsigned int penc;
@@ -100,7 +101,15 @@ static inline void __tlbie(unsigned long vpn, int
psize, int apsize, int ssize)
: "memory");
break;
}
- trace_tlbie(0, 0, va, 0, 0, 0, 0);
Does it help if you use the _rcuidle variant instead, to turn off all
rcu checks for tracing __tlbie()?
trace_tlbie_rcuidle(0, 0, va, 0, 0, 0, 0);
It helps if tracepoint is not enabled. But with tracepoint enabled kexec
still fails. I think we should not have tracepoint in kexec path at all.
If someone enables it, kexec will definitely fail regardless of
CONFIG_LOCKDEP.
Thanks,
-Mahesh.
From: Naveen N. Rao <hidden> Date: 2017-11-23 10:02:11
Mahesh Jagannath Salgaonkar wrote:
On 11/23/2017 12:37 AM, Naveen N. Rao wrote:
quoted
Mahesh J Salgaonkar wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN an=
d
quoted
quoted
causes kexec to fail.
Fix this by not calling trace_tlbie() from native_hpte_clear().
Signed-off-by: Mahesh Salgaonkar <redacted>
Reported-by: Aneesh Kumar K.V <redacted>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
---
=C2=A0arch/powerpc/mm/hash_native_64.c |=C2=A0=C2=A0 15 ++++++++++++---
=C2=A01 file changed, 12 insertions(+), 3 deletions(-)
[snip]
quoted
quoted
@@ -100,7 +101,15 @@ static inline void __tlbie(unsigned long vpn,=20
int
psize, int apsize, int ssize)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=20
Does it help if you use the _rcuidle variant instead, to turn off all
rcu checks for tracing __tlbie()?
=C2=A0=C2=A0=C2=A0=C2=A0trace_tlbie_rcuidle(0, 0, va, 0, 0, 0, 0);
=20
It helps if tracepoint is not enabled. But with tracepoint enabled kexec
still fails. I think we should not have tracepoint in kexec path at all.
If someone enables it, kexec will definitely fail regardless of
CONFIG_LOCKDEP.
Ok, thanks for confirming that other tracepoints don't interfere with=20
kexec. As Balbir points out, moving to TRACE_EVENT_CONDITION() with a=20
global in_kexec variable may be the other option, but is probably=20
overkill for a single tracepoint.
Acked-by: Naveen N. Rao <redacted>
Thanks,
Naveen
=
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
Yup. I missed it. Will resend the patch with fixes tag
Fixes: 0428491cba92 ("powerpc/mm: Trace tlbie(l) instructions")
I think the right thing in the longer run might be to do a TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is good.
Balbir Singh.
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-11-23 12:15:45
Balbir Singh [off-list ref] writes:
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
I think the right thing in the longer run might be to do a TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is good.
No I think the right thing is to not call trace points from kexec code,
it's too fragile. TRACE_EVENT_CONDITION wouldn't have saved us from this
RCU breakage.
cheers
From: Naveen N. Rao <hidden> Date: 2017-11-23 13:12:03
Michael Ellerman wrote:
Balbir Singh [off-list ref] writes:
=20
quoted
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN an=
d
quoted
quoted
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
I think the right thing in the longer run might be to do a TRACE_EVENT_C=
ONDITION
quoted
and have the condition do the right thing, but what you have for now is =
good.
=20
No I think the right thing is to not call trace points from kexec code,
it's too fragile. TRACE_EVENT_CONDITION wouldn't have saved us from this
RCU breakage.
I agree on the fragile part, though it appears to me that a=20
TRACE_EVENT_CONDITION() with a check for is_kexec (that needs to be=20
added) will prevent breakage since both the LOCKDEP block as well as the=20
tracepoint itself are guarded by the condition. So, none of the rcu code=20
should be executed as long as we set is_kexec at the right time. =20
However, since there are all of 1 tracepoint(s) affecting kexec, it is=20
probably not worth the effort at the moment.
- Naveen
=
On Fri, Nov 24, 2017 at 12:11 AM, Naveen N. Rao
[off-list ref] wrote:
Michael Ellerman wrote:
quoted
Balbir Singh [off-list ref] writes:
quoted
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
I think the right thing in the longer run might be to do a
TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is
good.
No I think the right thing is to not call trace points from kexec code,
it's too fragile. TRACE_EVENT_CONDITION wouldn't have saved us from this
RCU breakage.
I agree on the fragile part, though it appears to me that a
TRACE_EVENT_CONDITION() with a check for is_kexec (that needs to be added)
will prevent breakage since both the LOCKDEP block as well as the tracepoint
itself are guarded by the condition. So, none of the rcu code should be
executed as long as we set is_kexec at the right time. However, since there
are all of 1 tracepoint(s) affecting kexec, it is probably not worth the
effort at the moment.
+1, I am good with this change for now. I agree that we should not
call trace points
from kexec code, the tracepoint was for other paths, but we should
definitely avoid
this path.
Mahesh, is this path specific to hash or do we have similar issues in radix?
Balbir Singh.
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-11-24 06:10:10
Balbir Singh [off-list ref] writes:
On Fri, Nov 24, 2017 at 12:11 AM, Naveen N. Rao
[off-list ref] wrote:
quoted
Michael Ellerman wrote:
quoted
Balbir Singh [off-list ref] writes:
...
quoted
quoted
quoted
I think the right thing in the longer run might be to do a
TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is
good.
No I think the right thing is to not call trace points from kexec code,
it's too fragile. TRACE_EVENT_CONDITION wouldn't have saved us from this
RCU breakage.
I agree on the fragile part, though it appears to me that a
TRACE_EVENT_CONDITION() with a check for is_kexec (that needs to be added)
will prevent breakage since both the LOCKDEP block as well as the tracepoint
itself are guarded by the condition. So, none of the rcu code should be
executed as long as we set is_kexec at the right time. However, since there
are all of 1 tracepoint(s) affecting kexec, it is probably not worth the
effort at the moment.
+1, I am good with this change for now. I agree that we should not
call trace points
from kexec code, the tracepoint was for other paths, but we should
definitely avoid
this path.
Mahesh, is this path specific to hash or do we have similar issues in radix?
The radix code will trigger the tracepoints, so we should fix that. It
may not actually crash but that's secondary.
See:
mmu_cleanup_all()
-> radix__mmu_cleanup_all()
-> radix__flush_tlb_all()
-> trace_tlbie()
cheers
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-11-24 06:10:42
"Naveen N. Rao" [off-list ref] writes:
Michael Ellerman wrote:
quoted
Balbir Singh [off-list ref] writes:
quoted
On Thu, Nov 23, 2017 at 4:32 AM, Mahesh J Salgaonkar
[off-list ref] wrote:
quoted
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Effectively we can't enter the trace point code after we've set
HILE_BE. Do we need
a fixes tag? Or is this a side-effect of a new generic change?
I think the right thing in the longer run might be to do a TRACE_EVENT_CONDITION
and have the condition do the right thing, but what you have for now is good.
No I think the right thing is to not call trace points from kexec code,
it's too fragile. TRACE_EVENT_CONDITION wouldn't have saved us from this
RCU breakage.
I agree on the fragile part, though it appears to me that a
TRACE_EVENT_CONDITION() with a check for is_kexec (that needs to be
added) will prevent breakage since both the LOCKDEP block as well as the
tracepoint itself are guarded by the condition. So, none of the rcu code
should be executed as long as we set is_kexec at the right time.
Yes you're right, I misread that.
So maybe that is an option. But it still makes me nervous :)
cheers
From: Michael Ellerman <hidden> Date: 2017-11-24 09:46:40
On Wed, 2017-11-22 at 17:32:07 UTC, Mahesh J Salgaonkar wrote:
From: Mahesh Salgaonkar <redacted>
Rebooting into a new kernel with kexec fails in trace_tlbie() which is
called from native_hpte_clear(). This happens if the running kernel has
CONFIG_LOCKDEP enabled. With lockdep enabled, the tracepoints always
execute few RCU checks regardless of whether tracing is on or off.
We are already in the last phase of kexec sequence in real mode with
HILE_BE set. At this point the RCU check ends up in RCU_LOCKDEP_WARN and
causes kexec to fail.
Fix this by not calling trace_tlbie() from native_hpte_clear().
Signed-off-by: Mahesh Salgaonkar <redacted>
Reported-by: Aneesh Kumar K.V <redacted>
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
Acked-by: Naveen N. Rao <redacted>