[PATCH ppc-next] powerpc/fsl-booke: don't load early TLB at once

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries
at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voilá, the P4080 boots again.

Cc: Scott Wood <redacted>
Signed-off-by: David Lamparter <redacted>
---
I'm not subscribed to linuxppc-dev, please keep me in Cc:.  Also I
contribute to the kernel only very rarely and have no clue how to find
out when the appropriate time in the merge window is to submit things to
linuxppc-dev... my apologies.
---
 arch/powerpc/mm/fsl_booke_mmu.c  |  3 +-
 arch/powerpc/mm/mmu_decl.h       |  1 -
 arch/powerpc/mm/tlb_nohash_low.S | 63 --------------------------------
 3 files changed, 2 insertions(+), 65 deletions(-)

diff --git a/arch/powerpc/mm/fsl_booke_mmu.c b/arch/powerpc/mm/fsl_booke_mmu.c
index 080d49b26c3a..f301a4c32c66 100644
--- a/arch/powerpc/mm/fsl_booke_mmu.c
+++ b/arch/powerpc/mm/fsl_booke_mmu.c

@@ -143,6 +143,8 @@ static void settlbcam(int index, unsigned long virt, phys_addr_t phys,
 	tlbcam_addrs[index].start = virt;
 	tlbcam_addrs[index].limit = virt + size - 1;
 	tlbcam_addrs[index].phys = phys;
+
+	loadcam_entry(index);
 }
 
 unsigned long calc_cam_sz(unsigned long ram, unsigned long virt,

@@ -195,7 +197,6 @@ static unsigned long map_mem_in_cams_addr(phys_addr_t phys, unsigned long virt,
 	if (dryrun)
 		return amount_mapped;
 
-	loadcam_multi(0, i, max_cam_idx);
 	tlbcam_index = i;
 
 #ifdef CONFIG_PPC64

diff --git a/arch/powerpc/mm/mmu_decl.h b/arch/powerpc/mm/mmu_decl.h
index e5d779eed181..9a14de6a3ca9 100644
--- a/arch/powerpc/mm/mmu_decl.h
+++ b/arch/powerpc/mm/mmu_decl.h

@@ -140,7 +140,6 @@ extern int switch_to_as1(void);
 extern void restore_to_as0(int esel, int offset, void *dt_ptr, int bootcpu);
 #endif
 extern void loadcam_entry(unsigned int index);
-extern void loadcam_multi(int first_idx, int num, int tmp_idx);
 
 struct tlbcam {
 	u32	MAS0;

diff --git a/arch/powerpc/mm/tlb_nohash_low.S b/arch/powerpc/mm/tlb_nohash_low.S
index e066a658acac..b5fa6c6a940f 100644
--- a/arch/powerpc/mm/tlb_nohash_low.S
+++ b/arch/powerpc/mm/tlb_nohash_low.S

@@ -402,7 +402,6 @@ _GLOBAL(set_context)
  * extern void loadcam_entry(unsigned int index)
  *
  * Load TLBCAM[index] entry in to the L2 CAM MMU
- * Must preserve r7, r8, r9, and r10
  */
 _GLOBAL(loadcam_entry)
 	mflr	r5

@@ -426,66 +425,4 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_BIG_PHYS)
 	tlbwe
 	isync
 	blr
-
-/*
- * Load multiple TLB entries at once, using an alternate-space
- * trampoline so that we don't have to care about whether the same
- * TLB entry maps us before and after.
- *
- * r3 = first entry to write
- * r4 = number of entries to write
- * r5 = temporary tlb entry
- */
-_GLOBAL(loadcam_multi)
-	mflr	r8
-
-	/*
-	 * Set up temporary TLB entry that is the same as what we're
-	 * running from, but in AS=1.
-	 */
-	bl	1f
-1:	mflr	r6
-	tlbsx	0,r8
-	mfspr	r6,SPRN_MAS1
-	ori	r6,r6,MAS1_TS
-	mtspr	SPRN_MAS1,r6
-	mfspr	r6,SPRN_MAS0
-	rlwimi	r6,r5,MAS0_ESEL_SHIFT,MAS0_ESEL_MASK
-	mr	r7,r5
-	mtspr	SPRN_MAS0,r6
-	isync
-	tlbwe
-	isync
-
-	/* Switch to AS=1 */
-	mfmsr	r6
-	ori	r6,r6,MSR_IS|MSR_DS
-	mtmsr	r6
-	isync
-
-	mr	r9,r3
-	add	r10,r3,r4
-2:	bl	loadcam_entry
-	addi	r9,r9,1
-	cmpw	r9,r10
-	mr	r3,r9
-	blt	2b
-
-	/* Return to AS=0 and clear the temporary entry */
-	mfmsr	r6
-	rlwinm.	r6,r6,0,~(MSR_IS|MSR_DS)
-	mtmsr	r6
-	isync
-
-	li	r6,0
-	mtspr	SPRN_MAS1,r6
-	rlwinm	r6,r7,MAS0_ESEL_SHIFT,MAS0_ESEL_MASK
-	oris	r6,r6,MAS0_TLBSEL(1)@h
-	mtspr	SPRN_MAS0,r6
-	isync
-	tlbwe
-	isync
-
-	mtlr	r8
-	blr
 #endif

-- 
2.18.0

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries
at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voilá, the P4080 boots again.

It's not an optimization; it was required to get kdump working, at least
for certain choices of crash kernel location.  I just tried booting a 32-
bit kernel and did not see this problem -- but I don't have access to a
p4080ds anymore.  I tried with qemu e500mc, and also running a 32-bit
kernel on e6500 (needs a tiny change to get past SMP init, since 32-bit
isn't really supported on e6500, but you do get output even without that).

Do you have a JTAG that can be used to find out where it's hanging?  If
not, I can try to get early debug working (needs an early IOMMU mapping).

York, can you try booting the latest kernel on p4080ds?

-Scott

On Thu, Sep 20, 2018 at 07:31:33PM -0500, Scott Wood wrote:

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:

quoted

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries
at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voilá, the P4080 boots again.

It's not an optimization; it was required to get kdump working, at least

Oh, my apologies for misunderstanding the code.

for certain choices of crash kernel location.  I just tried booting a 32-
bit kernel and did not see this problem -- but I don't have access to a
p4080ds anymore.  I tried with qemu e500mc, and also running a 32-bit
kernel on e6500 (needs a tiny change to get past SMP init, since 32-bit
isn't really supported on e6500, but you do get output even without that).

Hrm, maybe I should just make it use the old mechanism under an #ifdef
CONFIG_PPC32?  Better to boot and have kdump not work than not boot.
(But obviously finding the actual problem would be best.)

Do you have a JTAG that can be used to find out where it's hanging?  If
not, I can try to get early debug working (needs an early IOMMU mapping).

I only have JTAG tools for ARM chips available; hardware wise I could
probably solder up an adapter but software wise I have absoutely no clue
how to fire up a session on anything PPC...  I'm a novice openocd user,
that's it.

York, can you try booting the latest kernel on p4080ds?

d9e1831a42 has been around for quite some time, 4.4 already has it.  I
was a bit surprised noone has run into this, but then again P4080 is not
exactly the most recent/interesting hardware.

Cheers,


-David

On 09/20/2018 05:31 PM, Scott Wood wrote:=0A=

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:=0A=

quoted

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries=

=0A=

quoted

at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)=0A=
=0A=
My dusty old P4080DS just completely fails to boot (no output at all)=0A=
without this revert.  I have no clue what's going on here, I just=0A=
bisected it down and since it looks like an optimization to me I just=0A=
reverted it - and voil=E1, the P4080 boots again.=0A=

=0A=
It's not an optimization; it was required to get kdump working, at least=

=0A=

for certain choices of crash kernel location.  I just tried booting a 32-=

=0A=

bit kernel and did not see this problem -- but I don't have access to a=

=0A=

p4080ds anymore.  I tried with qemu e500mc, and also running a 32-bit=0A=
kernel on e6500 (needs a tiny change to get past SMP init, since 32-bit=

=0A=

isn't really supported on e6500, but you do get output even without that)=

.=0A=

=0A=
Do you have a JTAG that can be used to find out where it's hanging?  If=

=0A=

not, I can try to get early debug working (needs an early IOMMU mapping).=

=0A=

=0A=
York, can you try booting the latest kernel on p4080ds?=0A=
=0A=

=0A=
Scott,=0A=
=0A=
I haven't tried P4080DS for a long time. What defconfig do you use for=0A=
this board? I tried latest master branch (commit a27fb6d983c7b5) with=0A=
corenet_basic_defconfig, it didn't boot up. Kernel has an exception very=0A=
early (pc ffffae80).=0A=
=0A=
However, before I claimed the board, someone booted Linux 4.14.71 on=0A=
this board. I need to track down where the image came from.=0A=
=0A=
York=0A=

On Fri, 2018-09-21 at 17:40 +0000, York Sun wrote:

On 09/20/2018 05:31 PM, Scott Wood wrote:

quoted

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:

quoted

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries
at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voilá, the P4080 boots again.

It's not an optimization; it was required to get kdump working, at least
for certain choices of crash kernel location.  I just tried booting a 32-
bit kernel and did not see this problem -- but I don't have access to a
p4080ds anymore.  I tried with qemu e500mc, and also running a 32-bit
kernel on e6500 (needs a tiny change to get past SMP init, since 32-bit
isn't really supported on e6500, but you do get output even without that).

Do you have a JTAG that can be used to find out where it's hanging?  If
not, I can try to get early debug working (needs an early IOMMU mapping).

York, can you try booting the latest kernel on p4080ds?

Scott,

I haven't tried P4080DS for a long time. What defconfig do you use for
this board? I tried latest master branch (commit a27fb6d983c7b5) with
corenet_basic_defconfig, it didn't boot up. Kernel has an exception very
early (pc ffffae80).

However, before I claimed the board, someone booted Linux 4.14.71 on
this board. I need to track down where the image came from.

Use corenet32_smp_defconfig

corenet_basic_defconfig is just a fragment used by the makefiles in assembling
the final config.

-Scott

On 09/21/2018 10:47 AM, Scott Wood wrote:=0A=

On Fri, 2018-09-21 at 17:40 +0000, York Sun wrote:=0A=

quoted

On 09/20/2018 05:31 PM, Scott Wood wrote:=0A=

quoted

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:=0A=

quoted

This is a *partial* revert of "powerpc/85xx: Load all early TLB entrie=

s=0A=

quoted

quoted

quoted

at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)=0A=
=0A=
My dusty old P4080DS just completely fails to boot (no output at all)=

=0A=

quoted

quoted

quoted

without this revert.  I have no clue what's going on here, I just=0A=
bisected it down and since it looks like an optimization to me I just=

=0A=

quoted

quoted

quoted

reverted it - and voil=E1, the P4080 boots again.=0A=

=0A=
It's not an optimization; it was required to get kdump working, at leas=

t=0A=

quoted

quoted

for certain choices of crash kernel location.  I just tried booting a 3=

2-=0A=

quoted

quoted

bit kernel and did not see this problem -- but I don't have access to a=

=0A=

quoted

quoted

p4080ds anymore.  I tried with qemu e500mc, and also running a 32-bit=

=0A=

quoted

quoted

kernel on e6500 (needs a tiny change to get past SMP init, since 32-bit=

=0A=

quoted

quoted

isn't really supported on e6500, but you do get output even without tha=

t).=0A=

quoted

quoted

=0A=
Do you have a JTAG that can be used to find out where it's hanging?  If=

=0A=

quoted

quoted

not, I can try to get early debug working (needs an early IOMMU mapping=

).=0A=

quoted

quoted

=0A=
York, can you try booting the latest kernel on p4080ds?=0A=
=0A=

=0A=
Scott,=0A=
=0A=
I haven't tried P4080DS for a long time. What defconfig do you use for=

=0A=

quoted

this board? I tried latest master branch (commit a27fb6d983c7b5) with=0A=
corenet_basic_defconfig, it didn't boot up. Kernel has an exception very=

=0A=

quoted

early (pc ffffae80).=0A=
=0A=
However, before I claimed the board, someone booted Linux 4.14.71 on=0A=
this board. I need to track down where the image came from.=0A=

=0A=
Use corenet32_smp_defconfig=0A=
=0A=
corenet_basic_defconfig is just a fragment used by the makefiles in assem=

bling=0A=

the final config.=0A=
=0A=

=0A=
Thanks for the instruction. Linux comes up OK with corenet32_smp_defconfig.=
=0A=
=0A=
root@p4080ds:~# uname -a=0A=
Linux p4080ds 4.19.0-rc4-00206-ga27fb6d #1 SMP Fri Sep 21 10:56:36 PDT=0A=
2018 ppc GNU/Linux=0A=
=0A=
York=0A=
=0A=

On Fri, Sep 21, 2018 at 03:00:19AM +0200, David Lamparter wrote:

On Thu, Sep 20, 2018 at 07:31:33PM -0500, Scott Wood wrote:

quoted

Do you have a JTAG that can be used to find out where it's hanging?  If
not, I can try to get early debug working (needs an early IOMMU mapping).

s/IOMMU mapping/MMU mapping for MMIO/ :-P

I only have JTAG tools for ARM chips available; hardware wise I could
probably solder up an adapter but software wise I have absoutely no clue
how to fire up a session on anything PPC...  I'm a novice openocd user,
that's it.

quoted

York, can you try booting the latest kernel on p4080ds?

d9e1831a42 has been around for quite some time, 4.4 already has it.  I
was a bit surprised noone has run into this, but then again P4080 is not
exactly the most recent/interesting hardware.

I don't suppose you're running a relocatable kernel at a non-zero
address, and/or are running in an environment that sets
HID0[EN_L2MMU_MHD] (neither standard U-boot nor Linux sets this bit,
though they probably should)?  On 32-bit, we're already running in an AS1
trampoline when loadcam_multi() is called, but loadcam_multi() sets up
its own.  This happens to not be catastrophic in standard scenarios, but
it does add a duplicate TLB entry, and we return to AS0 sooner than
expected.  I think your patch, plus ifdefs to make the change 32-bit
only, is the appropriate fix.

I also got an earlier udbg for e500 working (and happened to decide to
test it with a relocatable kernel); I'll send that out once I've cleaned
it up (or sooner with the extra TLB dumping included if the above doesn't
explain why you're hitting this bug).

-Scott

David Lamparter [off-list ref] writes:

This is a *partial* revert of "powerpc/85xx: Load all early TLB entries
at once" (d9e1831a420267a7ced708bb259d65b0a3c0344d.)

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voil=C3=A1, the P4080 boots again.

Cc: Scott Wood <redacted>
Signed-off-by: David Lamparter <redacted>
---
I'm not subscribed to linuxppc-dev, please keep me in Cc:.  Also I
contribute to the kernel only very rarely and have no clue how to find
out when the appropriate time in the merge window is to submit things to
linuxppc-dev... my apologies.

Don't worry about the timing of sending patches.

Your patch will be tracked in patchwork, so if it's not a good time
we'll just ignore it until it is a good time :)

cheers

(Sorry about the delay on my end, deadlines ...)

On Fri, Sep 21, 2018 at 06:07:48PM +0000, York Sun wrote:

On 09/21/2018 10:47 AM, Scott Wood wrote:

quoted

On Fri, 2018-09-21 at 17:40 +0000, York Sun wrote:

quoted

On 09/20/2018 05:31 PM, Scott Wood wrote:

quoted

On Fri, 2018-09-21 at 00:48 +0200, David Lamparter wrote:

quoted

My dusty old P4080DS just completely fails to boot (no output at all)
without this revert.  I have no clue what's going on here, I just
bisected it down and since it looks like an optimization to me I just
reverted it - and voilá, the P4080 boots again.

It's not an optimization; it was required to get kdump working, at least
for certain choices of crash kernel location.

[...]

quoted

quoted

quoted

York, can you try booting the latest kernel on p4080ds?

[...]

Thanks for the instruction. Linux comes up OK with corenet32_smp_defconfig.

root@p4080ds:~# uname -a
Linux p4080ds 4.19.0-rc4-00206-ga27fb6d #1 SMP Fri Sep 21 10:56:36 PDT
2018 ppc GNU/Linux

Well, shoot.  I guess the next likely thing is that I have something
weird enabled in my .config, so I'll try booting corenet32_smp_defconfig
on my P4080DS.  (I vaguely think I did that at some point but I can't
swear to it so I'll just retry.)  If that doesn't boot, could either of
you provide me with your built uImage so I can exclude toolchain
stupidity?

=> I'll send another mail when I got to try corenet32_smp_defconfig.


Aaaand, during the past week I noticed this is a rev1.0 P4080 chip, which
apparently only exists on some early P4080DS boards - and Freescale even
did a replacement program to get rev2.0 boards out (I guess this one was
sitting in some broom closet.)  I don't have access to erratas for this,
I only know the entire QMan/FMan stuff is royally f*cked - no idea
whether the PPC core has issues too.

FWIW, my board is running perfectly stable with that patch I posted and
I'll just carry it locally if it's an issue for this one specific board
I have here.  I just don't have sufficient information to tell if that
is indeed the case.

Thanks a lot for your input and help,


-David

On Sat, Sep 22, 2018 at 12:45:16AM -0500, Scott Wood wrote:

I don't suppose you're running a relocatable kernel at a non-zero

# CONFIG_RELOCATABLE is not set

address, and/or are running in an environment that sets
HID0[EN_L2MMU_MHD] (neither standard U-boot nor Linux sets this bit,

It's u-boot v2018.09-rc3;  bit "33" / 30 isn't even defined as a
constant, much less touched anywhere.  HID0 in u-boot seems to be
0x80000080 aka EMCP | EN_MAS7_UPDATE.

though they probably should)?  On 32-bit, we're already running in an AS1
trampoline when loadcam_multi() is called, but loadcam_multi() sets up
its own.  This happens to not be catastrophic in standard scenarios, but
it does add a duplicate TLB entry, and we return to AS0 sooner than
expected.  I think your patch, plus ifdefs to make the change 32-bit
only, is the appropriate fix.

I'll resend it with ifdefs inserted.  Meanwhile, still trying the
default config as mentioned in my previous mail.

I also got an earlier udbg for e500 working (and happened to decide to
test it with a relocatable kernel); I'll send that out once I've cleaned
it up (or sooner with the extra TLB dumping included if the above doesn't
explain why you're hitting this bug).

Thanks & Cheers,


-David