Re: arm64 regression in kernel 5.12 related to the (n)VHE

On Thu, 12 Aug 2021 09:24:14 +0100,
Rafał Miłecki [off-list ref] wrote:

On 12.08.2021 09:57, Marc Zyngier wrote:

quoted

On Thu, 12 Aug 2021 08:32:02 +0100,
Rafał Miłecki [off-list ref] wrote:

quoted

On 12.08.2021 08:51, Marc Zyngier wrote:

quoted

Interestingly, all your CPUs are booting at EL2. Which is great.  Can
you try and enable KVM on your existing 5.10 kernel? Just selecting
CONFIG_KVM should be enough. Does it boot correctly with KVM enabled?

My suspicion is that the firmware doesn't set SCR_EL3.HCE, and that
the HVC instruction UNDEFs at EL1. That would be bad news.

Interesting! I had to enable CONFIG_VIRTUALIZATION and CONFIG_NET first.
First I verified kernel built with those options still boots. It does.

Then I enabled CONFIG_KVM and kernel seems to hang around switching from
bootconsole to the console.

Starting program at 0x0000000000080000
/memory = 0x40000000
WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined
WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined
WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined
Booting Linux on physical CPU 0x0000000000 [0x420f1000]
Linux version 5.11.22-g0453a426c37b (rmilecki@localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #8 SMP Thu Aug 12 09:25:55 CEST 2021
Machine model: Asus GT-AC5300
earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
printk: bootconsole [bcm63xx_uart0] enabled
efi: UEFI not found.
[Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
Zone ranges:
   DMA      [mem 0x0000000000000000-0x000000003fffffff]
   DMA32    empty
   Normal   empty
Movable zone start for each node
Early memory node ranges
   node   0: [mem 0x0000000000000000-0x000000003fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
percpu: Embedded 18 pages/cpu s43904 r0 d29824 u73728
Detected VIPT I-cache on CPU0
CPU features: detected: ARM erratum 843419
Built 1 zonelists, mobility grouping on.  Total pages: 258048
Kernel command line: earlycon=bcm63xx_uart,0xff800640
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 1019556K/1048576K available (4352K kernel code, 678K rwdata, 860K rodata, 2496K init, 232K bss, 29020K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GIC: Using split EOI/Deactivate mode
random: get_random_bytes called from start_kernel+0x33c/0x52c with crng_init=0
arch_timer: cp15 timer(s) running at 50.00MHz (phys).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
Console: colour dummy device 80x25
printk: console [tty0] enabled
printk: bootconsole [bcm63xx_uart0] disabled


(Unless it's a false conclusion and CONFIG_KVM just breaks console
somehow)

No, that's because you don't pass the right console to your
kernel. Add something like "console=ttyS0,115200" to the kernel
command line, which will show what you are missing, as well as stop
the double-logging.

Anyway, the fact that it stops booting when you enable KVM confirms my
suspicion. The firmware on this system is probably crap enough not to
enable HVC. Let's confirm it further: please apply the patch below on
top of mainline and tell me that it now boots fine...

Thanks for the patch! It workarounds the issue. See below.

quoted

Are you in a position where you can actually fix the firmware? Or is
it some closed-source blob?

I'm just an end-user with no access to CFE sources and without any
business contact as Broadcom :(

I feared that would be the case. Florian's reply seems to indicate
that the "upstream" firmware implementation is correct, so the OEM
must have fumbled it somehow...

quoted

diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
index 43d212618834..fc95b103ef42 100644
--- a/arch/arm64/kernel/hyp-stub.S
+++ b/arch/arm64/kernel/hyp-stub.S

@@ -238,7 +238,7 @@ SYM_FUNC_START(switch_to_vhe)
    	// Turn the world upside down
  	mov	x0, #HVC_VHE_RESTART
-	hvc	#0
+//	hvc	#0
  1:
  	ret
  SYM_FUNC_END(switch_to_vhe)

This allows me to boot 5.13.9 and 5.14-rc5 without any reverts!

Enabling CONFIG_KVM still results in the:
Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt

That's expected. Can you please check the patch below? It should
result in a booting kernel which actually survives having KVM compiled
in. It should even display a warning telling you that your setup is
completely buggered.

That's obviously not the final version, but probably a good enough
approximation.

Thanks,

	M.

diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
index 43d212618834..74a130808b38 100644
--- a/arch/arm64/kernel/hyp-stub.S
+++ b/arch/arm64/kernel/hyp-stub.S

@@ -46,6 +46,14 @@ SYM_CODE_END(__hyp_stub_vectors)
 	.align 11
 
 SYM_CODE_START_LOCAL(elx_sync)
+	mrs	x4, CurrentEL
+	cmp	x4, #CurrentEL_EL1
+	b.ne	0f
+	mrs	x4, esr_el1
+	eor	x4, x4, #ESR_ELx_IL
+	cbz	x4, el1_undef
+
+0:
 	cmp	x0, #HVC_SET_VECTORS
 	b.ne	1f
 	msr	vbar_el2, x1

@@ -71,6 +79,18 @@ SYM_CODE_START_LOCAL(elx_sync)
 
 9:	mov	x0, xzr
 	eret
+
+el1_undef:
+	# Downgrade the sucker to EL1...
+	adr_l	x1, __boot_cpu_mode
+	mov	w0, #BOOT_CPU_MODE_EL1
+	str	w0, [x1]
+
+	mrs	x0, elr_el1
+	add	x0, x0, #4
+	msr	elr_el1, x0
+	mov_q	x0, HVC_STUB_ERR
+	eret
 SYM_CODE_END(elx_sync)
 
 // nVHE? No way! Give me the real thing!

@@ -236,6 +256,19 @@ SYM_FUNC_START(switch_to_vhe)
 	cmp	x0, #CurrentEL_EL1
 	b.ne	1f
 
+	// Check that HVC actually works...
+	adr_l	x1, __hyp_stub_vectors
+	msr	vbar_el1, x1
+	isb
+	mov	x0, #HVC_RESET_VECTORS
+	hvc	#0
+	adr_l	x1, vectors
+	msr	vbar_el1, x1
+	isb
+	mov_q	x1, HVC_STUB_ERR
+	cmp	x0, x1
+	b.eq	1f
+
 	// Turn the world upside down
 	mov	x0, #HVC_VHE_RESTART
 	hvc	#0

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel