Re: arm64 regression in kernel 5.12 related to the (n)VHE
From: Rafał Miłecki <zajec5@gmail.com>
Date: 2021-08-12 08:27:00
On 12.08.2021 09:57, Marc Zyngier wrote:
On Thu, 12 Aug 2021 08:32:02 +0100, Rafał Miłecki [off-list ref] wrote:quoted
On 12.08.2021 08:51, Marc Zyngier wrote:quoted
Interestingly, all your CPUs are booting at EL2. Which is great. Can you try and enable KVM on your existing 5.10 kernel? Just selecting CONFIG_KVM should be enough. Does it boot correctly with KVM enabled? My suspicion is that the firmware doesn't set SCR_EL3.HCE, and that the HVC instruction UNDEFs at EL1. That would be bad news.Interesting! I had to enable CONFIG_VIRTUALIZATION and CONFIG_NET first. First I verified kernel built with those options still boots. It does. Then I enabled CONFIG_KVM and kernel seems to hang around switching from bootconsole to the console. Starting program at 0x0000000000080000 /memory = 0x40000000 WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined Booting Linux on physical CPU 0x0000000000 [0x420f1000] Linux version 5.11.22-g0453a426c37b (rmilecki@localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #8 SMP Thu Aug 12 09:25:55 CEST 2021 Machine model: Asus GT-AC5300 earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '') printk: bootconsole [bcm63xx_uart0] enabled efi: UEFI not found. [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader! Zone ranges: DMA [mem 0x0000000000000000-0x000000003fffffff] DMA32 empty Normal empty Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000000000-0x000000003fffffff] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff] percpu: Embedded 18 pages/cpu s43904 r0 d29824 u73728 Detected VIPT I-cache on CPU0 CPU features: detected: ARM erratum 843419 Built 1 zonelists, mobility grouping on. Total pages: 258048 Kernel command line: earlycon=bcm63xx_uart,0xff800640 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) mem auto-init: stack:off, heap alloc:off, heap free:off Memory: 1019556K/1048576K available (4352K kernel code, 678K rwdata, 860K rodata, 2496K init, 232K bss, 29020K reserved, 0K cma-reserved) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 rcu: Hierarchical RCU implementation. rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 GIC: Using split EOI/Deactivate mode random: get_random_bytes called from start_kernel+0x33c/0x52c with crng_init=0 arch_timer: cp15 timer(s) running at 50.00MHz (phys). clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns Console: colour dummy device 80x25 printk: console [tty0] enabled printk: bootconsole [bcm63xx_uart0] disabled (Unless it's a false conclusion and CONFIG_KVM just breaks console somehow)No, that's because you don't pass the right console to your kernel. Add something like "console=ttyS0,115200" to the kernel command line, which will show what you are missing, as well as stop the double-logging. Anyway, the fact that it stops booting when you enable KVM confirms my suspicion. The firmware on this system is probably crap enough not to enable HVC. Let's confirm it further: please apply the patch below on top of mainline and tell me that it now boots fine...
Thanks for the patch! It workarounds the issue. See below.
Are you in a position where you can actually fix the firmware? Or is it some closed-source blob?
I'm just an end-user with no access to CFE sources and without any business contact as Broadcom :(
quoted hunk ↗ jump to hunk
diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 43d212618834..fc95b103ef42 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S@@ -238,7 +238,7 @@ SYM_FUNC_START(switch_to_vhe) // Turn the world upside down mov x0, #HVC_VHE_RESTART - hvc #0 +// hvc #0 1: ret SYM_FUNC_END(switch_to_vhe)
This allows me to boot 5.13.9 and 5.14-rc5 without any reverts! Enabling CONFIG_KVM still results in the: Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt Starting program at 0x0000000000080000 /memory = 0x40000000 WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined Booting Linux on physical CPU 0x0000000000 [0x420f1000] Linux version 5.14.0-rc5-g9c6405c34362-dirty (rmilecki@localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #4 SMP Thu Aug 12 10:11:31 CEST 2021 Machine model: Asus GT-AC5300 earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '') printk: bootconsole [bcm63xx_uart0] enabled efi: UEFI not found. [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader! Zone ranges: DMA [mem 0x0000000000000000-0x000000003fffffff] DMA32 empty Normal empty Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000000000-0x000000003fffffff] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff] percpu: Embedded 17 pages/cpu s38112 r0 d31520 u69632 Detected VIPT I-cache on CPU0 CPU features: detected: ARM erratum 843419 Built 1 zonelists, mobility grouping on. Total pages: 258048 Kernel command line: earlycon=bcm63xx_uart,0xff800640 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) mem auto-init: stack:off, heap alloc:off, heap free:off Memory: 1020452K/1048576K available (3648K kernel code, 654K rwdata, 708K rodata, 2432K init, 228K bss, 28124K reserved, 0K cma-reserved) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 rcu: Hierarchical RCU implementation. rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 Root IRQ handler: gic_handle_irq GIC: Using split EOI/Deactivate mode random: get_random_bytes called from start_kernel+0x4a0/0x6dc with crng_init=0 arch_timer: cp15 timer(s) running at 50.00MHz (phys). clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns Console: colour dummy device 80x25 printk: console [tty0] enabled printk: bootconsole [bcm63xx_uart0] disabled Booting Linux on physical CPU 0x0000000000 [0x420f1000] Linux version 5.14.0-rc5-g9c6405c34362-dirty (rmilecki@localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #4 SMP Thu Aug 12 10:11:31 CEST 2021 Machine model: Asus GT-AC5300 earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '') printk: bootconsole [bcm63xx_uart0] enabled efi: UEFI not found. [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader! Zone ranges: DMA [mem 0x0000000000000000-0x000000003fffffff] DMA32 empty Normal empty Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000000000-0x000000003fffffff] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff] percpu: Embedded 17 pages/cpu s38112 r0 d31520 u69632 Detected VIPT I-cache on CPU0 CPU features: detected: ARM erratum 843419 Built 1 zonelists, mobility grouping on. Total pages: 258048 Kernel command line: earlycon=bcm63xx_uart,0xff800640 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) mem auto-init: stack:off, heap alloc:off, heap free:off Memory: 1020452K/1048576K available (3648K kernel code, 654K rwdata, 708K rodata, 2432K init, 228K bss, 28124K reserved, 0K cma-reserved) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 rcu: Hierarchical RCU implementation. rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 Root IRQ handler: gic_handle_irq GIC: Using split EOI/Deactivate mode random: get_random_bytes called from start_kernel+0x4a0/0x6dc with crng_init=0 arch_timer: cp15 timer(s) running at 50.00MHz (phys). clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns Console: colour dummy device 80x25 printk: console [tty0] enabled printk: bootconsole [bcm63xx_uart0] disabled Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000) pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear) Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear) rcu: Hierarchical SRCU implementation. EFI services will not be available. smp: Bringing up secondary CPUs ... Detected VIPT I-cache on CPU1 CPU1: Booted secondary processor 0x0000000001 [0x420f1000] Detected VIPT I-cache on CPU2 CPU2: Booted secondary processor 0x0000000002 [0x420f1000] Detected VIPT I-cache on CPU3 CPU3: Booted secondary processor 0x0000000003 [0x420f1000] smp: Brought up 1 node, 4 CPUs SMP: Total of 4 processors activated. CPU features: detected: 32-bit EL0 Support CPU features: detected: CRC32 instructions CPU: All CPU(s) started at EL2 alternatives: patching kernel code devtmpfs: initialized clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns futex hash table entries: 1024 (order: 4, 65536 bytes, linear) pinctrl core: initialized pinctrl subsystem DMI not present or invalid. DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations thermal_sys: Registered thermal governor 'step_wise' ASID allocator initialised with 65536 entries iommu: Default domain type: Translated vgaarb: loaded usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb clocksource: Switched to clocksource arch_sys_counter PCI: CLS 0 bytes, default 64 workingset: timestamp_bits=62 max_order=18 bucket_order=0 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) io scheduler mq-deadline registered io scheduler kyber registered basic-mmio-gpio: probe of ff800500.gpio-controller failed with error -22 ff800640.serial: ttyS0 at MMIO 0xff800640 (irq = 24, base_baud = 1562500) is a bcm63xx_uart printk: console [ttyS0] enabled ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci-pci: EHCI PCI platform driver ehci-platform: EHCI generic platform driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver ohci-pci: OHCI PCI platform driver ohci-platform: OHCI generic platform driver i2c /dev entries driver usbcore: registered new interface driver usbhid usbhid: USB HID core driver brcmstb-usb-phy 8000c200.usb-phy: Clock not found in Device Tree brcmstb-usb-phy 8000c200.usb-phy: USB3.0 clock not found in Device Tree brcmstb-usb-phy 8000c200.usb-phy: Suspend Clock not found in Device Tree brcmstb-usb-phy 8000c200.usb-phy: IRQ wake not found brcmstb-usb-phy 8000c200.usb-phy: IRQ wakeup not found brcmstb-usb-phy 8000c200.usb-phy: Wake interrupt missing, system wake not supported ehci-platform 8000c300.usb: EHCI Host Controller ehci-platform 8000c300.usb: new USB bus registered, assigned bus number 1 ehci-platform 8000c300.usb: irq 19, io mem 0x8000c300 ehci-platform 8000c300.usb: USB 2.0 started, EHCI 1.00 hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ohci-platform 8000c400.usb: Generic Platform OHCI controller ohci-platform 8000c400.usb: new USB bus registered, assigned bus number 2 ohci-platform 8000c400.usb: irq 20, io mem 0x8000c400 hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected xhci-hcd 8000d000.usb: xHCI Host Controller xhci-hcd 8000d000.usb: new USB bus registered, assigned bus number 3 xhci-hcd 8000d000.usb: hcc params 0x0250f17c hci version 0x100 quirks 0x0000000000010010 xhci-hcd 8000d000.usb: irq 21, io mem 0x8000d000 hub 3-0:1.0: USB hub found hub 3-0:1.0: config failed, hub doesn't have any ports! (err -19) xhci-hcd 8000d000.usb: xHCI Host Controller xhci-hcd 8000d000.usb: new USB bus registered, assigned bus number 4 xhci-hcd 8000d000.usb: Host supports USB 3.0 SuperSpeed usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. hub 4-0:1.0: USB hub found hub 4-0:1.0: 2 ports detected Freeing unused kernel memory: 2432K Run /init as init process tmpfs: Unknown parameter 'mode' mount: mounting tmpfs: Unknown parameter 'mode' tmpfs on /dev/shtmpfs: Unknown parameter 'mode' m failed: Invalid argument mount: mounting tmpfs on /tmp failed: Invalid argument mount: mounting tmpfs on /run failed: Invalid argument Starting syslogd: OK Starting klogd: OK Runnirandom: dd: uninitialized urandom read (512 bytes read) ng sysctl: OK Saving random seed: OK Starting network: ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented ip: socket: Function not implemented FAIL Welcome to Buildroot buildroot login: _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel