Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt
From: Mohammad Amin Nili <hidden>
Date: 2025-11-22 11:33:03
Hello Oliver,
That's very odd. rcu_reclaim_tiny() is probably being folded into rcu_process_callbacks() by the compiler and the crash occurs when branching to the callback function from the rcu_head (https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95 <https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95>). That said, the "callback" address it branched to (0xc00c000000000000) is actually the base of the vmemmap (i.e. the struct page array) so I doubt that's actually the callback address stored in the rcu_head. You can use xmon to dump the registers and examine memory to confirm this. It's hard to say why this is happening, but it's pretty likely to either be the compiler optimizing away code you'd prefer to keep or a bug in the core itself. I'd compare the disasm of rcu_process_callbacks() with and without your wait loop added and see how the emitted code changes. If adding the loop changes nothing then it might be a logic bug in microwatt itself or some other timing induced problem.
I just replaced the wait-loop with the following (and it also prevents the kernel panic):
volatile uint32_t dumb = 0xFEDC0000; dumb++;
- Here is the disasm of the `rcu_process_callbacks` with the injected code: c00000000008c5e8 <rcu_process_callbacks>: c00000000008c5e8: 32 00 4c 3c addis r2,r12,50 c00000000008c5ec: 18 ca 42 38 addi r2,r2,-13800 c00000000008c5f0: a6 02 08 7c mflr r0 c00000000008c5f4: a1 c3 23 48 bl c0000000002c8994 <_savegpr0_27> c00000000008c5f8: a1 ff 21 f8 stdu r1,-96(r1) c00000000008c5fc: fa 00 6d 88 lbz r3,250(r13) c00000000008c600: 01 00 69 60 ori r9,r3,1 c00000000008c604: fa 00 2d 99 stb r9,250(r13) c00000000008c608: 2b 01 42 3d addis r10,r2,299 c00000000008c60c: 2b 01 22 3d addis r9,r2,299 c00000000008c610: d0 3c 4a e9 ld r10,15568(r10) c00000000008c614: c8 3c 29 39 addi r9,r9,15560 c00000000008c618: 00 48 2a 7c cmpd r10,r9 c00000000008c61c: 14 00 82 40 bne c00000000008c630 <rcu_process_callbacks+0x48> c00000000008c620: 6d d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8> c00000000008c624: 00 00 00 60 nop c00000000008c628: 60 00 21 38 addi r1,r1,96 c00000000008c62c: b8 c3 23 48 b c0000000002c89e4 <_restgpr0_27> c00000000008c630: 00 00 0a e9 ld r8,0(r10) c00000000008c634: 2b 01 e2 3c addis r7,r2,299 c00000000008c638: c8 3c e7 eb ld r31,15560(r7) c00000000008c63c: c8 3c 07 f9 std r8,15560(r7) c00000000008c640: 00 00 00 39 li r8,0 c00000000008c644: 00 00 0a f9 std r8,0(r10) c00000000008c648: 2b 01 42 3d addis r10,r2,299 c00000000008c64c: d8 3c 0a e9 ld r8,15576(r10) c00000000008c650: 2b 01 42 3d addis r10,r2,299 c00000000008c654: d0 3c 4a e9 ld r10,15568(r10) c00000000008c658: 00 50 28 7c cmpd r8,r10 c00000000008c65c: 08 00 82 40 bne c00000000008c664 <rcu_process_callbacks+0x7c> c00000000008c660: 10 00 29 f9 std r9,16(r9) c00000000008c664: 08 00 29 f9 std r9,8(r9) c00000000008c668: dc fe 80 3f lis r28,-292 <<<<====== "THIS IS INJECTED CODE" c00000000008c66c: 21 d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8> c00000000008c670: 00 00 00 60 nop c00000000008c674: 00 00 60 3b li r27,0 c00000000008c678: 00 00 3f 2c cmpdi r31,0 c00000000008c67c: ac ff 82 41 beq c00000000008c628 <rcu_process_callbacks+0x40> c00000000008c680: 00 00 df eb ld r30,0(r31) c00000000008c684: 00 00 3e 2c cmpdi r30,0 c00000000008c688: 08 00 82 41 beq c00000000008c690 <rcu_process_callbacks+0xa8> c00000000008c68c: 2c f2 00 7c dcbtct 0,r30 c00000000008c690: 2c 00 81 93 stw r28,44(r1) c00000000008c694: 08 00 bf eb ld r29,8(r31) c00000000008c698: 00 00 3d 2c cmpdi r29,0 c00000000008c69c: 2c 00 21 81 lwz r9,44(r1) c00000000008c6a0: 01 00 29 39 addi r9,r9,1 c00000000008c6a4: 2c 00 21 91 stw r9,44(r1) c00000000008c6a8: 10 00 82 40 bne c00000000008c6b8 <rcu_process_callbacks+0xd0> c00000000008c6ac: 78 fb e3 7f mr r3,r31 c00000000008c6b0: 15 f8 03 48 bl c0000000000cbec4 <kmem_dump_obj+0x8> c00000000008c6b4: 00 00 00 60 nop c00000000008c6b8: 08 00 7f fb std r27,8(r31) c00000000008c6bc: 78 fb e3 7f mr r3,r31 c00000000008c6c0: a6 03 a9 7f mtctr r29 c00000000008c6c4: 78 eb ac 7f mr r12,r29 c00000000008c6c8: 18 00 41 f8 std r2,24(r1) c00000000008c6cc: 78 f3 df 7f mr r31,r30 c00000000008c6d0: 21 04 80 4e bctrl c00000000008c6d4: 18 00 41 e8 ld r2,24(r1) c00000000008c6d8: a0 ff ff 4b b c00000000008c678 <rcu_process_callbacks+0x90> - And here is the disasm without the injected code: c00000000008c5e8 <rcu_process_callbacks>: c00000000008c5e8: 32 00 4c 3c addis r2,r12,50 c00000000008c5ec: 18 ca 42 38 addi r2,r2,-13800 c00000000008c5f0: a6 02 08 7c mflr r0 c00000000008c5f4: 95 c3 23 48 bl c0000000002c8988 <_savegpr0_28> c00000000008c5f8: c1 ff 21 f8 stdu r1,-64(r1) c00000000008c5fc: fa 00 6d 88 lbz r3,250(r13) c00000000008c600: 01 00 69 60 ori r9,r3,1 c00000000008c604: fa 00 2d 99 stb r9,250(r13) c00000000008c608: 2b 01 42 3d addis r10,r2,299 c00000000008c60c: 2b 01 22 3d addis r9,r2,299 c00000000008c610: d0 3c 4a e9 ld r10,15568(r10) c00000000008c614: c8 3c 29 39 addi r9,r9,15560 c00000000008c618: 00 48 2a 7c cmpd r10,r9 c00000000008c61c: 14 00 82 40 bne c00000000008c630 <rcu_process_callbacks+0x48> c00000000008c620: 6d d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8> c00000000008c624: 00 00 00 60 nop c00000000008c628: 40 00 21 38 addi r1,r1,64 c00000000008c62c: ac c3 23 48 b c0000000002c89d8 <_restgpr0_28> c00000000008c630: 00 00 0a e9 ld r8,0(r10) c00000000008c634: 2b 01 e2 3c addis r7,r2,299 c00000000008c638: c8 3c e7 eb ld r31,15560(r7) c00000000008c63c: c8 3c 07 f9 std r8,15560(r7) c00000000008c640: 00 00 00 39 li r8,0 c00000000008c644: 00 00 0a f9 std r8,0(r10) c00000000008c648: 2b 01 42 3d addis r10,r2,299 c00000000008c64c: d8 3c 0a e9 ld r8,15576(r10) c00000000008c650: 2b 01 42 3d addis r10,r2,299 c00000000008c654: d0 3c 4a e9 ld r10,15568(r10) c00000000008c658: 00 50 28 7c cmpd r8,r10 c00000000008c65c: 08 00 82 40 bne c00000000008c664 <rcu_process_callbacks+0x7c> c00000000008c660: 10 00 29 f9 std r9,16(r9) c00000000008c664: 08 00 29 f9 std r9,8(r9) c00000000008c668: 00 00 80 3b li r28,0 c00000000008c66c: 21 d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8> c00000000008c670: 00 00 00 60 nop c00000000008c674: 00 00 3f 2c cmpdi r31,0 c00000000008c678: b0 ff 82 41 beq c00000000008c628 <rcu_process_callbacks+0x40> c00000000008c67c: 00 00 df eb ld r30,0(r31) c00000000008c680: 00 00 3e 2c cmpdi r30,0 c00000000008c684: 08 00 82 41 beq c00000000008c68c <rcu_process_callbacks+0xa4> c00000000008c688: 2c f2 00 7c dcbtct 0,r30 c00000000008c68c: 08 00 bf eb ld r29,8(r31) c00000000008c690: 00 00 3d 2c cmpdi r29,0 c00000000008c694: 10 00 82 40 bne c00000000008c6a4 <rcu_process_callbacks+0xbc> c00000000008c698: 78 fb e3 7f mr r3,r31 c00000000008c69c: 19 f8 03 48 bl c0000000000cbeb4 <kmem_dump_obj+0x8> c00000000008c6a0: 00 00 00 60 nop c00000000008c6a4: 08 00 9f fb std r28,8(r31) c00000000008c6a8: 78 fb e3 7f mr r3,r31 c00000000008c6ac: a6 03 a9 7f mtctr r29 c00000000008c6b0: 78 eb ac 7f mr r12,r29 c00000000008c6b4: 18 00 41 f8 std r2,24(r1) c00000000008c6b8: 78 f3 df 7f mr r31,r30 c00000000008c6bc: 21 04 80 4e bctrl c00000000008c6c0: 18 00 41 e8 ld r2,24(r1) c00000000008c6c4: b0 ff ff 4b b c00000000008c674 <rcu_process_callbacks+0x8c> Sounds like some sorta compiler optimizations, right? Bests, Manili