--- Comment #1 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300115
--> https://bugzilla.kernel.org/attachment.cgi?id=300115&action=edit
kernel .config (5.15.10, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #2 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Probably hard to track.
Any chance to bisect the issue ?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #3 from Erhard F. (erhard_f@mailbox.org) ---
Bisecting will take some time. I'll report back as soon as I have any findings.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
[f698de00] [c0185d74] release_z3fold_page_locked+0x0/0x44 (unreliable)
[f698de20] [c01856c8] do_compact_page+0x334/0x508
[f698de80] [c004f354] process_one_work+0x1d4/0x288
[f698dec0] [c004f814] worker_thread+0x1b8/0x260
[f698df00] [c0055514] kthread+0x118/0x11c
[f698df30] [c0016268] ret_from_kernel_thread+0x5c/0x64
Rebooting in 40 seconds..
Which is interesting because on bug #213837 my not yet finished bisect is also
giving hints z3fold may be the problem...
I'll check out next whether the issue is reproduceable on 5.15.x when I use
zbud or zmalloc for zswap instead of z3fold.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #5 from Erhard F. (erhard_f@mailbox.org) ---
Ok, with zswap lzo/zbud I also get this memory corruption on 5.15.13. So most
probably it's not lzo/z3pool but something else. I'll start a bisect then...
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #6 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300318
--> https://bugzilla.kernel.org/attachment.cgi?id=300318&action=edit
bisect.log
Ok, finally got it. Interesting find:
# git bisect bad
db972a3787d12b1ce9ba7a31ec376d8a79e04c47 is the first bad commit
commit db972a3787d12b1ce9ba7a31ec376d8a79e04c47
Author: Christophe Leroy [off-list ref]
Date: Tue Dec 8 05:24:19 2020 +0000
powerpc/powermac: Fix low_sleep_handler with CONFIG_VMAP_STACK
low_sleep_handler() can't restore the context from standard
stack because the stack can hardly be accessed with MMU OFF.
Store everything in a global storage area instead of storing
a pointer to the stack in that global storage area.
To avoid a complete churn of the function, still use r1 as
the pointer to the storage area during restore.
Fixes: cd08f109e262 ("powerpc/32s: Enable CONFIG_VMAP_STACK")
Reported-by: Giuseppe Sacco [off-list ref]
Signed-off-by: Christophe Leroy [off-list ref]
Tested-by: Giuseppe Sacco [off-list ref]
Signed-off-by: Michael Ellerman [off-list ref]
Link:
https://lore.kernel.org/r/e3e0d8042a3ba75cb4a9546c19c408b5b5b28994.1607404931.git.christophe.leroy@csgroup.eu
arch/powerpc/platforms/Kconfig.cputype | 2 +-
arch/powerpc/platforms/powermac/sleep.S | 132 ++++++++++++++------------------
2 files changed, 60 insertions(+), 74 deletions(-)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #7 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Interesting ... Though confusing.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #8 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Looking closer, in fact that might be a false positive.
The huge difference with that bad commit is that:
- Before the commit, the kernel is built _without_ CONFIG_VMAP_STACK
- After the commit, the kernel is built _with_ CONFIG_VMAP_STACK
Would you be able to perform following tests:
- Disable VMAP_STACK and see if the problem still occurs.
- Disable ADB_PMU and see it the problem still occurs.
With the version which preceeds the bad commit, can you disable ADB_PMU and
enable VMAP_STACK and see what happens ?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #9 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300354
--> https://bugzilla.kernel.org/attachment.cgi?id=300354&action=edit
dmesg (5.10-rc2 with ADB_PMU disabled, PowerMac G4 DP)
Took a little time but I double checked the results (one time using distcc '-j8
-l2', one time native '-j3') to be sure:
ADB_PMU disabled, VMAP_STACK disabled ... "neverending build"
ADB_PMU enabled, VMAP_STACK disabled ... works ok
ADB_PMU disabled, VMAP_STACK enabled ... "neverending build"
ADB_PMU enabled, VMAP_STACK enabled ... memory corruption
Version used was git db972a3787d12b1ce9ba7a31ec376d8a79e04c47, which is the one
before a last 'git bisect bad' ends the git bisect.
The "neverending builds" happen when I run this kernel with ADB_PMU disabled.
The G4 runs for several hours building (?) without reaching the glibc test
stage. With ADB_PMU enabled I get a pass or memory corruption much earlier.
Also without ADB_PMU I get a kernel panic when rebooting or shutting down the
G4. Also the G4 does not reboot/poweroff in this case, I need to switch it off
manually.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #10 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Thanks for the tests.
I'm not surprised that the system doesn't poweroff or reboot without ADB_PMU
because the PMU manages power.
The "neverending build" is maybe because the PMU also manages RTC clock and
without it you get inconsistent time ?
Anyway, it looks like there is indeed something linked to VMAP_STACK.
I'm wondering whether you could be running out of vmalloc space. I initially
thought you were using KASAN, but it seems not according to your .config.
Could you try reducing CONFIG_LOWMEM_SIZE to 0x28000000 for instance and see if
the memory corruption still happens ?
To do this you'll need CONFIG_ADVANCED_OPTIONS and CONFIG_LOWMEM_SIZE_BOOL.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #11 from Erhard F. (erhard_f@mailbox.org) ---
(In reply to Christophe Leroy from comment #10)
I'm wondering whether you could be running out of vmalloc space. I initially
thought you were using KASAN, but it seems not according to your .config.
Correct, I was not using KASAN. I use it only for testing -rc kernels or when I
am particularly wary. This memory corruption I noticed during regular usage.
Seems running the kernel with slub_debug=FZP page_poison=1 is a good thing. ;)
Could you try reducing CONFIG_LOWMEM_SIZE to 0x28000000 for instance and see
if the memory corruption still happens ?
Thanks, that did the trick! With CONFIG_LOWMEM_SIZE=0x28000000 the memory
corruption is gone on VMAP_STACK enabled kernels. Tested it additionally on
current 5.16.4 where this works too.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
page:ef4c4ec4 refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x31069
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page detail
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #13 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300775
--> https://bugzilla.kernel.org/attachment.cgi?id=300775&action=edit
kernel .config (5.18-rc3, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #14 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Do you mean it still happens with the default values, or it also happens with
the reduced CONFIG_LOWMEM_SIZE ?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #15 from Erhard F. (erhard_f@mailbox.org) ---
It definitively still happens with the default values. Can test with the
reduced CONFIG_LOWMEM_SIZE next week and report back.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300774|0 |1
is obsolete| |
--- Comment #16 from Erhard F. (erhard_f@mailbox.org) ---
Do you mean it still happens with the default values, or it also happens
with the reduced CONFIG_LOWMEM_SIZE ?
Turns out the memory corruption also happens with the reduced
CONFIG_LOWMEM_SIZE=0x28000000.
Tested again on v5.18-rc6, both with CONFIG_LOWMEM_SIZE=0x28000000 and without.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300775|0 |1
is obsolete| |
--- Comment #17 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300930
--> https://bugzilla.kernel.org/attachment.cgi?id=300930&action=edit
kernel .config (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #19 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Yes KASAN can bring some additional inputs.
Maybe start with CONFIG_KFENCE, it is lighter than KASAN.
For the above problem, maybe CONFIG_DEBUG_STACKOVERFLOW can help.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #20 from Erhard F. (erhard_f@mailbox.org) ---
DEBUG_STACKOVERFLOW and KFENCE have been enabled already in the builds I did
here (see kernel attached kernel .config here).
However if I enable (inline) KASAN the kernel won't boot at all. I get dropped
out in OpenFirmware console with:
[...]
Finalizing device tree... using OF tree (promptr=ff847240)
Invalid memory access at %SRR0: 40000000 %SRR1: 00000000
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Michael Ellerman (michael@ellerman.id.au) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
CC| |michael@ellerman.id.au
--- Comment #21 from Michael Ellerman (michael@ellerman.id.au) ---
Increasing the stack size (CONFIG_THREAD_SHIFT) might avoid the stack overflows
and allow you to debug the original issue in isolation.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
page:e739d6ec refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x290a3
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
[...]
With THREAD_SHIFT=14 the stack issue does not show up.
A kernel with inline KASAN and same setup otherwise won't boot showing me this
at the OpenFirmware prompt:
[...]
Finalizing device tree... using OF tree (promptr=ff847240)
Invalid memory access at %SRR0: 40000000 %SRR1: 00000000
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #23 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 300978
--> https://bugzilla.kernel.org/attachment.cgi?id=300978&action=edit
kernel .config (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, outline KASAN,
PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #24 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Seems like with Inline KASAN your kernel is far too big compared to what we
support at the time being:
c2468000 T __end_rodata
c2800000 T __init_begin
c2800000 T _sinittext
c2801644 T prom_init
The init text section is behind the 32Mbytes boundary, it means that prom_init
and other functions are not called anymore directly but via a trampoline.
c000000c <__start>:
c000000c: 2c 05 00 00 cmpwi r5,0
c0000010: 41 82 00 1c beq c000002c <__start+0x20>
c0000014: 42 9f 00 05 bcl 20,4*cr7+so,c0000018 <__start+0xc>
c0000018: 7d 08 02 a6 mflr r8
c000001c: 3d 08 00 00 addis r8,r8,0
c0000020: 39 08 ff e8 addi r8,r8,-24
c0000024: 48 00 38 e5 bl c0003908 <setup_disp_bat+0x30>
...
c0003908: 3d 80 c2 80 lis r12,-15744
c000390c: 39 8c 16 44 addi r12,r12,5700
c0003910: 7d 89 03 a6 mtctr r12
c0003914: 4e 80 04 20 bctr
And it cannot work because at that time the kernel is not yet relocated to its
final location.
There was the same problem with PPC64 and it was fix by 24d33ac5b8ff
("powerpc/64s: Make prom_init require RELOCATABLE").
Don't know if a similar approach could work.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #25 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
The Kernel stack overflow looks odd.
Value of R1 is wrong and LR is NULL. Don't know how we ended up here, but
probably not by a real stack overflow.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #26 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Note that THREAD_SHIFT is set to 14 when using KASAN:
config THREAD_SHIFT
int "Thread shift" if EXPERT
range 13 15
default "15" if PPC_256K_PAGES
default "14" if PPC64
default "14" if KASAN
default "13"
help
Used to define the stack size. The default is almost always what you
want. Only change this if you know what you are doing.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #27 from Erhard F. (erhard_f@mailbox.org) ---
I opened a new bug for the stack issue which contains a bit more data.
Hopefully the output is of some help (see bug #216041).
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300113|0 |1
is obsolete| |
Attachment #300929|0 |1
is obsolete| |
--- Comment #28 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 301302
--> https://bugzilla.kernel.org/attachment.cgi?id=301302&action=edit
dmesg (5.19-rc4, PowerMac G4 DP)
Re-tried on v5.19-rc4 (without fadditional patches) + KFENCE.
My findings so far:
1. Memory corruption still persists.
2. Even without KASAN I need THREAD_SHIFT=14 or else I get the stack overflow
from bug #216041.
3. Memory corruption also happens with CONFIG_LOWMEM_SIZE=0x28000000.
4. But the "neverending build" commit mentioned in comment #9 is gone (be it
with default .config or CONFIG_LOWMEM_SIZE=0x28000000).
[...]
pagealloc: memory corruption
fffdfff0: 00 00 00 00 ....
CPU: 0 PID: 29136 Comm: localedef Not tainted 5.19.0-rc4-PMacG4 #3
Call Trace:
[f39b3c20] [c05eb9c0] dump_stack_lvl+0x60/0x90 (unreliable)
[f39b3c40] [c0232fb0] __kernel_unpoison_pages+0x1a8/0x1ec
[f39b3c90] [c02170dc] get_page_from_freelist+0xc20/0xe70
[f39b3d50] [c0217bdc] __alloc_pages+0x18c/0xe80
[f39b3e10] [c01f46b4] wp_page_copy+0x214/0xa1c
[f39b3e80] [c01fa0b8] handle_mm_fault+0x720/0xd64
[f39b3f00] [c00215dc] do_page_fault+0x1d4/0x830
[f39b3f30] [c000433c] DataAccess_virt+0x124/0x17c
page:ef4bd80c refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310ab
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300115|0 |1
is obsolete| |
Attachment #300930|0 |1
is obsolete| |
--- Comment #29 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 301303
--> https://bugzilla.kernel.org/attachment.cgi?id=301303&action=edit
kernel .config (5.19-rc4, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #30 from Michael Ellerman (michael@ellerman.id.au) ---
It's a bit of a stab in the dark, but can you try turning preempt off?
ie. CONFIG_PREEMPT_NONE=y
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
page:ef4bd80c refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310ab
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
Interesting thing is the memory corruption always seems to happen in the last
stage of installing, after building is done at copying over the binaries from
build directory to target directory:
[...]
if test -r
/var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h &&
cmp -s
/var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h
/var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h; \
then echo 'stubs.h unchanged'; \
else /usr/lib/portage/python3.10/ebuild-helpers/xattr/install -c -m 644
/var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h
/var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h; fi
rm -f
/var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h
make[1]: Leaving directory
'/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34'
quoted
quoted
Completed installing sys-libs/glibc-2.34-r13 into
/var/tmp/portage/sys-libs/glibc-2.34-r13/image
* Final size of build directory: 635640 KiB (620.7 MiB)
* Final size of installed tree: 109892 KiB (107.3 MiB)
making executable: /usr/lib/libc.so
compressme : 44.96% ( 3.80 KiB => 1.71 KiB, compressme.zst)
[...]
/var/tmp/portage/sys-libs/glibc-2.34-r13/image/usr/share/doc/glibc-2.34-r13/NEWS
: 33.98% ( 315 KiB => 107 KiB,
/var/tmp/portage/sys-libs/glibc-2.34-r13/image/usr/share/doc/glibc-2.34-r13/NEWS.zst)
strip: powerpc-unknown-linux-gnu-strip --strip-unneeded -N
__gentoo_check_ldflags__ -R .comment -R .GCC.command.line -R
.note.gnu.gold-version
/usr/lib/crt1.o
/usr/lib/Mcrt1.o
/usr/lib/gcrt1.o
/usr/lib/Scrt1.o
[...]
/lib/ld.so.1
/usr/lib/audit/sotruss-lib.so
/usr/bin/pldd
installsources: rsyncing source files
rsync: [sender] link_stat
"/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/iconv/charmap-kw.gperf"
failed: No such file or directory (2)
rsync: [sender] link_stat
"/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/locale/charmap-kw.gperf"
failed: No such file or directory (2)
rsync: [sender] link_stat
"/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/locale/locfile-kw.gperf"
failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1326) [sender=3.2.4]
quoted
quoted
Installing (1 of 1) sys-libs/glibc-2.34-r13::gentoo
* Defaulting /etc/host.conf:multi to on
* Last-minute run tests with ./ld.so.1 in /lib ...
[...]
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #32 from Erhard F. (erhard_f@mailbox.org) ---
(In reply to Michael Ellerman from comment #30)
It's a bit of a stab in the dark, but can you try turning preempt off?
ie. CONFIG_PREEMPT_NONE=y
Looks like your intuition was not bad at all. ;) CONFIG_PREEMPT_NONE=y had no
effect but when I disable SMP at all '# CONFIG_SMP is not set' I get no memory
corruption and also no stack overflow issues.
Also no special treatment with Advanced Options or setting THREAD_SHIFT
manually was necessary. The G4 just does fine, albeit with 1 of it's 2 CPUs
only with disabled SMP.
For testing I did 6 of this glibc testsuite builds in a row without getting
issues. With SMP enabled I get memory corruption or stack overflow at the 1st
build allmost all of the time.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #33 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 301337
--> https://bugzilla.kernel.org/attachment.cgi?id=301337&action=edit
dmesg (5.19-rc5, outline KASAN, PowerMac G4 DP)
Re-tested on 5.19-rc5 +
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/2ee707512b8b212b079b877f4ceb525a1606a3fb.1656655567.git.christophe.leroy@csgroup.eu/
I can run the kernel with outline KASAN, default THREAD_SHIFT and without
advanced options necessary. Also I don't get the stack issue (bug #216041) any
longer.
However as long as CONFIG_SMP=y (CONFIG_NR_CPUS=2) is set I still get the
memory corruption:
[...]
pagealloc: memory corruption
f5fcfff0: 00 00 00 00 ....
CPU: 1 PID: 27635 Comm: estrip Not tainted 5.19.0-rc5-PMacG4+ #1
Call Trace:
[f380b9b0] [c0829ebc] dump_stack_lvl+0x60/0x90 (unreliable)
[f380b9d0] [c0307528] __kernel_unpoison_pages+0x1d8/0x220
[f380ba20] [c02dd3bc] post_alloc_hook+0x108/0x144
[f380ba50] [c02e0a70] get_page_from_freelist+0x9e0/0x1278
[f380bb90] [c02e1e04] __alloc_pages+0x250/0x1078
[f380bcf0] [c02af098] wp_page_copy+0x128/0xdb8
[f380bde0] [c02b6fdc] handle_mm_fault+0x954/0x1138
[f380bed0] [c0029938] ___do_page_fault+0x250/0x84c
[f380bf10] [c002a168] do_page_fault+0x28/0x5c
[f380bf30] [c000433c] DataAccess_virt+0x124/0x17c
page:ef4bd6ec refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310a3
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #301302|0 |1
is obsolete| |
--- Comment #34 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 301639
--> https://bugzilla.kernel.org/attachment.cgi?id=301639&action=edit
dmesg (6.0-rc2, outline KASAN, PowerMac G4 DP)
Getting a more interesting backtrace with v6.0.0-rc2 + outline KASAN:
[...]
BUG: KASAN: slab-out-of-bounds in handle_mm_fault+0x27c/0x10f4
Read of size 4 at addr c32edd48 by task cc1plus/1230
CPU: 1 PID: 1230 Comm: cc1plus Tainted: G T 6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2bd40] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2bd60] [c032b8d8] print_report+0x30c/0x688
[f4d2bdb0] [c032befc] kasan_report+0xe4/0x214
[f4d2be00] [c02ce4d8] handle_mm_fault+0x27c/0x10f4
[f4d2bed0] [c002cc98] ___do_page_fault+0x25c/0x8d0
[f4d2bf10] [c002d560] do_page_fault+0x28/0x6c
[f4d2bf30] [c000433c] DataAccess_virt+0x124/0x17c
Allocated by task 1:
__kasan_slab_alloc+0xd0/0x134
kmem_cache_alloc+0x21c/0x66c
__kernfs_new_node+0xe8/0x354
kernfs_new_node+0x84/0xfc
__kernfs_create_file+0x50/0x204
sysfs_add_file_mode_ns+0xf4/0x1f0
internal_create_group+0x1f0/0x620
btrfs_init_sysfs+0x264/0x350
init_btrfs_fs+0x24/0x280
do_one_initcall+0xc0/0x34c
kernel_init_freeable+0x2c0/0x400
kernel_init+0x28/0x178
ret_from_kernel_thread+0x5c/0x64
The buggy address belongs to the object at c32edd50
which belongs to the cache kernfs_node_cache of size 88
The buggy address is located 8 bytes to the left of
88-byte region [c32edd50, c32edda8)
The buggy address belongs to the physical page:
page:eee4a954 refcount:1 mapcount:0 mapping:00000000 index:0x0 pfn:0x32ed
flags: 0x200(slab|zone=0)
raw: 00000200 00000100 00000122 c1852520 00000000 001e003c ffffffff 00000001
raw: 00000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
c32edc00: 00 00 fc fc fc fc fc fc 00 00 00 00 00 00 00 00
c32edc80: 00 00 00 fc fc fc fc fc fc 00 00 00 00 00 00 00
c32edd00: 00 00 00 00 fc fc fc fc fc fc 00 00 00 00 00 00
^
c32edd80: 00 00 00 00 00 fc fc fc fc fc fc 00 00 00 00 00
c32ede00: 00 00 00 00 00 00 fc fc fc fc fc fc 00 00 00 00
==================================================================
Disabling lock debugging due to kernel taint
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
[...]
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
_swap_info_get: Bad swap file entry 64cccccc
BUG: Bad page map in process cc1plus pte:cccccccc pmd:032ed000
addr:9a352000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a352
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G B W T 6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
Instruction dump:
7ecfb378 82410014 82c10018 4bffff04 3d40c170 578901be 83aa5580 1d290024
7fbd4a14 387d0004 7fbceb78 48063245 <813d0004> 712a0001 40820304 7f83e378
---[ end trace 0000000000000000 ]---
Fixing recursive fault but reboot is needed!
I deleted about 120.000 lines of "get_swap_device: Bad swap file entry
64cccccc" in the kernel dmesg to make it more compact. swap partition is 8192
MiB large at /dev/sdb6.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #35 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 301640
--> https://bugzilla.kernel.org/attachment.cgi?id=301640&action=edit
kernel .config (6.0-rc2, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #36 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
Would be nice to give it a new try with KCSAN enabled.
To get KCSAN on powerpc/32, apply following series:
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=354731
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #37 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 304308
--> https://bugzilla.kernel.org/attachment.cgi?id=304308&action=edit
dmesg (6.3.3, KCSAN, PowerMac G4 DP)
Thanks for taking another look into this Christophe!
Applied the patches on top of 6.3.3 and these are my findings so far:
1. KCSAN works fine on my G4 and passes self tests.
2. It does not generate any additional output when I hit the "pagealloc: memory
corruption".
3. When setting CONFIG_KCSAN_WEAK_MEMORY=y my G4 won't finish booting. Early
boot works, the screen shows some dmesg but booting gets stuck there never
reaching console. I also don't get any netconsole output with
CONFIG_KCSAN_WEAK_MEMORY=y.
4. As soon as I set CONFIG_KCSAN_EARLY_ENABLE=y dmesg shows plenty of data
races!
netconsole output and kernel .config attached.
To provoke the memory corruption 'stress' is a good tool. stress -m2 --vm-bytes
915M provokes the corruption easily and --vm-bytes 915M is small enough to not
provoke the OOM killer on my G4 DP with its' 2 CPUs and 2 GiB RAM.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #38 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 304309
--> https://bugzilla.kernel.org/attachment.cgi?id=304309&action=edit
kernel .config (6.3.3, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
page:e314e657 refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x31065
flags: 0x80000000(zone=2)
page_type: 0xffffffff()
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300354|0 |1
is obsolete| |
Attachment #300977|0 |1
is obsolete| |
Attachment #301337|0 |1
is obsolete| |
Attachment #301639|0 |1
is obsolete| |
--- Comment #40 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 305297
--> https://bugzilla.kernel.org/attachment.cgi?id=305297&action=edit
dmesg (5.5-rc5, PowerMac G4 DP)
Re-visiting this bug as it's reproducible on v6.6-rc7.
This time I tried the other way round. CONFIG_VMAP_STACK was added for ppc with
commit cd08f109e26231b279bcc0388428afcac6408ec6 (at about kernel v5.5-rc5
time). So I did a git checkout cd08f109e26231b279bcc0388428afcac6408ec6 and
started from there with a further reduced kernel .config.
I added two additional patches to get the G4 to boot with VMAP_STACK enabled:
4119622 "powerpc/32s: Fix kasan_early_hash_table() for CONFIG_VMAP_STACK" and
232ca1e "powerpc/32s: Fix DSI and ISI exceptions for CONFIG_VMAP_STACK".
Then I burdened the memory subsystem with "stress -c 2 --vm 2 --vm-bytes 896M"
as before and hit the issue in less than 20 sec. Not hitting the issue means my
G4 runs "stress -c 2 --vm 2 --vm-bytes 896M" for about half an hour without
side effects.
So it looks like the issue was here from the start when CONFIG_VMAP_STACK was
added for ppc. (see dmesg)
I don't hit the issue when:
1. nr_cpus=1 is set + VMAP_STACK enabled
2. VMAP_STACK disabled
Setting LOWMEM_SIZE to 0x28000000 does not seem to have an effect on it.
This bug really plays hard to get... T'll do further KCSAN checks in recent
kernels and open separate issues if KCSAN digs up something useful.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
--- Comment #41 from Christophe Leroy (christophe.leroy@csgroup.eu) ---
I'm out of office until 06 Nov.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=215389
Erhard F. (erhard_f@mailbox.org) changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #300978|0 |1
is obsolete| |
Attachment #301303|0 |1
is obsolete| |
Attachment #301640|0 |1
is obsolete| |
--- Comment #42 from Erhard F. (erhard_f@mailbox.org) ---
Created attachment 305299
--> https://bugzilla.kernel.org/attachment.cgi?id=305299&action=edit
kernel .config (5.5-rc5, PowerMac G4 DP)
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.