From: Paul Menzel <hidden> Date: 2022-02-07 16:45:21
Dear Linux folks,
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
GNU/Linux 3.10.0, stripped
segfaults in QEMU. From one of the log files
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip
10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84
4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4
38000000 <f821ff81> 7c0803a6 f8010000 e9028010
Executing the init, which just seems to be an endless loop, from
userspace work:
$ strace ./tools/testing/selftests/rcutorture/initrd/init
execve("./tools/testing/selftests/rcutorture/initrd/init",
["./tools/testing/selftests/rcutor"...], 0x7ffffdb9e860 /* 31 vars */) = 0
brk(NULL) = 0x1001d940000
brk(0x1001d940b98) = 0x1001d940b98
set_tid_address(0x1001d9400d0) = 2890832
set_robust_list(0x1001d9400e0, 24) = 0
uname({sysname="Linux",
nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"...,
4096) = 61
getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
brk(0x1001d970b98) = 0x1001d970b98
brk(0x1001d980000) = 0x1001d980000
mprotect(0x100e0000, 65536, PROT_READ) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
^C{tv_sec=0, tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted
by signal)
strace: Process 2890832 detached
Any ideas, what `mkinitrd.sh` [2] should do differently?
cat>init.c<<'___EOF___'#ifndef NOLIBC#include<unistd.h>#include<sys/time.h>#endifvolatileunsignedlongdelaycount;intmain(intargc,intargv[]){inti;structtimevaltv;structtimevaltvb;for(;;){sleep(1);/* Need some userspace time. */if(gettimeofday(&tvb,NULL))continue;do{for(i=0;i<1000*100;i++)delaycount=i*i;if(gettimeofday(&tv,NULL))break;tv.tv_sec-=tvb.tv_sec;if(tv.tv_sec>1)break;tv.tv_usec+=tv.tv_sec*1000*1000;tv.tv_usec-=tvb.tv_usec;}while(tv.tv_usec<1000);}return0;}___EOF___# build using nolibc on supported archs (smaller executable) and fall# back to regular glibc on other ones.ifecho-e"#if __x86_64__||__i386__||__i486__||__i586__||__i686__"\
"||__ARM_EABI__||__aarch64__\nyes\n#endif"\
|${CROSS_COMPILE}gcc-E-nostdlib-xc-\
|grep-q'^yes';then# architecture supported by nolibc${CROSS_COMPILE}gcc-fno-asynchronous-unwind-tables-fno-ident\
-nostdlib-include../../../../include/nolibc/nolibc.h\
-s-static-Os-oinitinit.c-lgccelse${CROSS_COMPILE}gcc-s-static-Os-oinitinit.cfi
From: "Paul E. McKenney" <paulmck@kernel.org> Date: 2022-02-07 17:52:26
On Mon, Feb 07, 2022 at 05:44:47PM +0100, Paul Menzel wrote:
Dear Linux folks,
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
GNU/Linux 3.10.0, stripped
segfaults in QEMU. From one of the log files
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18
lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84
4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4
38000000 <f821ff81> 7c0803a6 f8010000 e9028010
Executing the init, which just seems to be an endless loop, from userspace
work:
$ strace ./tools/testing/selftests/rcutorture/initrd/init
execve("./tools/testing/selftests/rcutorture/initrd/init",
["./tools/testing/selftests/rcutor"...], 0x7ffffdb9e860 /* 31 vars */) = 0
brk(NULL) = 0x1001d940000
brk(0x1001d940b98) = 0x1001d940b98
set_tid_address(0x1001d9400d0) = 2890832
set_robust_list(0x1001d9400e0, 24) = 0
uname({sysname="Linux",
nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"..., 4096)
= 61
getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
brk(0x1001d970b98) = 0x1001d970b98
brk(0x1001d980000) = 0x1001d980000
mprotect(0x100e0000, 65536, PROT_READ) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^C{tv_sec=0,
tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
strace: Process 2890832 detached
Huh. In PowerPC, is there some difference between system calls
executed in initrd and those same system calls executed in userspace?
And just to make sure, the above strace was from exactly the same
binary "init" file that is included in initrd, correct?
Adding Willy Tarreau for his thoughts.
Thanx, Paul
Any ideas, what `mkinitrd.sh` [2] should do differently?
cat>init.c<<'___EOF___'#ifndef NOLIBC#include<unistd.h>#include<sys/time.h>#endifvolatileunsignedlongdelaycount;intmain(intargc,intargv[]){inti;structtimevaltv;structtimevaltvb;for(;;){sleep(1);/* Need some userspace time. */if(gettimeofday(&tvb,NULL))continue;do{for(i=0;i<1000*100;i++)delaycount=i*i;if(gettimeofday(&tv,NULL))break;tv.tv_sec-=tvb.tv_sec;if(tv.tv_sec>1)break;tv.tv_usec+=tv.tv_sec*1000*1000;tv.tv_usec-=tvb.tv_usec;}while(tv.tv_usec<1000);}return0;}___EOF___# build using nolibc on supported archs (smaller executable) and fall# back to regular glibc on other ones.ifecho-e"#if __x86_64__||__i386__||__i486__||__i586__||__i686__"\
"||__ARM_EABI__||__aarch64__\nyes\n#endif"\
|${CROSS_COMPILE}gcc-E-nostdlib-xc-\
|grep-q'^yes';then# architecture supported by nolibc${CROSS_COMPILE}gcc-fno-asynchronous-unwind-tables-fno-ident\
-nostdlib-include../../../../include/nolibc/nolibc.h\
-s-static-Os-oinitinit.c-lgccelse${CROSS_COMPILE}gcc-s-static-Os-oinitinit.cfi
Huh. In PowerPC, is there some difference between system calls
executed in initrd and those same system calls executed in userspace?
I've faced some issues in the past with certain syscalls not working
exactly the same on pid 1 (I think it was setsid() or setpgrp(), but
I could be wrong, that was ~10 years ago). Maybe here we're seeing
something similar with set_tid_address() or set_robust_list().
And just to make sure, the above strace was from exactly the same
binary "init" file that is included in initrd, correct?
Adding Willy Tarreau for his thoughts.
Thanx, Paul
quoted
Any ideas, what `mkinitrd.sh` [2] should do differently?
I think that we could add a fork() to see if the PID changes anything:
quoted
#ifndef NOLIBC
#include <unistd.h>
#include <sys/time.h>
#endif
volatile unsigned long delaycount;
int main(int argc, int argv[])
{
int i;
struct timeval tv;
struct timeval tvb;
Could you try with this ugly hack here ?
+ if (fork() > 0) {
+ wait(NULL);
+ return 0;
+ }
quoted
for (;;) {
sleep(1);
/* Need some userspace time. */
if (gettimeofday(&tvb, NULL))
continue;
do {
for (i = 0; i < 1000 * 100; i++)
delaycount = i * i;
if (gettimeofday(&tv, NULL))
break;
tv.tv_sec -= tvb.tv_sec;
if (tv.tv_sec > 1)
break;
tv.tv_usec += tv.tv_sec * 1000 * 1000;
tv.tv_usec -= tvb.tv_usec;
} while (tv.tv_usec < 1000);
}
return 0;
}
Dear Paul
I am also very interested in the topic.
The Open source lab of Oregon State University has lent me a 8 core
power ppc64el VM for 3 months, I guess I can try reproducing this bug
in the Virtual Machine by executing qemu in non hardware accelerated
mode (using -no-kvm argument).
I am currently doing research on
https://lore.kernel.org/rcu/20220201175023.GW4285@paulmck-ThinkPad-P17-Gen-1/T/#mc7e5f8ec99e3794bec1e38fbbb130e71172e4759,
I think I can give a preliminary short report on that previous topic
tomorrow. And I am very interested in doing a search on the new topic
the day after tomorrow.
Thank you both for providing me an opportunity to improve myself ;-)
Thanks again
Zhouyi
On Tue, Feb 8, 2022 at 12:10 PM Paul E. McKenney [off-list ref] wrote:
On Mon, Feb 07, 2022 at 05:44:47PM +0100, Paul Menzel wrote:
quoted
Dear Linux folks,
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
GNU/Linux 3.10.0, stripped
segfaults in QEMU. From one of the log files
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18
lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84
4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4
38000000 <f821ff81> 7c0803a6 f8010000 e9028010
Executing the init, which just seems to be an endless loop, from userspace
work:
$ strace ./tools/testing/selftests/rcutorture/initrd/init
execve("./tools/testing/selftests/rcutorture/initrd/init",
["./tools/testing/selftests/rcutor"...], 0x7ffffdb9e860 /* 31 vars */) = 0
brk(NULL) = 0x1001d940000
brk(0x1001d940b98) = 0x1001d940b98
set_tid_address(0x1001d9400d0) = 2890832
set_robust_list(0x1001d9400e0, 24) = 0
uname({sysname="Linux",
nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"..., 4096)
= 61
getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
brk(0x1001d970b98) = 0x1001d970b98
brk(0x1001d980000) = 0x1001d980000
mprotect(0x100e0000, 65536, PROT_READ) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^C{tv_sec=0,
tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
strace: Process 2890832 detached
Huh. In PowerPC, is there some difference between system calls
executed in initrd and those same system calls executed in userspace?
And just to make sure, the above strace was from exactly the same
binary "init" file that is included in initrd, correct?
Adding Willy Tarreau for his thoughts.
Thanx, Paul
quoted
Any ideas, what `mkinitrd.sh` [2] should do differently?
cat>init.c<<'___EOF___'#ifndef NOLIBC#include<unistd.h>#include<sys/time.h>#endifvolatileunsignedlongdelaycount;intmain(intargc,intargv[]){inti;structtimevaltv;structtimevaltvb;for(;;){sleep(1);/* Need some userspace time. */if(gettimeofday(&tvb,NULL))continue;do{for(i=0;i<1000*100;i++)delaycount=i*i;if(gettimeofday(&tv,NULL))break;tv.tv_sec-=tvb.tv_sec;if(tv.tv_sec>1)break;tv.tv_usec+=tv.tv_sec*1000*1000;tv.tv_usec-=tvb.tv_usec;}while(tv.tv_usec<1000);}return0;}___EOF___# build using nolibc on supported archs (smaller executable) and fall# back to regular glibc on other ones.ifecho-e"#if __x86_64__||__i386__||__i486__||__i586__||__i686__"\
"||__ARM_EABI__||__aarch64__\nyes\n#endif"\
|${CROSS_COMPILE}gcc-E-nostdlib-xc-\
|grep-q'^yes';then# architecture supported by nolibc${CROSS_COMPILE}gcc-fno-asynchronous-unwind-tables-fno-ident\
-nostdlib-include../../../../include/nolibc/nolibc.h\
-s-static-Os-oinitinit.c-lgccelse${CROSS_COMPILE}gcc-s-static-Os-oinitinit.cfi
Hi,
The mailing list forward the emails to me in periodic style, very
sorry not seeing Willy's email until I visited
https://lore.kernel.org/rcu/20220207180901.GB14608@1wt.eu/T/#u, I am
also very interested in testing Willy's proposal.
Thanks a lot
Zhouyi
On Tue, Feb 8, 2022 at 1:46 PM Zhouyi Zhou [off-list ref] wrote:
Dear Paul
I am also very interested in the topic.
The Open source lab of Oregon State University has lent me a 8 core
power ppc64el VM for 3 months, I guess I can try reproducing this bug
in the Virtual Machine by executing qemu in non hardware accelerated
mode (using -no-kvm argument).
I am currently doing research on
https://lore.kernel.org/rcu/20220201175023.GW4285@paulmck-ThinkPad-P17-Gen-1/T/#mc7e5f8ec99e3794bec1e38fbbb130e71172e4759,
I think I can give a preliminary short report on that previous topic
tomorrow. And I am very interested in doing a search on the new topic
the day after tomorrow.
Thank you both for providing me an opportunity to improve myself ;-)
Thanks again
Zhouyi
On Tue, Feb 8, 2022 at 12:10 PM Paul E. McKenney [off-list ref] wrote:
quoted
On Mon, Feb 07, 2022 at 05:44:47PM +0100, Paul Menzel wrote:
quoted
Dear Linux folks,
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
GNU/Linux 3.10.0, stripped
segfaults in QEMU. From one of the log files
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18
lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84
4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4
38000000 <f821ff81> 7c0803a6 f8010000 e9028010
Executing the init, which just seems to be an endless loop, from userspace
work:
$ strace ./tools/testing/selftests/rcutorture/initrd/init
execve("./tools/testing/selftests/rcutorture/initrd/init",
["./tools/testing/selftests/rcutor"...], 0x7ffffdb9e860 /* 31 vars */) = 0
brk(NULL) = 0x1001d940000
brk(0x1001d940b98) = 0x1001d940b98
set_tid_address(0x1001d9400d0) = 2890832
set_robust_list(0x1001d9400e0, 24) = 0
uname({sysname="Linux",
nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"..., 4096)
= 61
getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
brk(0x1001d970b98) = 0x1001d970b98
brk(0x1001d980000) = 0x1001d980000
mprotect(0x100e0000, 65536, PROT_READ) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
0x7ffffb22c8a8) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^C{tv_sec=0,
tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
strace: Process 2890832 detached
Huh. In PowerPC, is there some difference between system calls
executed in initrd and those same system calls executed in userspace?
And just to make sure, the above strace was from exactly the same
binary "init" file that is included in initrd, correct?
Adding Willy Tarreau for his thoughts.
Thanx, Paul
quoted
Any ideas, what `mkinitrd.sh` [2] should do differently?
cat>init.c<<'___EOF___'#ifndef NOLIBC#include<unistd.h>#include<sys/time.h>#endifvolatileunsignedlongdelaycount;intmain(intargc,intargv[]){inti;structtimevaltv;structtimevaltvb;for(;;){sleep(1);/* Need some userspace time. */if(gettimeofday(&tvb,NULL))continue;do{for(i=0;i<1000*100;i++)delaycount=i*i;if(gettimeofday(&tv,NULL))break;tv.tv_sec-=tvb.tv_sec;if(tv.tv_sec>1)break;tv.tv_usec+=tv.tv_sec*1000*1000;tv.tv_usec-=tvb.tv_usec;}while(tv.tv_usec<1000);}return0;}___EOF___# build using nolibc on supported archs (smaller executable) and fall# back to regular glibc on other ones.ifecho-e"#if __x86_64__||__i386__||__i486__||__i586__||__i686__"\
"||__ARM_EABI__||__aarch64__\nyes\n#endif"\
|${CROSS_COMPILE}gcc-E-nostdlib-xc-\
|grep-q'^yes';then# architecture supported by nolibc${CROSS_COMPILE}gcc-fno-asynchronous-unwind-tables-fno-ident\
-nostdlib-include../../../../include/nolibc/nolibc.h\
-s-static-Os-oinitinit.c-lgccelse${CROSS_COMPILE}gcc-s-static-Os-oinitinit.cfi
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2022-02-08 10:10:02
Paul Menzel [off-list ref] writes:
Dear Linux folks,
Hi Paul,
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for
GNU/Linux 3.10.0, stripped
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84 4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4 38000000 <f821ff81> 7c0803a6 f8010000 e9028010
The disassembly from 3c40100f is:
lis r2,4111
addi r2,r2,31744
mr r9,r1
rldicr r1,r1,0,59
li r0,0
stdu r1,-128(r1) <- fault
mtlr r0
std r0,0(r1)
ld r8,-32752(r2)
I think you'll find that's the code at the ELF entry point. You can
check with:
$ readelf -e tools/testing/selftests/rcutorture/initrd/init | grep Entry
Entry point address: 0x10000c0c
$ objdump -d tools/testing/selftests/rcutorture/initrd/init | grep -m 1 -A 8 10000c0c
10000c0c: 0e 10 40 3c lis r2,4110
10000c10: 00 7b 42 38 addi r2,r2,31488
10000c14: 78 0b 29 7c mr r9,r1
10000c18: e4 06 21 78 rldicr r1,r1,0,59
10000c1c: 00 00 00 38 li r0,0
10000c20: 81 ff 21 f8 stdu r1,-128(r1)
10000c24: a6 03 08 7c mtlr r0
10000c28: 00 00 01 f8 std r0,0(r1)
10000c2c: 10 80 02 e9 ld r8,-32752(r2)
The fault you're seeing is the first store using the stack pointer (r1),
which is setup by the kernel.
The fault address f0656d90 is weirdly low, the stack should be up near 128TB.
I'm not sure how we end up with a bad r1.
Can you dump some info about the kernel that was built, something like:
$ file /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/vmlinux
And maybe paste/attach the full log, maybe there's a clue somewhere.
cheers
From: Paul Menzel <hidden> Date: 2022-02-08 12:13:18
Dear Michael,
Thank you for looking into this.
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
Paul Menzel writes:
[…]
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+
(pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu
clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28
17:13:04 CET 2022
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems
with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
)
quoted
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84 4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4 38000000 <f821ff81> 7c0803a6 f8010000 e9028010
The disassembly from 3c40100f is:
lis r2,4111
addi r2,r2,31744
mr r9,r1
rldicr r1,r1,0,59
li r0,0
stdu r1,-128(r1) <- fault
mtlr r0
std r0,0(r1)
ld r8,-32752(r2)
I think you'll find that's the code at the ELF entry point. You can
check with:
$ readelf -e tools/testing/selftests/rcutorture/initrd/init | grep Entry
Entry point address: 0x10000c0c
$ objdump -d tools/testing/selftests/rcutorture/initrd/init | grep -m 1 -A 8 10000c0c
10000c0c: 0e 10 40 3c lis r2,4110
10000c10: 00 7b 42 38 addi r2,r2,31488
10000c14: 78 0b 29 7c mr r9,r1
10000c18: e4 06 21 78 rldicr r1,r1,0,59
10000c1c: 00 00 00 38 li r0,0
10000c20: 81 ff 21 f8 stdu r1,-128(r1)
10000c24: a6 03 08 7c mtlr r0
10000c28: 00 00 01 f8 std r0,0(r1)
10000c2c: 10 80 02 e9 ld r8,-32752(r2)
The fault you're seeing is the first store using the stack pointer (r1),
which is setup by the kernel.
The fault address f0656d90 is weirdly low, the stack should be up near 128TB.
I'm not sure how we end up with a bad r1.
Can you dump some info about the kernel that was built, something like:
$ file /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/vmlinux
And maybe paste/attach the full log, maybe there's a clue somewhere.
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial
stdio -m 512 -kernel
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0
torture.disable_onoff_at_boot locktorture.onoff_interval=3
locktorture.onoff_holdoff=30 locktorture.stat_interval=15
locktorture.shutdown_secs=60 locktorture.verbose=1"
Kind regards,
Paul
[1]:
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
From: Paul Menzel <hidden> Date: 2022-02-08 12:28:22
[Correct sha1 for test for 2022.02.01-21.52.37]
Am 08.02.22 um 13:12 schrieb Paul Menzel:
Dear Michael,
Thank you for looking into this.
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+ (pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28 17:13:04 CET 2022
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
I was able to reproduce this with the above, but the report and the
attached logs at the end are from:
$ git log --oneline -6 b37a34a8cf5a
b37a34a8cf5a Problems with rcutorture on ppc64le: allmodconfig(2)
and other failures
9a78ddead89a ata: libata-sata: improve sata_link_debounce()
567da2eaf099 ata: libata-sata: remove debounce delay by default
70ae61851660 ata: libata-sata: introduce struct sata_deb_timing
9ebb6433d9c3 ata: libata-sata: Simplify sata_link_resume() interface
26291c54e111 (tag: v5.17-rc2) Linux 5.17-rc2
quoted
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project
developers
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
)
quoted
quoted
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84 4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4 38000000 <f821ff81> 7c0803a6 f8010000 e9028010
The disassembly from 3c40100f is:
lis r2,4111
addi r2,r2,31744
mr r9,r1
rldicr r1,r1,0,59
li r0,0
stdu r1,-128(r1) <- fault
mtlr r0
std r0,0(r1)
ld r8,-32752(r2)
I think you'll find that's the code at the ELF entry point. You can
check with:
$ readelf -e tools/testing/selftests/rcutorture/initrd/init | grep
Entry
Entry point address: 0x10000c0c
$ objdump -d tools/testing/selftests/rcutorture/initrd/init | grep
-m 1 -A 8 10000c0c
10000c0c: 0e 10 40 3c lis r2,4110
10000c10: 00 7b 42 38 addi r2,r2,31488
10000c14: 78 0b 29 7c mr r9,r1
10000c18: e4 06 21 78 rldicr r1,r1,0,59
10000c1c: 00 00 00 38 li r0,0
10000c20: 81 ff 21 f8 stdu r1,-128(r1)
10000c24: a6 03 08 7c mtlr r0
10000c28: 00 00 01 f8 std r0,0(r1)
10000c2c: 10 80 02 e9 ld r8,-32752(r2)
The fault you're seeing is the first store using the stack pointer (r1),
which is setup by the kernel.
The fault address f0656d90 is weirdly low, the stack should be up near
128TB.
I'm not sure how we end up with a bad r1.
Can you dump some info about the kernel that was built, something like:
$ file
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/vmlinux
And maybe paste/attach the full log, maybe there's a clue somewhere.
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial
stdio -m 512 -kernel
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0
torture.disable_onoff_at_boot locktorture.onoff_interval=3
locktorture.onoff_holdoff=30 locktorture.stat_interval=15
locktorture.shutdown_secs=60 locktorture.verbose=1"
Kind regards,
Paul
[1]: https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2022-02-11 01:49:09
Paul Menzel [off-list ref] writes:
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+
(pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu
clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28
17:13:04 CET 2022
OK. In general rc1 kernels can have issues, so it might be worth
rebooting the host into either v5.17-rc3 or a distro or stable kernel.
Just to rule out any issues on the host.
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems
with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
quoted
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
OK, that's one difference between our setups, but I'd be surprised if it
explains this bug, but I guess anything's possible.
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
OK yeah I see that too.
Removing "threadirqs" from tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot
seems to fix it.
I still see some preempt related warnings, we clearly have some bugs
with preempt enabled.
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial
stdio -m 512 -kernel
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0
torture.disable_onoff_at_boot locktorture.onoff_interval=3
locktorture.onoff_holdoff=30 locktorture.stat_interval=15
locktorture.shutdown_secs=60 locktorture.verbose=1"
That works fine for me, boots and runs the test, then shuts down.
I assume you see the segfault on every boot, not intermittently?
So the differences between our setups are the host kernel and the qemu
version. Can you try a different host kernel easily?
The other thing would be to try a different qemu version, you might need
to build from source, but it's not that hard :)
cheers
From: Paul Menzel <hidden> Date: 2022-02-11 14:20:18
Dear Michael,
Am 11.02.22 um 02:48 schrieb Michael Ellerman:
Paul Menzel writes:
quoted
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+ (x@eddb.molgen.mpg.de) (Ubuntu clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28 17:13:04 CET 2022
OK. In general rc1 kernels can have issues, so it might be worth
rebooting the host into either v5.17-rc3 or a distro or stable kernel.
Just to rule out any issues on the host.
Yes, that was a good test. It works with Ubuntu’s 5.13 Linux kernel.
$ more /proc/version
Linux version 5.13.0-28-generic (buildd@bos02-ppc64el-013) (gcc
(Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37)
#31-Ubuntu SMP Thu Jan 13 17:40:19 UTC 2022
I have to do more tests, but it could be LLVM/clang related.
quoted
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
quoted
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
OK, that's one difference between our setups, but I'd be surprised if it
explains this bug, but I guess anything's possible.
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
OK yeah I see that too.
Removing "threadirqs" from tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot
seems to fix it.
Nice find. I have no idea, what that means though.
I still see some preempt related warnings, we clearly have some bugs
with preempt enabled.
quoted
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8 \
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial stdio -m 512 \
-kernel /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux \
-append "debug_boot_weak_hash panic=-1 console=ttyS0 \
torture.disable_onoff_at_boot locktorture.onoff_interval=3 \
locktorture.onoff_holdoff=30 locktorture.stat_interval=15 \
locktorture.shutdown_secs=60 locktorture.verbose=1"
That works fine for me, boots and runs the test, then shuts down.
I assume you see the segfault on every boot, not intermittently?
So the differences between our setups are the host kernel and the qemu
version. Can you try a different host kernel easily?
The other thing would be to try a different qemu version, you might need
to build from source, but it's not that hard :)
Indeed. I needed to find a current Meson, but then it didn’t make a
difference, as found out above, it’s related to the Linux kernel.
Kind regards,
Paul
From: Paul Menzel <hidden> Date: 2022-02-11 15:43:46
Dear Michael,
Am 11.02.22 um 15:19 schrieb Paul Menzel:
Am 11.02.22 um 02:48 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
quoted
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel
and of the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+ (x@eddb.molgen.mpg.de) (Ubuntu clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28 17:13:04 CET 2022
OK. In general rc1 kernels can have issues, so it might be worth
rebooting the host into either v5.17-rc3 or a distro or stable kernel.
Just to rule out any issues on the host.
Yes, that was a good test. It works with Ubuntu’s 5.13 Linux kernel.
$ more /proc/version
Linux version 5.13.0-28-generic (buildd@bos02-ppc64el-013) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #31-Ubuntu SMP Thu Jan 13 17:40:19 UTC 2022
I have to do more tests, but it could be LLVM/clang related.
Building commit f1baf68e1383 (Merge tag 'net-5.17-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net) with the ata
patches on top with GCC, I am unable to reproduce the issue. Before I
built it with
make -j100 LLVM=1 LLVM_IAS=0 bindeb-pkg
[…]
Kind regards,
Paul
Dear Paul
I try to reproduce the bug in ppc64 VM in Oregon State University
using the vmlinux extracted from
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
the ppc64 VM in which I run the qemu without hardware acceleration is:
Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc
version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb
3 18:43:11 UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166)
The qemu command I use to test:
cd /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$
$qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M
pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log
-m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1
console=ttyS0 rcutorture.onoff_interval=200
rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12
rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2 threadirqs tree.use_softirq=0
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15
rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1
rcutorture.verbose=1"
The console.log is uploaded to:
http://154.223.142.244/logs/20220310/console.paul.log
The log tells us it is illegal instruction that causes the trouble:
[ 4.246387][ T1] init[1]: illegal instruction (4) at 1002c308
nip 1002c308 lr 10001684 code 1 in init[10000000+d0000]
[ 4.251400][ T1] init[1]: code: f90d88c0 f92a0008 f9480008
7c2004ac 2c2d0000 f9490000 386d88d0 380000e8
[ 4.253416][ T1] init[1]: code: 41820098 e92d8f98 75290010
4182008c <44000001> 2c2d0000 60000000 8902f438
Meanwhile, the vmlinux compiled by myself runs smoothly.
Then I modify mkinitrd.sh to let it panic manually:
http://154.223.142.244/logs/20220310/mkinitrd.sh
The log tells us it is a segfault (instead of a illegal instruction):
http://154.223.142.244/logs/20220310/console.zhouyi.log
Then I use gdb to debug the init in host:
ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb
tools/testing/selftests/rcutorture/initrd/init
(gdb) run
Starting program:
/home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init
Program received signal SIGSEGV, Segmentation fault.
0x0000000010000b2c in ?? ()
(gdb) x/10i $pc
=> 0x10000b2c: stw r9,0(r9)
0x10000b30: trap
0x10000b34: .long 0x0
0x10000b38: .long 0x0
0x10000b3c: .long 0x0
0x10000b40: lis r2,4110
0x10000b44: addi r2,r2,31488
0x10000b48: mr r9,r1
0x10000b4c: rldicr r1,r1,0,59
0x10000b50: li r0,0
(gdb) p $r9
$1 = 0
(gdb) x/30x $pc - 0x30
0x10000afc: 0x38840040 0x387f0040 0xf8010040 0x48026919
0x10000b0c: 0x60000000 0xe8010040 0x7c0803a6 0x4bffff24
0x10000b1c: 0x00000000 0x01000000 0x00000180 0x39200000
0x10000b2c: 0x91290000 0x7fe00008 0x00000000 0x00000000
which matches the hex content of
http://154.223.142.244/logs/20220310/console.zhouyi.log:
[ 5.077431][ T1] init[1]: segfault (11) at 0 nip 10000b2c lr
10001024 code 1 in init[10000000+d0000]
[ 5.087167][ T1] init[1]: code: 38840040 387f0040 f8010040
48026919 60000000 e8010040 7c0803a6 4bffff24
[ 5.093987][ T1] init[1]: code: 00000000 01000000 00000180
39200000 <91290000> 7fe00008 00000000 00000000
Conclusions: there might be something wrong when packing the init into
vmlinux in your environment.
I will continue to do research on this interesting problem with you.
Thanks
Kind Regards
Zhouyi
On Tue, Feb 8, 2022 at 8:12 PM Paul Menzel [off-list ref] wrote:
Dear Michael,
Thank you for looking into this.
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+
(pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu
clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28
17:13:04 CET 2022
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems
with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
quoted
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
)
quoted
quoted
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84 4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4 38000000 <f821ff81> 7c0803a6 f8010000 e9028010
The disassembly from 3c40100f is:
lis r2,4111
addi r2,r2,31744
mr r9,r1
rldicr r1,r1,0,59
li r0,0
stdu r1,-128(r1) <- fault
mtlr r0
std r0,0(r1)
ld r8,-32752(r2)
I think you'll find that's the code at the ELF entry point. You can
check with:
$ readelf -e tools/testing/selftests/rcutorture/initrd/init | grep Entry
Entry point address: 0x10000c0c
$ objdump -d tools/testing/selftests/rcutorture/initrd/init | grep -m 1 -A 8 10000c0c
10000c0c: 0e 10 40 3c lis r2,4110
10000c10: 00 7b 42 38 addi r2,r2,31488
10000c14: 78 0b 29 7c mr r9,r1
10000c18: e4 06 21 78 rldicr r1,r1,0,59
10000c1c: 00 00 00 38 li r0,0
10000c20: 81 ff 21 f8 stdu r1,-128(r1)
10000c24: a6 03 08 7c mtlr r0
10000c28: 00 00 01 f8 std r0,0(r1)
10000c2c: 10 80 02 e9 ld r8,-32752(r2)
The fault you're seeing is the first store using the stack pointer (r1),
which is setup by the kernel.
The fault address f0656d90 is weirdly low, the stack should be up near 128TB.
I'm not sure how we end up with a bad r1.
Can you dump some info about the kernel that was built, something like:
$ file /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/vmlinux
And maybe paste/attach the full log, maybe there's a clue somewhere.
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial
stdio -m 512 -kernel
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0
torture.disable_onoff_at_boot locktorture.onoff_interval=3
locktorture.onoff_holdoff=30 locktorture.stat_interval=15
locktorture.shutdown_secs=60 locktorture.verbose=1"
Kind regards,
Paul
[1]:
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
From: "Paul E. McKenney" <paulmck@kernel.org> Date: 2022-03-10 04:49:35
On Thu, Mar 10, 2022 at 10:37:12AM +0800, Zhouyi Zhou wrote:
Dear Paul
I try to reproduce the bug in ppc64 VM in Oregon State University
using the vmlinux extracted from
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
the ppc64 VM in which I run the qemu without hardware acceleration is:
Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc
version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb
3 18:43:11 UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166)
The qemu command I use to test:
cd /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$
$qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M
pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log
-m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1
console=ttyS0 rcutorture.onoff_interval=200
rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12
rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2 threadirqs tree.use_softirq=0
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15
rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1
rcutorture.verbose=1"
The console.log is uploaded to:
http://154.223.142.244/logs/20220310/console.paul.log
The log tells us it is illegal instruction that causes the trouble:
[ 4.246387][ T1] init[1]: illegal instruction (4) at 1002c308
nip 1002c308 lr 10001684 code 1 in init[10000000+d0000]
[ 4.251400][ T1] init[1]: code: f90d88c0 f92a0008 f9480008
7c2004ac 2c2d0000 f9490000 386d88d0 380000e8
[ 4.253416][ T1] init[1]: code: 41820098 e92d8f98 75290010
4182008c <44000001> 2c2d0000 60000000 8902f438
Meanwhile, the vmlinux compiled by myself runs smoothly.
Then I modify mkinitrd.sh to let it panic manually:
http://154.223.142.244/logs/20220310/mkinitrd.sh
The log tells us it is a segfault (instead of a illegal instruction):
http://154.223.142.244/logs/20220310/console.zhouyi.log
Then I use gdb to debug the init in host:
ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb
tools/testing/selftests/rcutorture/initrd/init
(gdb) run
Starting program:
/home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init
Program received signal SIGSEGV, Segmentation fault.
0x0000000010000b2c in ?? ()
(gdb) x/10i $pc
=> 0x10000b2c: stw r9,0(r9)
0x10000b30: trap
0x10000b34: .long 0x0
0x10000b38: .long 0x0
0x10000b3c: .long 0x0
0x10000b40: lis r2,4110
0x10000b44: addi r2,r2,31488
0x10000b48: mr r9,r1
0x10000b4c: rldicr r1,r1,0,59
0x10000b50: li r0,0
(gdb) p $r9
$1 = 0
(gdb) x/30x $pc - 0x30
0x10000afc: 0x38840040 0x387f0040 0xf8010040 0x48026919
0x10000b0c: 0x60000000 0xe8010040 0x7c0803a6 0x4bffff24
0x10000b1c: 0x00000000 0x01000000 0x00000180 0x39200000
0x10000b2c: 0x91290000 0x7fe00008 0x00000000 0x00000000
which matches the hex content of
http://154.223.142.244/logs/20220310/console.zhouyi.log:
[ 5.077431][ T1] init[1]: segfault (11) at 0 nip 10000b2c lr
10001024 code 1 in init[10000000+d0000]
[ 5.087167][ T1] init[1]: code: 38840040 387f0040 f8010040
48026919 60000000 e8010040 7c0803a6 4bffff24
[ 5.093987][ T1] init[1]: code: 00000000 01000000 00000180
39200000 <91290000> 7fe00008 00000000 00000000
Conclusions: there might be something wrong when packing the init into
vmlinux in your environment.
Quite possibly! Or the compiler might not be being invoked properly
by the mkinitrd.sh script.
I will continue to do research on this interesting problem with you.
Please let me know how it goes!
Thanx, Paul
Thanks
Kind Regards
Zhouyi
On Tue, Feb 8, 2022 at 8:12 PM Paul Menzel [off-list ref] wrote:
quoted
Dear Michael,
Thank you for looking into this.
Am 08.02.22 um 11:09 schrieb Michael Ellerman:
quoted
Paul Menzel writes:
[…]
quoted
quoted
On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
5.17-rc2+ with rcutorture tests
I'm not sure if that's the host kernel version or the version you're
using of rcutorture? Can you tell us the sha1 of your host kernel and of
the tree you're running rcutorture from?
The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately,
I am unable to find the exact sha1.
$ more /proc/version
Linux version 5.17.0-rc1+
(pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu
clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28
17:13:04 CET 2022
The Linux tree, from where I run rcutorture from, is at commit
dfd42facf1e4 (Linux 5.17-rc3) with four patches on top:
$ git log --oneline -6
207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems
with rcutorture on ppc64le: allmodconfig(2) and other failures
8c82f96fbe57 ata: libata-sata: improve sata_link_debounce()
a447541d925f ata: libata-sata: remove debounce delay by default
afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing
f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface
dfd42facf1e4 (tag: v5.17-rc3) Linux 5.17-rc3
quoted
quoted
$ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
the built init
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for GNU/Linux 3.10.0, stripped
Mine looks pretty much identical:
$ file tools/testing/selftests/rcutorture/initrd/init
tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, BuildID[sha1]=86078bf6e5d54ab0860d36aa9a65d52818b972c8, for GNU/Linux 3.10.0, stripped
quoted
segfaults in QEMU. From one of the log files
But mine doesn't segfault, it runs fine and the test completes.
What qemu version are you using?
I tried 4.2.1 and 6.2.0, both worked.
$ qemu-system-ppc64le --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
Sorry, that was the wrong path/test. The correct one for the excerpt
below is:
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/console.log
(For TREE03, QEMU does not start the Linux kernel at all, that means no
output after:
Booting Linux via __start() @ 0x0000000000400000 ...
)
quoted
quoted
[ 1.119803][ T1] Run /init as init process
[ 1.122011][ T1] init[1]: segfault (11) at f0656d90 nip 10000a18 lr 0 code 1 in init[10000000+d0000]
[ 1.124863][ T1] init[1]: code: 2c2903e7 f9210030 4081ff84 4bffff58 00000000 01000000 00000580 3c40100f
[ 1.128823][ T1] init[1]: code: 38427c00 7c290b78 782106e4 38000000 <f821ff81> 7c0803a6 f8010000 e9028010
The disassembly from 3c40100f is:
lis r2,4111
addi r2,r2,31744
mr r9,r1
rldicr r1,r1,0,59
li r0,0
stdu r1,-128(r1) <- fault
mtlr r0
std r0,0(r1)
ld r8,-32752(r2)
I think you'll find that's the code at the ELF entry point. You can
check with:
$ readelf -e tools/testing/selftests/rcutorture/initrd/init | grep Entry
Entry point address: 0x10000c0c
$ objdump -d tools/testing/selftests/rcutorture/initrd/init | grep -m 1 -A 8 10000c0c
10000c0c: 0e 10 40 3c lis r2,4110
10000c10: 00 7b 42 38 addi r2,r2,31488
10000c14: 78 0b 29 7c mr r9,r1
10000c18: e4 06 21 78 rldicr r1,r1,0,59
10000c1c: 00 00 00 38 li r0,0
10000c20: 81 ff 21 f8 stdu r1,-128(r1)
10000c24: a6 03 08 7c mtlr r0
10000c28: 00 00 01 f8 std r0,0(r1)
10000c2c: 10 80 02 e9 ld r8,-32752(r2)
The fault you're seeing is the first store using the stack pointer (r1),
which is setup by the kernel.
The fault address f0656d90 is weirdly low, the stack should be up near 128TB.
I'm not sure how we end up with a bad r1.
Can you dump some info about the kernel that was built, something like:
$ file /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/vmlinux
And maybe paste/attach the full log, maybe there's a clue somewhere.
You can now download the content of
`/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01`
[1, 65 MB].
Can you reproduce the segmentation fault with the line below?
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=8
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial
stdio -m 512 -kernel
/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0
torture.disable_onoff_at_boot locktorture.onoff_interval=3
locktorture.onoff_holdoff=30 locktorture.stat_interval=15
locktorture.shutdown_secs=60 locktorture.verbose=1"
Kind regards,
Paul
[1]:
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
From: Paul Menzel <hidden> Date: 2022-03-10 08:11:04
Dear Zhouyi,
Thank you for still looking into this.
Am 10.03.22 um 03:37 schrieb Zhouyi Zhou:
I try to reproduce the bug in ppc64 VM in Oregon State University
using the vmlinux extracted from
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
the ppc64 VM in which I run the qemu without hardware acceleration is:
Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb 3 18:43:11 UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166)
The qemu command I use to test:
cd /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$
$qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M
pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log
-m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1
console=ttyS0 rcutorture.onoff_interval=200
rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12
rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2 threadirqs tree.use_softirq=0
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15
rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1
rcutorture.verbose=1"
The console.log is uploaded to:
http://154.223.142.244/logs/20220310/console.paul.log
The log tells us it is illegal instruction that causes the trouble:
[ 4.246387][ T1] init[1]: illegal instruction (4) at 1002c308 nip 1002c308 lr 10001684 code 1 in init[10000000+d0000]
[ 4.251400][ T1] init[1]: code: f90d88c0 f92a0008 f9480008 7c2004ac 2c2d0000 f9490000 386d88d0 380000e8
[ 4.253416][ T1] init[1]: code: 41820098 e92d8f98 75290010 4182008c <44000001> 2c2d0000 60000000 8902f438
Meanwhile, the vmlinux compiled by myself runs smoothly.
How did you build it? Using GCC or clang? I forgot, if the problem was
only reproducible if the host Linux kernel was built with clang or the
VM kernel.
I only see the change:
-
+ int *ptr = 0;
+ *ptr = 0;
The log tells us it is a segfault (instead of a illegal instruction):
http://154.223.142.244/logs/20220310/console.zhouyi.log
Then I use gdb to debug the init in host:
ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb
tools/testing/selftests/rcutorture/initrd/init
(gdb) run
Starting program:
/home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init
Program received signal SIGSEGV, Segmentation fault.
0x0000000010000b2c in ?? ()
(gdb) x/10i $pc
=> 0x10000b2c: stw r9,0(r9)
0x10000b30: trap
0x10000b34: .long 0x0
0x10000b38: .long 0x0
0x10000b3c: .long 0x0
0x10000b40: lis r2,4110
0x10000b44: addi r2,r2,31488
0x10000b48: mr r9,r1
0x10000b4c: rldicr r1,r1,0,59
0x10000b50: li r0,0
(gdb) p $r9
$1 = 0
(gdb) x/30x $pc - 0x30
0x10000afc: 0x38840040 0x387f0040 0xf8010040 0x48026919
0x10000b0c: 0x60000000 0xe8010040 0x7c0803a6 0x4bffff24
0x10000b1c: 0x00000000 0x01000000 0x00000180 0x39200000
0x10000b2c: 0x91290000 0x7fe00008 0x00000000 0x00000000
which matches the hex content of
http://154.223.142.244/logs/20220310/console.zhouyi.log:
[ 5.077431][ T1] init[1]: segfault (11) at 0 nip 10000b2c lr 10001024 code 1 in init[10000000+d0000]
[ 5.087167][ T1] init[1]: code: 38840040 387f0040 f8010040 48026919 60000000 e8010040 7c0803a6 4bffff24
[ 5.093987][ T1] init[1]: code: 00000000 01000000 00000180 39200000 <91290000> 7fe00008 00000000 00000000
Conclusions: there might be something wrong when packing the init into
vmlinux in your environment.
I will continue to do research on this interesting problem with you.
As written I think it’s a problem with LLVM/clang. Unfortunately, I
won’t be able to retest before next week.
Kind regards,
Paul
Dear Paul
On Thu, Mar 10, 2022 at 4:10 PM Paul Menzel [off-list ref] wrote:
Dear Zhouyi,
Thank you for still looking into this.
You are very welcome ;-)
Am 10.03.22 um 03:37 schrieb Zhouyi Zhou:
quoted
I try to reproduce the bug in ppc64 VM in Oregon State University
using the vmlinux extracted from
https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz
the ppc64 VM in which I run the qemu without hardware acceleration is:
Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb 3 18:43:11 UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166)
The qemu command I use to test:
cd /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$
$qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M
pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log
-m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1
console=ttyS0 rcutorture.onoff_interval=200
rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12
rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2 threadirqs tree.use_softirq=0
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15
rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1
rcutorture.verbose=1"
The console.log is uploaded to:
http://154.223.142.244/logs/20220310/console.paul.log
The log tells us it is illegal instruction that causes the trouble:
[ 4.246387][ T1] init[1]: illegal instruction (4) at 1002c308 nip 1002c308 lr 10001684 code 1 in init[10000000+d0000]
[ 4.251400][ T1] init[1]: code: f90d88c0 f92a0008 f9480008 7c2004ac 2c2d0000 f9490000 386d88d0 380000e8
[ 4.253416][ T1] init[1]: code: 41820098 e92d8f98 75290010 4182008c <44000001> 2c2d0000 60000000 8902f438
Meanwhile, the vmlinux compiled by myself runs smoothly.
How did you build it? Using GCC or clang? I forgot, if the problem was
I built vmlinux(es) using GCC and clang both. The compiled vmlinux(es)
runs smoothly.
only reproducible if the host Linux kernel was built with clang or the
VM kernel.
Yes, I also remember this, the dependence of how the host Linux kernel
is built makes things more complex.
I only see the change:
-
+ int *ptr = 0;
+ *ptr = 0;
Yes, I make the segfault happen manually.
quoted
The log tells us it is a segfault (instead of a illegal instruction):
http://154.223.142.244/logs/20220310/console.zhouyi.log
Then I use gdb to debug the init in host:
ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb
tools/testing/selftests/rcutorture/initrd/init
(gdb) run
Starting program:
/home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init
Program received signal SIGSEGV, Segmentation fault.
0x0000000010000b2c in ?? ()
(gdb) x/10i $pc
=> 0x10000b2c: stw r9,0(r9)
0x10000b30: trap
0x10000b34: .long 0x0
0x10000b38: .long 0x0
0x10000b3c: .long 0x0
0x10000b40: lis r2,4110
0x10000b44: addi r2,r2,31488
0x10000b48: mr r9,r1
0x10000b4c: rldicr r1,r1,0,59
0x10000b50: li r0,0
(gdb) p $r9
$1 = 0
(gdb) x/30x $pc - 0x30
0x10000afc: 0x38840040 0x387f0040 0xf8010040 0x48026919
0x10000b0c: 0x60000000 0xe8010040 0x7c0803a6 0x4bffff24
0x10000b1c: 0x00000000 0x01000000 0x00000180 0x39200000
0x10000b2c: 0x91290000 0x7fe00008 0x00000000 0x00000000
which matches the hex content of
http://154.223.142.244/logs/20220310/console.zhouyi.log:
[ 5.077431][ T1] init[1]: segfault (11) at 0 nip 10000b2c lr 10001024 code 1 in init[10000000+d0000]
[ 5.087167][ T1] init[1]: code: 38840040 387f0040 f8010040 48026919 60000000 e8010040 7c0803a6 4bffff24
[ 5.093987][ T1] init[1]: code: 00000000 01000000 00000180 39200000 <91290000> 7fe00008 00000000 00000000
Conclusions: there might be something wrong when packing the init into
vmlinux in your environment.
I will continue to do research on this interesting problem with you.
As written I think it’s a problem with LLVM/clang. Unfortunately, I
won’t be able to retest before next week.
Roger that, no need to hurry ;-)
Kind regards
Zhouyi