(Christophe, you've seen some of this before, however there are new
results added at the end. I've Cc'd the mailing lists this time.)
On Wed, 4 Aug 2021, Stan Johnson wrote:
On 8/4/21 8:41 PM, Finn Thain wrote:
quoted
$ curl https://lore.kernel.org/lkml/9b64dde3-6ebd-b446-41d9-61e8cb0d8c39@csgroup.eu/raw
../message.mbox
ok
$ sha1 ../message.mbox
SHA1 (../message.mbox) = 436ce0adf893c46c84c54607f73c838897caeeea
quoted
On Wed, 4 Aug 2021, Christophe Leroy wrote:
quoted
Can you check if they happen at commit c16728835
$ git checkout c16728835eec
Checking out files: 100% (20728/20728), done.
Note: checking out 'c16728835eec'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at c16728835eec powerpc/32: Manage KUAP in C
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ strings vmlinux | fgrep 'Linux version'
Linux version 5.12.0-rc3-pmac-00078-geb51c431b81 (johnson@ThinkPad) (powerpc-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP Wed Aug 4 21:50:47 MDT 2021
1) PB 3400c
Hangs at boot (Mac OS screen), no serial console output
2) Wallstreet
X fails, errors ("Kernel attempted to write user page", "BUG: Unable to
handle kernel instruction fetch"), see Wallstreet_console-1.txt.
The log shows that the error "Kernel attempted to write user page
(b3399774) - exploit attempt?" happens after commit c16728835eec
("powerpc/32: Manage KUAP in C").
quoted
quoted
Can you check if they DO NOT happen at preceding commit c16728835~
$ git checkout c16728835~
Previous HEAD position was c16728835eec powerpc/32: Manage KUAP in C
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-gc9f6e8dd045
3) PB 3400c
Hangs at boot (Mac OS screen)
4) Wallstreet
X fails, errors in console log (different than test 2), see
Wallstreet_console-2.txt.
This log shows that the errors "xfce4-session[1775]: bus error (7)" and
"kernel BUG at arch/powerpc/kernel/interrupt.c:49!" happen prior to commit
c16728835eec ("powerpc/32: Manage KUAP in C").
$ git checkout 0b45359aa2df
...
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-ge06b29ce146
5) PB 3400c
Hangs at boot (Mac OS screen)
6) Wallstreet
X failed (X login succeeded, but setting up desktop failed), errors in
console log, see Wallstreet_console-3.txt.
(No need for those two tests: it's exactly the same code and almost the
same failure modes: "kernel BUG at arch/powerpc/kernel/interrupt.c:50".)
On Thu, 5 Aug 2021, Stan Johnson wrote:
On 8/5/21 12:47 AM, Finn Thain wrote:
quoted
On Wed, 4 Aug 2021, Christophe Leroy wrote:
quoted
Could you test without CONFIG_PPC_KUAP
...
$ git checkout c16728835eec
...
HEAD is now at c16728835eec powerpc/32: Manage KUAP in C
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ scripts/config -d CONFIG_PPC_KUAP
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
# CONFIG_PPC_KUAP is not set
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
7) PB 3400c
Hangs at boot (Mac OS screen)
8) Wallstreet
Everything works, no errors (see Wallstreet_console-4.txt).
That would seem to implicate CONFIG_PPC_KUAP itself. (Note that all builds
up until this one have CONFIG_PPC_KUAP=y.)
quoted
quoted
Could you test with CONFIG_PPC_KUAP and CONFIG_PPC_KUAP_DEBUG
...
$scripts/config -e CONFIG_PPC_KUAP
$ scripts/config -e CONFIG_PPC_KUAP_DEBUG
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
CONFIG_PPC_KUAP=y
CONFIG_PPC_KUAP_DEBUG=y
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
9) PB 3400c
Hangs at boot (Mac OS screen)
10) Wallstreet
X failed at first login, worked at second login, one error in console
log ("BUG: Unable to handle kernel instruction fetch"), see
Wallstreet_console-5.txt.
One might expect to see "Kernel attempted to write user page (b3399774) -
exploit attempt?" again here (see c16728835eec build above) but instead
this log says "Oops: Kernel access of bad area, sig: 11".
BTW, this procedure could be made simpler and easier if I pushed git
branches to a public repo for Stan to build, which included Christophe's
fix plus hard-wired Kconfig changes. That way, the .config file could be
held constant and the commit hash in the serial console log would be more
meaningful.
+nicholas piggin for the C interrupt stuff
Le 06/08/2021 à 03:06, Finn Thain a écrit :
(Christophe, you've seen some of this before, however there are new
results added at the end. I've Cc'd the mailing lists this time.)
On Wed, 4 Aug 2021, Stan Johnson wrote:
quoted
On 8/4/21 8:41 PM, Finn Thain wrote:
quoted
$ curl https://lore.kernel.org/lkml/9b64dde3-6ebd-b446-41d9-61e8cb0d8c39@csgroup.eu/raw
../message.mbox
ok
$ sha1 ../message.mbox
SHA1 (../message.mbox) = 436ce0adf893c46c84c54607f73c838897caeeea
quoted
On Wed, 4 Aug 2021, Christophe Leroy wrote:
quoted
Can you check if they happen at commit c16728835
$ git checkout c16728835eec
Checking out files: 100% (20728/20728), done.
Note: checking out 'c16728835eec'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at c16728835eec powerpc/32: Manage KUAP in C
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ strings vmlinux | fgrep 'Linux version'
Linux version 5.12.0-rc3-pmac-00078-geb51c431b81 (johnson@ThinkPad) (powerpc-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP Wed Aug 4 21:50:47 MDT 2021
1) PB 3400c
Hangs at boot (Mac OS screen), no serial console output
2) Wallstreet
X fails, errors ("Kernel attempted to write user page", "BUG: Unable to
handle kernel instruction fetch"), see Wallstreet_console-1.txt.
The log shows that the error "Kernel attempted to write user page
(b3399774) - exploit attempt?" happens after commit c16728835eec
("powerpc/32: Manage KUAP in C").
I think I found a possible cause for this. After the above patch, locking KUAP on interrupt is done
in interrupt_enter_prepare(). But in case of NMI interrupt, that function is not called. That means
that when leaving interrupt through interrupt_exit_kernel_prepare(), the supposedly saved previous
KUAP status is garbage.
An easy way to fix that is to add missing stuff in interrupt_nmi_enter_prepare(), I'll do that at
least for testing, but at the end it is not so easy, because of booke32 and 40x.
The problem on booke32 and 40x is that the "critical interrupts" exit goes through interrupt_return
when they happened in user mode and bypass interrupt_return when they happened in kernel mode. So it
is not easy to manage.
quoted
quoted
quoted
Can you check if they DO NOT happen at preceding commit c16728835~
$ git checkout c16728835~
Previous HEAD position was c16728835eec powerpc/32: Manage KUAP in C
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-gc9f6e8dd045
3) PB 3400c
Hangs at boot (Mac OS screen)
4) Wallstreet
X fails, errors in console log (different than test 2), see
Wallstreet_console-2.txt.
This log shows that the errors "xfce4-session[1775]: bus error (7)" and
"kernel BUG at arch/powerpc/kernel/interrupt.c:49!" happen prior to commit
c16728835eec ("powerpc/32: Manage KUAP in C").
As mentionned by Nic, this is due to r11 being cloberred. For the time being the only r11 clobber
identified is the one I have provided a fix for. I'm wondering whether it was applied for all
further tests or not.
quoted
$ git checkout 0b45359aa2df
...
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-ge06b29ce146
5) PB 3400c
Hangs at boot (Mac OS screen)
6) Wallstreet
X failed (X login succeeded, but setting up desktop failed), errors in
console log, see Wallstreet_console-3.txt.
(No need for those two tests: it's exactly the same code and almost the
same failure modes: "kernel BUG at arch/powerpc/kernel/interrupt.c:50".)
On Thu, 5 Aug 2021, Stan Johnson wrote:
quoted
On 8/5/21 12:47 AM, Finn Thain wrote:
quoted
On Wed, 4 Aug 2021, Christophe Leroy wrote:
quoted
Could you test without CONFIG_PPC_KUAP
...
$ git checkout c16728835eec
...
HEAD is now at c16728835eec powerpc/32: Manage KUAP in C
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ scripts/config -d CONFIG_PPC_KUAP
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
# CONFIG_PPC_KUAP is not set
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
7) PB 3400c
Hangs at boot (Mac OS screen)
8) Wallstreet
Everything works, no errors (see Wallstreet_console-4.txt).
That would seem to implicate CONFIG_PPC_KUAP itself. (Note that all builds
up until this one have CONFIG_PPC_KUAP=y.)
Yes I believe so, see at the begining of this mail.
quoted
quoted
quoted
Could you test with CONFIG_PPC_KUAP and CONFIG_PPC_KUAP_DEBUG
...
$scripts/config -e CONFIG_PPC_KUAP
$ scripts/config -e CONFIG_PPC_KUAP_DEBUG
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
CONFIG_PPC_KUAP=y
CONFIG_PPC_KUAP_DEBUG=y
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
9) PB 3400c
Hangs at boot (Mac OS screen)
10) Wallstreet
X failed at first login, worked at second login, one error in console
log ("BUG: Unable to handle kernel instruction fetch"), see
Wallstreet_console-5.txt.
One might expect to see "Kernel attempted to write user page (b3399774) -
exploit attempt?" again here (see c16728835eec build above) but instead
this log says "Oops: Kernel access of bad area, sig: 11".
Maybe the test should be done a second time. As r11 is garbage it may or may not be a user address.
If it is a user address the we get "Kernel attempted to write user page". If it is a random kernel
address, we likely get "Kernel access of bad area" instead.
BTW, this procedure could be made simpler and easier if I pushed git
branches to a public repo for Stan to build, which included Christophe's
fix plus hard-wired Kconfig changes. That way, the .config file could be
held constant and the commit hash in the serial console log would be more
meaningful.
I like the idea, I think I'm going to provide testing fixes through a git repo, that will for sure
make things easier.
Thanks
Christophe
On Fri, 6 Aug 2021, Christophe Leroy wrote:
quoted
quoted
quoted
quoted
Can you check if they DO NOT happen at preceding commit c16728835~
$ git checkout c16728835~
Previous HEAD position was c16728835eec powerpc/32: Manage KUAP in C
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap
save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be
lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean
olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-gc9f6e8dd045
3) PB 3400c
Hangs at boot (Mac OS screen)
4) Wallstreet
X fails, errors in console log (different than test 2), see
Wallstreet_console-2.txt.
This log shows that the errors "xfce4-session[1775]: bus error (7)" and
"kernel BUG at arch/powerpc/kernel/interrupt.c:49!" happen prior to commit
c16728835eec ("powerpc/32: Manage KUAP in C").
As mentionned by Nic, this is due to r11 being cloberred. For the time being
the only r11 clobber identified is the one I have provided a fix for. I'm
wondering whether it was applied for all further tests or not.
Your fix was applied to this build with "git am ../message.mbox".
...
quoted
quoted
quoted
quoted
Could you test with CONFIG_PPC_KUAP and CONFIG_PPC_KUAP_DEBUG
...
$scripts/config -e CONFIG_PPC_KUAP
$ scripts/config -e CONFIG_PPC_KUAP_DEBUG
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean
olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
CONFIG_PPC_KUAP=y
CONFIG_PPC_KUAP_DEBUG=y
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
9) PB 3400c
Hangs at boot (Mac OS screen)
10) Wallstreet
X failed at first login, worked at second login, one error in console
log ("BUG: Unable to handle kernel instruction fetch"), see
Wallstreet_console-5.txt.
One might expect to see "Kernel attempted to write user page (b3399774) -
exploit attempt?" again here (see c16728835eec build above) but instead
this log says "Oops: Kernel access of bad area, sig: 11".
Maybe the test should be done a second time. As r11 is garbage it may or
may not be a user address. If it is a user address the we get "Kernel
attempted to write user page". If it is a random kernel address, we
likely get "Kernel access of bad area" instead.
Your fix was applied here also.
Le 06/08/2021 à 11:43, Finn Thain a écrit :
On Fri, 6 Aug 2021, Christophe Leroy wrote:
quoted
quoted
quoted
quoted
quoted
Can you check if they DO NOT happen at preceding commit c16728835~
$ git checkout c16728835~
Previous HEAD position was c16728835eec powerpc/32: Manage KUAP in C
HEAD is now at 0b45359aa2df powerpc/8xx: Create C version of kuap
save/restore/check helpers
$ git am ../message.mbox
warning: Patch sent with format=flowed; space at the end of lines might be
lost.
Applying: powerpc/32: Dismantle EXC_XFER_STD/LITE/TEMPLATE
$ cp ../dot-config-powermac-5.13 .config
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean
olddefconfig vmlinux
Linux version 5.12.0-rc3-pmac-00077-gc9f6e8dd045
3) PB 3400c
Hangs at boot (Mac OS screen)
4) Wallstreet
X fails, errors in console log (different than test 2), see
Wallstreet_console-2.txt.
This log shows that the errors "xfce4-session[1775]: bus error (7)" and
"kernel BUG at arch/powerpc/kernel/interrupt.c:49!" happen prior to commit
c16728835eec ("powerpc/32: Manage KUAP in C").
As mentionned by Nic, this is due to r11 being cloberred. For the time being
the only r11 clobber identified is the one I have provided a fix for. I'm
wondering whether it was applied for all further tests or not.
Your fix was applied to this build with "git am ../message.mbox".
Ok good.
quoted
...
quoted
quoted
quoted
quoted
Could you test with CONFIG_PPC_KUAP and CONFIG_PPC_KUAP_DEBUG
...
$scripts/config -e CONFIG_PPC_KUAP
$ scripts/config -e CONFIG_PPC_KUAP_DEBUG
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean
olddefconfig vmlinux
$ grep CONFIG_PPC_KUAP .config
CONFIG_PPC_KUAP=y
CONFIG_PPC_KUAP_DEBUG=y
Linux version 5.12.0-rc3-pmac-00078-g5cac2bc3752
9) PB 3400c
Hangs at boot (Mac OS screen)
10) Wallstreet
X failed at first login, worked at second login, one error in console
log ("BUG: Unable to handle kernel instruction fetch"), see
Wallstreet_console-5.txt.
One might expect to see "Kernel attempted to write user page (b3399774) -
exploit attempt?" again here (see c16728835eec build above) but instead
this log says "Oops: Kernel access of bad area, sig: 11".
Maybe the test should be done a second time. As r11 is garbage it may or
may not be a user address. If it is a user address the we get "Kernel
attempted to write user page". If it is a random kernel address, we
likely get "Kernel access of bad area" instead.
Your fix was applied here also.
Anyway, it would be worth trying to boot a few times more with the same kernel, because as I said
the value is random, so it may or may not hit userspace, hence the possible difference of message,
either "Kernel attempted to write user page" or "Kernel access of bad area" depending on whether the
address is a user address or not.
I have cooked a tentative fix for that KUAP stuff.
Could you try the branch 'bugtest' at https://github.com/chleroy/linux.git
Thanks
Christophe
On Fri, 6 Aug 2021, Christophe Leroy wrote:
I have cooked a tentative fix for that KUAP stuff.
Could you try the branch 'bugtest' at https://github.com/chleroy/linux.git
Thanks, Christophe.
Stan, please test the following build.
$ git remote add chleroy-linux https://github.com/chleroy/linux.git -f -t bugtest
...
$ git checkout chleroy-linux/bugtest
HEAD is now at 63e3756d1bdf powerpc/interrupts: Also perform KUAP/KUEP lock and usertime accounting on NMI
$ cp ../dot-config-powermac-5.13 .config
$ scripts/config -e CONFIG_PPC_KUAP -e CONFIG_PPC_KUAP_DEBUG -e CONFIG_VMAP_STACK
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ egrep "CONFIG_PPC_KUAP|CONFIG_VMAP_STACK" .config
$ strings vmlinux |grep "Linux version"
If that kernel produces errors, I'd try a second build as well:
$ scripts/config -e CONFIG_PPC_KUAP -e CONFIG_PPC_KUAP_DEBUG -d CONFIG_VMAP_STACK
$ make ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- -j4 clean olddefconfig vmlinux
$ egrep "CONFIG_PPC_KUAP|CONFIG_VMAP_STACK" .config
$ strings vmlinux |grep "Linux version"
Please boot using the same kernel parameters as last time and capture the
serial console logs. In case we're still dealing with intermittent bugs it
might be necessary to repeat these tests so I suggest you retain the
vmlinux files.