From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-02-24 03:26:08
There's code in prom_instantiate_sml() to do a "SML handover" (Stored
Measurement Log) from OF to Linux, before Linux shuts down Open
Firmware.
This involves creating a buffer to hold the SML, and creating two device
tree properties to record its base address and size. The kernel then
later reads those properties from the device tree to find the SML.
When the code was initially added in commit 4a727429abec ("PPC64: Add
support for instantiating SML from Open Firmware") the powerpc kernel
was always built big endian, so the properties were created big endian
by default.
However since then little endian support was added to powerpc, and now
the code lacks conversions to big endian when creating the properties.
This means on little endian kernels the device tree properties are
little endian, which is contrary to the device tree spec, and in
contrast to all other device tree properties.
To cope with that a workaround was added in tpm_read_log_of() to skip
the endian conversion if the properties were created via the SML
handover.
A better solution is to encode the properties as big endian as they
should be, and remove the workaround.
Typically changing the encoding of a property like this would present
problems for kexec. However the SML is not propagated across kexec, so
changing the encoding of the properties is a non-issue.
Fixes: e46e22f12b19 ("tpm: enhance read_log_of() to support Physical TPM event log")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/kernel/prom_init.c | 8 ++++++--
drivers/char/tpm/eventlog/of.c | 23 ++++-------------------
2 files changed, 10 insertions(+), 21 deletions(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-02-24 03:27:00
The TPM code in prom_init.c creates a small buffer of memory to store
the TPM's SML (Stored Measurement Log). It's communicated to Linux via
the linux,sml-base/size device tree properties of the TPM node.
When kexec'ing that buffer can be overwritten, or when kdump'ing it may
not be mapped by the second kernel. The latter can lead to a crash when
booting the second kernel such as:
tpm_ibmvtpm 71000003: CRQ initialization completed
BUG: Unable to handle kernel data access on read at 0xc00000002ffb0000
Faulting instruction address: 0xc0000000200a70e0
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d #314
Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf000005 of:SLOF,git-5b4c5a pSeries
NIP: c0000000200a70e0 LR: c0000000203dd5dc CTR: 0000000000000800
REGS: c000000024543280 TRAP: 0300 Not tainted (6.2.0-rc2-00134-g9307ce092f5d)
MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 24002280 XER: 00000006
CFAR: c0000000200a70c8 DAR: c00000002ffb0000 DSISR: 40000000 IRQMASK: 0
...
NIP memcpy_power7+0x400/0x7d0
LR kmemdup+0x5c/0x80
Call Trace:
memcpy_power7+0x274/0x7d0 (unreliable)
kmemdup+0x5c/0x80
tpm_read_log_of+0xe8/0x1b0
tpm_bios_log_setup+0x60/0x210
tpm_chip_register+0x134/0x320
tpm_ibmvtpm_probe+0x520/0x7d0
vio_bus_probe+0x9c/0x460
really_probe+0x104/0x420
__driver_probe_device+0xb0/0x170
driver_probe_device+0x58/0x180
__driver_attach+0xd8/0x250
bus_for_each_dev+0xb4/0x140
driver_attach+0x34/0x50
bus_add_driver+0x1e8/0x2d0
driver_register+0xb4/0x1c0
__vio_register_driver+0x74/0x9c
ibmvtpm_module_init+0x34/0x48
do_one_initcall+0x80/0x320
kernel_init_freeable+0x304/0x3ac
kernel_init+0x30/0x1a0
ret_from_kernel_thread+0x5c/0x64
To fix the crash, add the SML region to the usable memory areas for the
kdump kernel, so that the second kernel will map the region. To avoid
corruption of the region, add the region to the reserved memory areas,
so that the second kernel does not use the memory for something else.
Reported-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/include/asm/kexec_ranges.h | 1 +
arch/powerpc/kexec/file_load_64.c | 12 ++++++++++++
arch/powerpc/kexec/ranges.c | 20 ++++++++++++++++++++
3 files changed, 33 insertions(+)
@@ -350,6 +350,26 @@ int add_rtas_mem_range(struct crash_mem **mem_ranges)returnret;}+intadd_sml_mem_range(structcrash_mem**mem_ranges)+{+structdevice_node*dn;+intret=0;+u64base;+u32size;++// Matches the device type in tpm_ibmvtpm.c+for_each_node_by_type(dn,"IBM,vtpm"){+if(of_property_read_u64(dn,"linux,sml-base",&base)==0&&+of_property_read_u32(dn,"linux,sml-size",&size)==0){+ret=add_mem_range(mem_ranges,base,size);+if(ret)+break;+}+}++returnret;+}+/***add_opal_mem_range-AddsOPALregiontothegivenmemoryrangeslist.*@mem_ranges:Rangelisttoaddthememoryrangeto.
From: Stefan Berger <stefanb@linux.ibm.com> Date: 2023-02-27 23:09:49
On 2/23/23 22:25, Michael Ellerman wrote:
There's code in prom_instantiate_sml() to do a "SML handover" (Stored
Measurement Log) from OF to Linux, before Linux shuts down Open
Firmware.
This involves creating a buffer to hold the SML, and creating two device
tree properties to record its base address and size. The kernel then
later reads those properties from the device tree to find the SML.
When the code was initially added in commit 4a727429abec ("PPC64: Add
support for instantiating SML from Open Firmware") the powerpc kernel
was always built big endian, so the properties were created big endian
by default.
However since then little endian support was added to powerpc, and now
the code lacks conversions to big endian when creating the properties.
This means on little endian kernels the device tree properties are
little endian, which is contrary to the device tree spec, and in
contrast to all other device tree properties.
To cope with that a workaround was added in tpm_read_log_of() to skip
the endian conversion if the properties were created via the SML
handover.
A better solution is to encode the properties as big endian as they
should be, and remove the workaround.
Typically changing the encoding of a property like this would present
problems for kexec. However the SML is not propagated across kexec, so
changing the encoding of the properties is a non-issue.
Fixes: e46e22f12b19 ("tpm: enhance read_log_of() to support Physical TPM event log")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
From: Jarkko Sakkinen <jarkko@kernel.org> Date: 2023-02-28 03:11:23
On Mon, Feb 27, 2023 at 06:08:31PM -0500, Stefan Berger wrote:
On 2/23/23 22:25, Michael Ellerman wrote:
quoted
There's code in prom_instantiate_sml() to do a "SML handover" (Stored
Measurement Log) from OF to Linux, before Linux shuts down Open
Firmware.
This involves creating a buffer to hold the SML, and creating two device
tree properties to record its base address and size. The kernel then
later reads those properties from the device tree to find the SML.
When the code was initially added in commit 4a727429abec ("PPC64: Add
support for instantiating SML from Open Firmware") the powerpc kernel
was always built big endian, so the properties were created big endian
by default.
However since then little endian support was added to powerpc, and now
the code lacks conversions to big endian when creating the properties.
This means on little endian kernels the device tree properties are
little endian, which is contrary to the device tree spec, and in
contrast to all other device tree properties.
To cope with that a workaround was added in tpm_read_log_of() to skip
the endian conversion if the properties were created via the SML
handover.
A better solution is to encode the properties as big endian as they
should be, and remove the workaround.
Typically changing the encoding of a property like this would present
problems for kexec. However the SML is not propagated across kexec, so
changing the encoding of the properties is a non-issue.
Fixes: e46e22f12b19 ("tpm: enhance read_log_of() to support Physical TPM event log")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-02-28 11:22:35
Jarkko Sakkinen [off-list ref] writes:
On Mon, Feb 27, 2023 at 06:08:31PM -0500, Stefan Berger wrote:
quoted
On 2/23/23 22:25, Michael Ellerman wrote:
quoted
There's code in prom_instantiate_sml() to do a "SML handover" (Stored
Measurement Log) from OF to Linux, before Linux shuts down Open
Firmware.
This involves creating a buffer to hold the SML, and creating two device
tree properties to record its base address and size. The kernel then
later reads those properties from the device tree to find the SML.
When the code was initially added in commit 4a727429abec ("PPC64: Add
support for instantiating SML from Open Firmware") the powerpc kernel
was always built big endian, so the properties were created big endian
by default.
However since then little endian support was added to powerpc, and now
the code lacks conversions to big endian when creating the properties.
This means on little endian kernels the device tree properties are
little endian, which is contrary to the device tree spec, and in
contrast to all other device tree properties.
To cope with that a workaround was added in tpm_read_log_of() to skip
the endian conversion if the properties were created via the SML
handover.
A better solution is to encode the properties as big endian as they
should be, and remove the workaround.
Typically changing the encoding of a property like this would present
problems for kexec. However the SML is not propagated across kexec, so
changing the encoding of the properties is a non-issue.
Fixes: e46e22f12b19 ("tpm: enhance read_log_of() to support Physical TPM event log")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
2/2 does not have a fixes tag.
True. Arguably the bug goes back to the introduction of
kexec_file_load() support, although the patch won't backport that far
due to code refactoring. So that would be:
Fixes: a0458284f062 ("powerpc: Add support code for kexec_file_load()")
cheers
From: Stefan Berger <stefanb@linux.ibm.com> Date: 2023-02-28 19:19:49
On 2/23/23 22:25, Michael Ellerman wrote:
The TPM code in prom_init.c creates a small buffer of memory to store
the TPM's SML (Stored Measurement Log). It's communicated to Linux via
the linux,sml-base/size device tree properties of the TPM node.
When kexec'ing that buffer can be overwritten, or when kdump'ing it may
not be mapped by the second kernel. The latter can lead to a crash when
booting the second kernel such as:
tpm_ibmvtpm 71000003: CRQ initialization completed
BUG: Unable to handle kernel data access on read at 0xc00000002ffb0000
Faulting instruction address: 0xc0000000200a70e0
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d #314
Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf000005 of:SLOF,git-5b4c5a pSeries
NIP: c0000000200a70e0 LR: c0000000203dd5dc CTR: 0000000000000800
REGS: c000000024543280 TRAP: 0300 Not tainted (6.2.0-rc2-00134-g9307ce092f5d)
MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 24002280 XER: 00000006
CFAR: c0000000200a70c8 DAR: c00000002ffb0000 DSISR: 40000000 IRQMASK: 0
...
NIP memcpy_power7+0x400/0x7d0
LR kmemdup+0x5c/0x80
Call Trace:
memcpy_power7+0x274/0x7d0 (unreliable)
kmemdup+0x5c/0x80
tpm_read_log_of+0xe8/0x1b0
tpm_bios_log_setup+0x60/0x210
tpm_chip_register+0x134/0x320
tpm_ibmvtpm_probe+0x520/0x7d0
vio_bus_probe+0x9c/0x460
really_probe+0x104/0x420
__driver_probe_device+0xb0/0x170
driver_probe_device+0x58/0x180
__driver_attach+0xd8/0x250
bus_for_each_dev+0xb4/0x140
driver_attach+0x34/0x50
bus_add_driver+0x1e8/0x2d0
driver_register+0xb4/0x1c0
__vio_register_driver+0x74/0x9c
ibmvtpm_module_init+0x34/0x48
do_one_initcall+0x80/0x320
kernel_init_freeable+0x304/0x3ac
kernel_init+0x30/0x1a0
ret_from_kernel_thread+0x5c/0x64
I have not been able to reproduce this particular crash issue with a 6.2 kernel running on P10 PowerVM when NOT applying your patches.
For my tests I have used the following parameter with the 16GB VM: crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G
What I noticed is that the log gets corrupted when the 2 patches are applied:
After fresh boot:
cp /sys/kernel/security/tpm0/binary_bios_measurements ./
ls -l binary_bios_measurements
-r--r-----. 1 root root 10051 Feb 28 12:09 binary_bios_measurements
cp /sys/kernel/security/tpm0/binary_bios_measurements ./
ls -l binary_bios_measurements
-r--r-----. 1 root root 32 Feb 28 12:10 binary_bios_measurements
od -t x1 < binary_bios_measurements
0000000 d0 0d fe ed 00 00 77 80 00 00 00 a0 00 00 4f 4c
0000020 00 00 00 28 00 00 00 11 00 00 00 11 00 00 00 00
0000040
The contents have changed and these first 4 bytes of it are always the same once it has become this 32 byte file, otherwise they would be zero.
The address and size parameters passed around in this patch seem good, though.
Stefan
From: Jarkko Sakkinen <jarkko@kernel.org> Date: 2023-03-01 23:10:56
On Tue, Feb 28, 2023 at 10:21:36PM +1100, Michael Ellerman wrote:
Jarkko Sakkinen [off-list ref] writes:
quoted
On Mon, Feb 27, 2023 at 06:08:31PM -0500, Stefan Berger wrote:
quoted
On 2/23/23 22:25, Michael Ellerman wrote:
quoted
There's code in prom_instantiate_sml() to do a "SML handover" (Stored
Measurement Log) from OF to Linux, before Linux shuts down Open
Firmware.
This involves creating a buffer to hold the SML, and creating two device
tree properties to record its base address and size. The kernel then
later reads those properties from the device tree to find the SML.
When the code was initially added in commit 4a727429abec ("PPC64: Add
support for instantiating SML from Open Firmware") the powerpc kernel
was always built big endian, so the properties were created big endian
by default.
However since then little endian support was added to powerpc, and now
the code lacks conversions to big endian when creating the properties.
This means on little endian kernels the device tree properties are
little endian, which is contrary to the device tree spec, and in
contrast to all other device tree properties.
To cope with that a workaround was added in tpm_read_log_of() to skip
the endian conversion if the properties were created via the SML
handover.
A better solution is to encode the properties as big endian as they
should be, and remove the workaround.
Typically changing the encoding of a property like this would present
problems for kexec. However the SML is not propagated across kexec, so
changing the encoding of the properties is a non-issue.
Fixes: e46e22f12b19 ("tpm: enhance read_log_of() to support Physical TPM event log")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
2/2 does not have a fixes tag.
True. Arguably the bug goes back to the introduction of
kexec_file_load() support, although the patch won't backport that far
due to code refactoring. So that would be:
Fixes: a0458284f062 ("powerpc: Add support code for kexec_file_load()")
Hmm... IMHO, it would still make sense to document this.
BR, Jarkko
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-03-02 22:07:00
Stefan Berger [off-list ref] writes:
On 2/23/23 22:25, Michael Ellerman wrote:
quoted
The TPM code in prom_init.c creates a small buffer of memory to store
the TPM's SML (Stored Measurement Log). It's communicated to Linux via
the linux,sml-base/size device tree properties of the TPM node.
When kexec'ing that buffer can be overwritten, or when kdump'ing it may
not be mapped by the second kernel. The latter can lead to a crash when
booting the second kernel such as:
tpm_ibmvtpm 71000003: CRQ initialization completed
BUG: Unable to handle kernel data access on read at 0xc00000002ffb0000
Faulting instruction address: 0xc0000000200a70e0
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d #314
Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf000005 of:SLOF,git-5b4c5a pSeries
NIP: c0000000200a70e0 LR: c0000000203dd5dc CTR: 0000000000000800
REGS: c000000024543280 TRAP: 0300 Not tainted (6.2.0-rc2-00134-g9307ce092f5d)
MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 24002280 XER: 00000006
CFAR: c0000000200a70c8 DAR: c00000002ffb0000 DSISR: 40000000 IRQMASK: 0
...
NIP memcpy_power7+0x400/0x7d0
LR kmemdup+0x5c/0x80
Call Trace:
memcpy_power7+0x274/0x7d0 (unreliable)
kmemdup+0x5c/0x80
tpm_read_log_of+0xe8/0x1b0
tpm_bios_log_setup+0x60/0x210
tpm_chip_register+0x134/0x320
tpm_ibmvtpm_probe+0x520/0x7d0
vio_bus_probe+0x9c/0x460
really_probe+0x104/0x420
__driver_probe_device+0xb0/0x170
driver_probe_device+0x58/0x180
__driver_attach+0xd8/0x250
bus_for_each_dev+0xb4/0x140
driver_attach+0x34/0x50
bus_add_driver+0x1e8/0x2d0
driver_register+0xb4/0x1c0
__vio_register_driver+0x74/0x9c
ibmvtpm_module_init+0x34/0x48
do_one_initcall+0x80/0x320
kernel_init_freeable+0x304/0x3ac
kernel_init+0x30/0x1a0
ret_from_kernel_thread+0x5c/0x64
I have not been able to reproduce this particular crash issue with a
6.2 kernel running on P10 PowerVM when NOT applying your patches.
The crash only happens for a crashdump kernel, not a regular kexec.
And depending on where the SML is in memory, compared to where the
crashkernel is, the SML might be mapped accidentally in which case there
is no crash.
For my tests I have used the following parameter with the 16GB VM:
crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G
So you should be seeing a 2GB crashkernel reservation at 512MB.
What I noticed is that the log gets corrupted when the 2 patches are applied:
After fresh boot:
quoted
cp /sys/kernel/security/tpm0/binary_bios_measurements ./
ls -l binary_bios_measurements
-r--r-----. 1 root root 10051 Feb 28 12:09 binary_bios_measurements