From: Rafael J. Wysocki <hidden> Date: 2016-08-08 13:26:15
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
Thanks,
Rafael
---
arch/x86/include/asm/init.h | 4 ++--
arch/x86/mm/ident_map.c | 19 +++++++++++--------
arch/x86/power/hibernate_64.c | 2 +-
3 files changed, 14 insertions(+), 11 deletions(-)
Index: linux-pm/arch/x86/include/asm/init.h
===================================================================
On Mon, Aug 08, 2016 at 03:31:31PM +0200, Rafael J. Wysocki wrote:
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Reported-by: Borislav Petkov <redacted>
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
Is that the only patch which needs to be tested? Ontop of which tree?
CONFIG_RANDOMIZE_MEMORY blew up s2d on my laptop here so I'll run it
once I have the required info from you :)
Thanks.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-08 13:54:53
On Mon, Aug 8, 2016 at 3:40 PM, Borislav Petkov [off-list ref] wrote:
On Mon, Aug 08, 2016 at 03:31:31PM +0200, Rafael J. Wysocki wrote:
quoted
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Reported-by: Borislav Petkov <redacted>
quoted
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
Is that the only patch which needs to be tested? Ontop of which tree?
That should be the only one on top of plain 4.8-rc1.
If it doesn't help, we need more work to do. :-)
CONFIG_RANDOMIZE_MEMORY blew up s2d on my laptop here so I'll run it
once I have the required info from you :)
From: Thomas Garnier <hidden> Date: 2016-08-08 18:05:53
On Mon, Aug 8, 2016 at 6:54 AM, Rafael J. Wysocki [off-list ref] wrote:
On Mon, Aug 8, 2016 at 3:40 PM, Borislav Petkov [off-list ref] wrote:
quoted
On Mon, Aug 08, 2016 at 03:31:31PM +0200, Rafael J. Wysocki wrote:
quoted
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Reported-by: Borislav Petkov <redacted>
quoted
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
I tested it on my setup couple times. Worked well.
quoted
Is that the only patch which needs to be tested? Ontop of which tree?
That should be the only one on top of plain 4.8-rc1.
If it doesn't help, we need more work to do. :-)
quoted
CONFIG_RANDOMIZE_MEMORY blew up s2d on my laptop here so I'll run it
once I have the required info from you :)
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-08 20:01:45
On Mon, Aug 8, 2016 at 8:00 PM, Thomas Garnier [off-list ref] wrote:
On Mon, Aug 8, 2016 at 6:54 AM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Mon, Aug 8, 2016 at 3:40 PM, Borislav Petkov [off-list ref] wrote:
quoted
On Mon, Aug 08, 2016 at 03:31:31PM +0200, Rafael J. Wysocki wrote:
quoted
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Reported-by: Borislav Petkov <redacted>
quoted
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
I tested it on my setup couple times. Worked well.
On Mon, Aug 08, 2016 at 03:54:48PM +0200, Rafael J. Wysocki wrote:
That should be the only one on top of plain 4.8-rc1.
If it doesn't help, we need more work to do. :-)
Yes, we do.
The machine triple-faults *after* reading up the hibernation image.
It hits 100%, then tries to switch to the boot kernel and BOOM, BIOS
screen.
I'm attaching the .config in case you want to reproduce it. The machine
is an IVB thinkpad x230.
Thanks.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
Unfortunately this applied on top of -rc1 still doesn't solve the reboot
after reading hibernation image (I'd guess due to triple fault) with
CONFIG_RANDOMIZE_MEMORY=y on my system.
With CONFIG_RANDOMIZE_MEMORY=n, the system resumes correctly.
--
Jiri Kosina
SUSE Labs
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-09 11:47:38
On Tue, Aug 9, 2016 at 9:02 AM, Borislav Petkov [off-list ref] wrote:
On Mon, Aug 08, 2016 at 03:54:48PM +0200, Rafael J. Wysocki wrote:
quoted
That should be the only one on top of plain 4.8-rc1.
If it doesn't help, we need more work to do. :-)
Yes, we do.
The machine triple-faults *after* reading up the hibernation image.
It hits 100%, then tries to switch to the boot kernel and BOOM, BIOS
screen.
I'm attaching the .config in case you want to reproduce it. The machine
is an IVB thinkpad x230.
Yes, I'm going to try to reproduce it.
I'm wondering what the difference between your .config and the Thomas'
.config is, as he has CONFIG_RANDOMIZE_MEMORY=y set too.
Thomas, can you attach your .config, please?
Thanks,
Rafael
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-09 11:56:14
On Tue, Aug 9, 2016 at 11:23 AM, Jiri Kosina [off-list ref] wrote:
On Mon, 8 Aug 2016, Rafael J. Wysocki wrote:
quoted
From: Rafael J. Wysocki <redacted>
The low-level resume-from-hibernation code on x86-64 uses
kernel_ident_mapping_init() to create the temoprary identity mapping,
but that function assumes that the offset between kernel virtual
addresses and physical addresses is aligned on the PGD level.
However, with a randomized identity mapping base, it may be aligned
on the PUD level and if that happens, the temporary identity mapping
created by set_up_temporary_mappings() will not reflect the actual
kernel identity mapping and the image restoration will fail as a
result (leading to a kernel panic most of the time).
To fix this problem, rework kernel_ident_mapping_init() to support
unaligned offsets between KVA and PA up to the PMD level and make
set_up_temporary_mappings() use it as approprtiate.
Reported-by: Thomas Garnier <redacted>
Suggested-by: Yinghai Lu <yinghai@kernel.org>
Signed-off-by: Rafael J. Wysocki <redacted>
Acked-by: Yinghai Lu <yinghai@kernel.org>
---
This is sort of urgent, because hibernation doesn't work with KASLR on x86-64
in 4.8-rc1 AFAICS and this should make them work together again.
Unless anyone sees any problems with it, I'll queue it up for 4.8-rc2.
Thomas, would it be possible to test it with KASLR enabled, please?
Unfortunately this applied on top of -rc1 still doesn't solve the reboot
after reading hibernation image (I'd guess due to triple fault) with
CONFIG_RANDOMIZE_MEMORY=y on my system.
With CONFIG_RANDOMIZE_MEMORY=n, the system resumes correctly.
Here's a list of commits from Thomas that are related to memory randomization.
210e7a43fa90 mm: SLUB freelist randomization
7c00fce98c3e mm: reorganize SLAB freelist randomization
4ff5308744f5 x86/mm: Do not reference phys addr beyond kernel
90397a417796 x86/mm: Add memory hotplug support for KASLR memory randomization
a95ae27c2ee1 x86/mm: Enable KASLR for vmalloc memory regions
021182e52fe0 x86/mm: Enable KASLR for physical mapping memory regions
0483e1fa6e09 x86/mm: Implement ASLR for kernel memory regions
b234e8a09003 x86/mm: Separate variable for trampoline PGD
faa379332f3c x86/mm: Add PUD VA support for physical mapping
59b3d0206d74 x86/mm: Update physical mapping variable names
d899a7d146a2 x86/mm: Refactor KASLR entropy functions
I wonder if it is viable to revert them one by one top-to-bottom and
see which one of them causes things to fail?
Thanks,
Rafael
Here's a list of commits from Thomas that are related to memory randomization.
210e7a43fa90 mm: SLUB freelist randomization
7c00fce98c3e mm: reorganize SLAB freelist randomization
4ff5308744f5 x86/mm: Do not reference phys addr beyond kernel
90397a417796 x86/mm: Add memory hotplug support for KASLR memory randomization
a95ae27c2ee1 x86/mm: Enable KASLR for vmalloc memory regions
021182e52fe0 x86/mm: Enable KASLR for physical mapping memory regions
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
--
Jiri Kosina
SUSE Labs
210e7a43fa90 mm: SLUB freelist randomization
7c00fce98c3e mm: reorganize SLAB freelist randomization
4ff5308744f5 x86/mm: Do not reference phys addr beyond kernel
90397a417796 x86/mm: Add memory hotplug support for KASLR memory randomization
a95ae27c2ee1 x86/mm: Enable KASLR for vmalloc memory regions
021182e52fe0 x86/mm: Enable KASLR for physical mapping memory regions
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
--
Jiri Kosina
SUSE Labs
From: Thomas Garnier <hidden> Date: 2016-08-09 15:00:36
On Tue, Aug 9, 2016 at 6:30 AM, Jiri Kosina [off-list ref] wrote:
On Tue, 9 Aug 2016, Jiri Kosina wrote:
quoted
quoted
210e7a43fa90 mm: SLUB freelist randomization
7c00fce98c3e mm: reorganize SLAB freelist randomization
4ff5308744f5 x86/mm: Do not reference phys addr beyond kernel
90397a417796 x86/mm: Add memory hotplug support for KASLR memory randomization
a95ae27c2ee1 x86/mm: Enable KASLR for vmalloc memory regions
021182e52fe0 x86/mm: Enable KASLR for physical mapping memory regions
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
My .config is attached. It is basically defconfig (x86_64) + kvmconfig
plus the following:
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_RANDOMIZE_MEMORY=y
CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
You mean it is something different from the previous KASLR bugs we saw?
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
My .config is attached. It is basically defconfig (x86_64) + kvmconfig
plus the following:
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_RANDOMIZE_MEMORY=y
CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
The config I am reproducing the bug with (on thinkpad x200s) can be found
at
http://www.jikos.cz/jikos/junk/.config
Either later today or tomorrow I could test with the same physical start
and align values you're using to see whether that'd make any difference.
quoted
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
You mean it is something different from the previous KASLR bugs we saw?
No, I just wanted to explicitly point out that "x86/power/64: Always
create temporary identity mapping correctly" is not a fix for this issue.
--
Jiri Kosina
SUSE Labs
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-09 16:18:08
On Tue, Aug 9, 2016 at 5:05 PM, Jiri Kosina [off-list ref] wrote:
On Tue, 9 Aug 2016, Thomas Garnier wrote:
quoted
quoted
quoted
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
My .config is attached. It is basically defconfig (x86_64) + kvmconfig
plus the following:
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_RANDOMIZE_MEMORY=y
CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
The config I am reproducing the bug with (on thinkpad x200s) can be found
at
http://www.jikos.cz/jikos/junk/.config
Either later today or tomorrow I could test with the same physical start
and align values you're using to see whether that'd make any difference.
quoted
quoted
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
You mean it is something different from the previous KASLR bugs we saw?
No, I just wanted to explicitly point out that "x86/power/64: Always
create temporary identity mapping correctly" is not a fix for this issue.
It is better to say that the $subject patch is not sufficient to fix
it, because I'm quite confident that it is necessary for that. :-)
Without the $subject patch kernel_ident_mapping_init() makes
assumptions that simply are not met in the randomized identity mapping
base case. Moreover, hibernation works for Thomas with $subject patch
applied, but it doesn't without it.
So there is something else that we are missing.
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
But is this possible even? Thomas?
Anyway, I'll try to reproduce this issue later today.
Thanks,
Rafael
From: Thomas Garnier <hidden> Date: 2016-08-09 16:29:03
On Tue, Aug 9, 2016 at 9:18 AM, Rafael J. Wysocki [off-list ref] wrote:
On Tue, Aug 9, 2016 at 5:05 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Tue, 9 Aug 2016, Thomas Garnier wrote:
quoted
quoted
quoted
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
My .config is attached. It is basically defconfig (x86_64) + kvmconfig
plus the following:
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_RANDOMIZE_MEMORY=y
CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
The config I am reproducing the bug with (on thinkpad x200s) can be found
at
http://www.jikos.cz/jikos/junk/.config
Either later today or tomorrow I could test with the same physical start
and align values you're using to see whether that'd make any difference.
quoted
quoted
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
You mean it is something different from the previous KASLR bugs we saw?
No, I just wanted to explicitly point out that "x86/power/64: Always
create temporary identity mapping correctly" is not a fix for this issue.
It is better to say that the $subject patch is not sufficient to fix
it, because I'm quite confident that it is necessary for that. :-)
Without the $subject patch kernel_ident_mapping_init() makes
assumptions that simply are not met in the randomized identity mapping
base case. Moreover, hibernation works for Thomas with $subject patch
applied, but it doesn't without it.
So there is something else that we are missing.
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
But is this possible even? Thomas?
I had a similar theory before when I was investigating the original
crash. How is it avoided even without KASLR?
Given the space for the physical memory mapping, I doubt this issue
would happen all the time though.
Anyway, I'll try to reproduce this issue later today.
Thanks,
Rafael
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
So this made me to actually test a scenario where I'd suspend a kernel
that's known-broken (i.e. contains 021182e52fe), and then have it resumed
by a kernel that has 021182e52fe reverted. It resumed successfully.
Just a datapoint.
--
Jiri Kosina
SUSE Labs
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-09 20:50:22
On Tue, Aug 9, 2016 at 6:27 PM, Thomas Garnier [off-list ref] wrote:
On Tue, Aug 9, 2016 at 9:18 AM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Tue, Aug 9, 2016 at 5:05 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Tue, 9 Aug 2016, Thomas Garnier wrote:
quoted
quoted
quoted
Okay, I did one-by-one reverts, and the one above, i.e.
commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
Author: Thomas Garnier [off-list ref]
Date: Tue Jun 21 17:47:03 2016 -0700
x86/mm: Enable KASLR for physical mapping memory regions
is the one that is the culprit on my machine. With this one reverted,
resume hibernation doesn't reboot (tripple fault?), but proceeds
succesfully.
My .config is attached. It is basically defconfig (x86_64) + kvmconfig
plus the following:
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
CONFIG_X86_NEED_RELOCS=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_RANDOMIZE_MEMORY=y
CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa
CONFIG_X86_PTDUMP_CORE=y
CONFIG_X86_PTDUMP=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_KGDB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
The config I am reproducing the bug with (on thinkpad x200s) can be found
at
http://www.jikos.cz/jikos/junk/.config
Either later today or tomorrow I could test with the same physical start
and align values you're using to see whether that'd make any difference.
quoted
quoted
As discussed with Rafael privately, I also tried this very patch
(x86/power/64: Always create temporary identity mapping correctly) on top
of the reverted revert of 021182e52fe01c1f7b1 (see the full log below),
but such kernel triple faults on resume as well.
87c38d2 x86/power/64: Always create temporary identity mapping correctly
3cb504a Revert "Revert "x86/mm: Enable KASLR for physical mapping memory regions""
758850d Revert "x86/mm: Enable KASLR for physical mapping memory regions"
4a02dfb Revert "x86/mm: Enable KASLR for vmalloc memory regions"
037863f Revert "x86/mm: Add memory hotplug support for KASLR memory randomization"
3416a21 Revert "x86/mm: Do not reference phys addr beyond kernel"
69227be Revert "mm: reorganize SLAB freelist randomization"
a1d8d71 Revert "mm: SLUB freelist randomization"
IOW, 021182e52f introduces a bug for which there is no existing fix yet.
You mean it is something different from the previous KASLR bugs we saw?
No, I just wanted to explicitly point out that "x86/power/64: Always
create temporary identity mapping correctly" is not a fix for this issue.
It is better to say that the $subject patch is not sufficient to fix
it, because I'm quite confident that it is necessary for that. :-)
Without the $subject patch kernel_ident_mapping_init() makes
assumptions that simply are not met in the randomized identity mapping
base case. Moreover, hibernation works for Thomas with $subject patch
applied, but it doesn't without it.
So there is something else that we are missing.
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
But is this possible even? Thomas?
I had a similar theory before when I was investigating the original
crash. How is it avoided even without KASLR?
It doesn't have to be actively avoided then. restore_jump_address is
a kernel text address and if __PAGE_OFFSET is the same for both the
restore and image kernels, it is guaranteed to be above the identity
mapping in both of them.
If the base of the identity mapping is randomized in both of them,
though, that may not be guaranteed any more.
Given the space for the physical memory mapping, I doubt this issue
would happen all the time though.
It should not, but it's not impossible for it to happen every time, at
least in a small number of attempts.
Thanks,
Rafael
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-09 21:23:38
On Tue, Aug 9, 2016 at 10:02 PM, Jiri Kosina [off-list ref] wrote:
On Tue, 9 Aug 2016, Rafael J. Wysocki wrote:
quoted
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
So this made me to actually test a scenario where I'd suspend a kernel
that's known-broken (i.e. contains 021182e52fe), and then have it resumed
by a kernel that has 021182e52fe reverted. It resumed successfully.
Just a datapoint.
That indicates the problem is somewhere in the restore kernel and no
surprises there.
I am able to reproduce the original problem (a triple fault on resume
with CONFIG_RANDOMIZE_MEMORY set) without the $subject patch, but the
patch fixes it for me.
Question is why it is not sufficient for you and Boris and the above
theory is about the only one I can come up with ATM.
I'm going to compare the configs etc, but I guess I just end up
writing a patch to test that theory unless someone has any other idea
in the meantime.
Thanks,
Rafael
From: Rafael J. Wysocki <hidden> Date: 2016-08-10 00:16:02
On Tuesday, August 09, 2016 11:23:31 PM Rafael J. Wysocki wrote:
On Tue, Aug 9, 2016 at 10:02 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Tue, 9 Aug 2016, Rafael J. Wysocki wrote:
quoted
I have a murky suspicion, but it is really weird. Namely, what if
restore_jump_address in set_up_temporary_text_mapping() happens to be
covered by the restore kernel's identity mapping? Then, the image
kernel's entry point may get overwritten by something else in
core_restore_code().
So this made me to actually test a scenario where I'd suspend a kernel
that's known-broken (i.e. contains 021182e52fe), and then have it resumed
by a kernel that has 021182e52fe reverted. It resumed successfully.
Just a datapoint.
That indicates the problem is somewhere in the restore kernel and no
surprises there.
I am able to reproduce the original problem (a triple fault on resume
with CONFIG_RANDOMIZE_MEMORY set) without the $subject patch, but the
patch fixes it for me.
Question is why it is not sufficient for you and Boris and the above
theory is about the only one I can come up with ATM.
I'm going to compare the configs etc, but I guess I just end up
writing a patch to test that theory unless someone has any other idea
in the meantime.
For the lack of better ideas, below is a patch to try.
It avoids the possible issue with the restore kernel's identity mapping overlap
with restore_jump_address by creating special super-simple page tables just
for the final jump to the image kernel.
It is on top of the $subject patch. My test box still works with this applied,
but then it worked without it as well.
If it doesn't help, the identity mapping created by set_up_temporary_mappings()
is still not adequate for some reason most likely and we'll need to find out
why.
Thanks,
Rafael
---
arch/x86/power/hibernate_64.c | 40 +++++++++++++++++++++++++++++++-------
arch/x86/power/hibernate_asm_64.S | 10 +++++++++
2 files changed, 43 insertions(+), 7 deletions(-)
Index: linux-pm/arch/x86/power/hibernate_64.c
===================================================================
@@ -38,14 +38,20 @@ unsigned long jump_address_phys;unsignedlongrestore_cr3__visible;unsignedlongtemp_level4_pgt__visible;+unsignedlongjump_level4_pgt__visible;unsignedlongrelocated_restore_code__visible;-staticintset_up_temporary_text_mapping(pgd_t*pgd)+staticintset_up_temporary_text_mapping(void){+pgd_t*pgd;pmd_t*pmd;pud_t*pud;+pgd=(pgd_t*)get_safe_page(GFP_ATOMIC);+if(!pgd)+return-ENOMEM;+/**Thenewmappingonlyhastocoverthepagecontainingtheimage*kernel'sentrypoint(jump_address_phys),becausetheswitchoverto
@@ -74,6 +80,23 @@ static int set_up_temporary_text_mappingset_pgd(pgd+pgd_index(restore_jump_address),__pgd(__pa(pud)|_KERNPG_TABLE));+pud=(pud_t*)get_safe_page(GFP_ATOMIC);+if(!pud)+return-ENOMEM;++pmd=(pmd_t*)get_safe_page(GFP_ATOMIC);+if(!pmd)+return-ENOMEM;++set_pmd(pmd+pmd_index(relocated_restore_code),+__pmd((__pa(relocated_restore_code)&PMD_MASK)|__PAGE_KERNEL_LARGE_EXEC));+set_pud(pud+pud_index(relocated_restore_code),+__pud(__pa(pmd)|_KERNPG_TABLE));+set_pgd(pgd+pgd_index(relocated_restore_code),+__pgd(__pa(pud)|_KERNPG_TABLE));++jump_level4_pgt=__pa(pgd);+return0;}
@@ -98,11 +121,6 @@ static int set_up_temporary_mappings(voiif(!pgd)return-ENOMEM;-/* Prepare a temporary mapping for the kernel text */-result=set_up_temporary_text_mapping(pgd);-if(result)-returnresult;-/* Set up the direct mapping from scratch */for(i=0;i<nr_pfn_mapped;i++){mstart=pfn_mapped[i].start<<PAGE_SHIFT;
@@ -122,7 +140,10 @@ static int relocate_restore_code(void)pgd_t*pgd;pud_t*pud;-relocated_restore_code=get_safe_page(GFP_ATOMIC);+do+relocated_restore_code=get_safe_page(GFP_ATOMIC);+while((relocated_restore_code&PMD_MASK)==(restore_jump_address&PMD_MASK));+if(!relocated_restore_code)return-ENOMEM;
@@ -162,6 +183,11 @@ int swsusp_arch_resume(void)if(error)returnerror;+/* Prepare a temporary mapping for the jump to the image kernel */+error=set_up_temporary_text_mapping();+if(error)+returnerror;+restore_image();return0;}
From: Thomas Garnier <hidden> Date: 2016-08-10 18:33:01
On Wed, Aug 10, 2016 at 6:18 AM, Jiri Kosina [off-list ref] wrote:
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
The last patch I sent had a problem, because if restore_jump_address really
overlapped with the identity mapping of the restore kernel, it might share
PGD or PUD entries with that mapping and that should have been taken into
account.
Here goes an update. Again, this works on my test machine, but then the
previous version worked on it too ...
Unfortunately still exactly the same symptoms during resume even with this
one.
What type of machines are you testing it on? What is the memory size?
Processor generation?
Ok, I want to know if the problem is the PUD alignment or the change
of PAGE_OFFSET based all together. Can you test the following change?
(on top of everything else with KASLR enabled). It will randomize the
memory sections only on PGD level.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
The last patch I sent had a problem, because if restore_jump_address really
overlapped with the identity mapping of the restore kernel, it might share
PGD or PUD entries with that mapping and that should have been taken into
account.
Here goes an update. Again, this works on my test machine, but then the
previous version worked on it too ...
Unfortunately still exactly the same symptoms during resume even with this
one.
Thanks,
--
Jiri Kosina
SUSE Labs
For the lack of better ideas, below is a patch to try.
It avoids the possible issue with the restore kernel's identity mapping overlap
with restore_jump_address by creating special super-simple page tables just
for the final jump to the image kernel.
It is on top of the $subject patch. My test box still works with this applied,
but then it worked without it as well.
If it doesn't help, the identity mapping created by set_up_temporary_mappings()
is still not adequate for some reason most likely and we'll need to find out
why.
Unfortunately, still with $subject patch + this one, triple fault and
reboot after reading the hibernation image.
Due to being slightly out of ideas currently, I'll play a little bit more
with the relocation offsets to see whether that makes any difference.
--
Jiri Kosina
SUSE Labs
From: Rafael J. Wysocki <hidden> Date: 2016-08-10 20:00:39
On Wednesday, August 10, 2016 09:50:15 AM Jiri Kosina wrote:
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
For the lack of better ideas, below is a patch to try.
It avoids the possible issue with the restore kernel's identity mapping overlap
with restore_jump_address by creating special super-simple page tables just
for the final jump to the image kernel.
It is on top of the $subject patch. My test box still works with this applied,
but then it worked without it as well.
If it doesn't help, the identity mapping created by set_up_temporary_mappings()
is still not adequate for some reason most likely and we'll need to find out
why.
Unfortunately, still with $subject patch + this one, triple fault and
reboot after reading the hibernation image.
The last patch I sent had a problem, because if restore_jump_address really
overlapped with the identity mapping of the restore kernel, it might share
PGD or PUD entries with that mapping and that should have been taken into
account.
Here goes an update. Again, this works on my test machine, but then the
previous version worked on it too ...
---
arch/x86/power/hibernate_64.c | 53 ++++++++++++++++++++++++++++++--------
arch/x86/power/hibernate_asm_64.S | 10 +++++++
2 files changed, 53 insertions(+), 10 deletions(-)
Index: linux-pm/arch/x86/power/hibernate_64.c
===================================================================
@@ -38,14 +38,22 @@ unsigned long jump_address_phys;unsignedlongrestore_cr3__visible;unsignedlongtemp_level4_pgt__visible;+unsignedlongjump_level4_pgt__visible;unsignedlongrelocated_restore_code__visible;-staticintset_up_temporary_text_mapping(pgd_t*pgd)+staticintset_up_temporary_text_mapping(void){+unsignedlongpgd_idx=pgd_index(restore_jump_address);+unsignedlongpud_idx=pud_index(restore_jump_address);+pgd_t*pgd;pmd_t*pmd;pud_t*pud;+pgd=(pgd_t*)get_safe_page(GFP_ATOMIC);+if(!pgd)+return-ENOMEM;+/**Thenewmappingonlyhastocoverthepagecontainingtheimage*kernel'sentrypoint(jump_address_phys),becausetheswitchoverto
@@ -69,10 +77,32 @@ static int set_up_temporary_text_mappingset_pmd(pmd+pmd_index(restore_jump_address),__pmd((jump_address_phys&PMD_MASK)|__PAGE_KERNEL_LARGE_EXEC));-set_pud(pud+pud_index(restore_jump_address),+set_pud(pud+pud_idx,__pud(__pa(pmd)|_KERNPG_TABLE));+set_pgd(pgd+pgd_idx,__pgd(__pa(pud)|_KERNPG_TABLE));++if(pgd_idx!=pgd_index(relocated_restore_code)){+pud=(pud_t*)get_safe_page(GFP_ATOMIC);+if(!pud)+return-ENOMEM;++set_pgd(pgd+pgd_index(relocated_restore_code),+__pgd(__pa(pud)|_KERNPG_TABLE));+}elseif(pud_idx==pud_index(relocated_restore_code)){+gotoset_pmd;+}++pmd=(pmd_t*)get_safe_page(GFP_ATOMIC);+if(!pmd)+return-ENOMEM;++set_pud(pud+pud_index(relocated_restore_code),__pud(__pa(pmd)|_KERNPG_TABLE));-set_pgd(pgd+pgd_index(restore_jump_address),-__pgd(__pa(pud)|_KERNPG_TABLE));++set_pmd:+set_pmd(pmd+pmd_index(relocated_restore_code),+__pmd((__pa(relocated_restore_code)&PMD_MASK)|__PAGE_KERNEL_LARGE_EXEC));++jump_level4_pgt=__pa(pgd);return0;}
@@ -98,11 +128,6 @@ static int set_up_temporary_mappings(voiif(!pgd)return-ENOMEM;-/* Prepare a temporary mapping for the kernel text */-result=set_up_temporary_text_mapping(pgd);-if(result)-returnresult;-/* Set up the direct mapping from scratch */for(i=0;i<nr_pfn_mapped;i++){mstart=pfn_mapped[i].start<<PAGE_SHIFT;
@@ -122,7 +147,10 @@ static int relocate_restore_code(void)pgd_t*pgd;pud_t*pud;-relocated_restore_code=get_safe_page(GFP_ATOMIC);+do+relocated_restore_code=get_safe_page(GFP_ATOMIC);+while((relocated_restore_code&PMD_MASK)==(restore_jump_address&PMD_MASK));+if(!relocated_restore_code)return-ENOMEM;
@@ -162,6 +190,11 @@ int swsusp_arch_resume(void)if(error)returnerror;+/* Prepare a temporary mapping for the jump to the image kernel */+error=set_up_temporary_text_mapping();+if(error)+returnerror;+restore_image();return0;}
So far, I'm unable to reproduce the problem (with the $subject patch
applied) on two different Intel-base machines with 4 Gig and 8 Gig of
RAM.
One thing that's clearly different on my machines is that they both
have usable memory at the end of the e820 map (and the one where Jiri
can reproduce the problem has reserved memory at the end of it, just
like yours).
Thomas, what about the e820 map on your machine?
I'm not sure why that would matter, though.
Thanks,
Rafael
So far, I'm unable to reproduce the problem (with the $subject patch
applied) on two different Intel-base machines with 4 Gig and 8 Gig of
RAM.
So I used your .config to generate one for my test machine and with
that I can reproduce.
The hardware configuration doesn't matter, then, the issue is config-related.
Thanks,
Rafael
Ok, I want to know if the problem is the PUD alignment or the change
of PAGE_OFFSET based all together. Can you test the following change?
(on top of everything else with KASLR enabled). It will randomize the
memory sections only on PGD level.
I applied this on top of both fixes from Rafael from this thread; still no
change in behavior, i.e. reboot immediately after reading the hibernation
image.
--
Jiri Kosina
SUSE Labs
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-10 21:58:34
On Wed, Aug 10, 2016 at 11:52 PM, Jiri Kosina [off-list ref] wrote:
On Wed, 10 Aug 2016, Thomas Garnier wrote:
quoted
Ok, I want to know if the problem is the PUD alignment or the change
of PAGE_OFFSET based all together. Can you test the following change?
(on top of everything else with KASLR enabled). It will randomize the
memory sections only on PGD level.
I applied this on top of both fixes from Rafael from this thread; still no
change in behavior, i.e. reboot immediately after reading the hibernation
image.
Same for me on the box where I can reproduce the problem now.
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-11 00:36:59
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
Thanks,
Rafael
From: Thomas Garnier <hidden> Date: 2016-08-11 01:17:10
On Wed, Aug 10, 2016 at 5:35 PM, Rafael J. Wysocki [off-list ref] wrote:
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Nice find!
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2016-08-11 01:35:45
On Thu, Aug 11, 2016 at 3:17 AM, Thomas Garnier [off-list ref] wrote:
On Wed, Aug 10, 2016 at 5:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Nice find!
quoted
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
Well, I don't think this is related.
In the meantime, I went back to my original .config and verified that
setting CONFIG_DEBUG_LOCK_ALLOC in it caused hibernation to fail (with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied), so
this really matters somehow.
Besides, now that I have a reproducer, I can check various other
things and for example this change (sorry for broken whitespace):
Index: linux-pm/arch/x86/mm/kaslr.c
===================================================================
makes hibernation work for me again in the above configuration. To
me, this means that the $subject patch works as expected and the
problem really is related to the vaddr value being too big.
Thanks,
Rafael
From: Thomas Garnier <hidden> Date: 2016-08-11 18:47:32
On Wed, Aug 10, 2016 at 6:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted hunk
On Thu, Aug 11, 2016 at 3:17 AM, Thomas Garnier [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 5:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Nice find!
quoted
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
Well, I don't think this is related.
In the meantime, I went back to my original .config and verified that
setting CONFIG_DEBUG_LOCK_ALLOC in it caused hibernation to fail (with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied), so
this really matters somehow.
Besides, now that I have a reproducer, I can check various other
things and for example this change (sorry for broken whitespace):
Index: linux-pm/arch/x86/mm/kaslr.c
===================================================================
I think it works because the address is fixed now (just PUD aligned).
/*
* Jump the region and add a minimum padding based on
makes hibernation work for me again in the above configuration. To
me, this means that the $subject patch works as expected and the
problem really is related to the vaddr value being too big.
I managed to debug the restoration and found that a first access
violation happens here:
(gdb) x/20i 0xffffffffb20a46de
0xffffffffb20a46de <trace_suspend_resume+14>:
mov eax,DWORD PTR gs:[rip+0x4df65a4b] # 0xa130 <cpu_number>
Handled by do_async_page_fault which will fault as well on this instruction:
=> 0xffffffffb2047ca1 <do_async_page_fault+1>:
mov eax,DWORD PTR gs:[rip+0x4dfc4e58] # 0xcb00 <apf_reason+64>
So there is a problem with the gs register not being restored
correctly at this stage.
In create_image, there is tracing (trace_suspend_resume) before and
after the suspend. Except at this stage, gs was not yet restored. It
uses the old gs leading to the double fault.
I tested this fix to be correct:
From: Rafael J. Wysocki <hidden> Date: 2016-08-11 21:28:22
On Thursday, August 11, 2016 11:47:27 AM Thomas Garnier wrote:
On Wed, Aug 10, 2016 at 6:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Thu, Aug 11, 2016 at 3:17 AM, Thomas Garnier [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 5:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Nice find!
quoted
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
Well, I don't think this is related.
In the meantime, I went back to my original .config and verified that
setting CONFIG_DEBUG_LOCK_ALLOC in it caused hibernation to fail (with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied), so
this really matters somehow.
Besides, now that I have a reproducer, I can check various other
things and for example this change (sorry for broken whitespace):
Index: linux-pm/arch/x86/mm/kaslr.c
===================================================================
I think it works because the address is fixed now (just PUD aligned).
That's exactly right.
quoted
/*
* Jump the region and add a minimum padding based on
makes hibernation work for me again in the above configuration. To
me, this means that the $subject patch works as expected and the
problem really is related to the vaddr value being too big.
I managed to debug the restoration and found that a first access
violation happens here:
So you were able to get a bit farther than I did. :-)
(gdb) x/20i 0xffffffffb20a46de
0xffffffffb20a46de <trace_suspend_resume+14>:
mov eax,DWORD PTR gs:[rip+0x4df65a4b] # 0xa130 <cpu_number>
Handled by do_async_page_fault which will fault as well on this instruction:
=> 0xffffffffb2047ca1 <do_async_page_fault+1>:
mov eax,DWORD PTR gs:[rip+0x4dfc4e58] # 0xcb00 <apf_reason+64>
So there is a problem with the gs register not being restored
correctly at this stage.
In create_image, there is tracing (trace_suspend_resume) before and
after the suspend. Except at this stage, gs was not yet restored. It
uses the old gs leading to the double fault.
Nice catch!
I established that the problem happened when there was a difference between
the page_offset_base values in the restore and image kernels, so my conclusion
was that the code leaked some information related to virtual addresses from
the restore kernel back to the mage one. Which is exactly what you found. :-)
@@ -300,12 +300,12 @@ static int create_image(int platform_mode)save_processor_state();trace_suspend_resume(TPS("machine_suspend"),PM_EVENT_HIBERNATE,true);error=swsusp_arch_suspend();+/* Restore control flow magically appears here */+restore_processor_state();trace_suspend_resume(TPS("machine_suspend"),PM_EVENT_HIBERNATE,false);if(error)printk(KERN_ERR"PM: Error %d creating hibernation image\n",error);-/* Restore control flow magically appears here */-restore_processor_state();if(!in_suspend)events_check_enabled=false;
Let me know if it works for you. Note that I don't know why this issue
popup with the different config.
Yes, works like charm here (on top of 4.8-rc1 plus the $subject patch).
Please resend with a changelog and sign-off (BTW, your email client
damages whitespace in patches, any chance to avoid that?).
Thanks,
Rafael
From: Thomas Garnier <hidden> Date: 2016-08-11 21:33:03
On Thu, Aug 11, 2016 at 2:33 PM, Rafael J. Wysocki [off-list ref] wrote:
On Thursday, August 11, 2016 11:47:27 AM Thomas Garnier wrote:
quoted
On Wed, Aug 10, 2016 at 6:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Thu, Aug 11, 2016 at 3:17 AM, Thomas Garnier [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 5:35 PM, Rafael J. Wysocki [off-list ref] wrote:
quoted
On Wed, Aug 10, 2016 at 11:59 PM, Jiri Kosina [off-list ref] wrote:
quoted
On Wed, 10 Aug 2016, Rafael J. Wysocki wrote:
quoted
So I used your .config to generate one for my test machine and with
that I can reproduce.
Was that the config I've sent, or did Boris provide one as well? Which one
are you able to reproduce with please?
It's the Boris' one.
Moreover, I have found the options that make the difference: unsetting
CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCK_ALLOC (which also will
unset CONFIG_LOCKDEP AFAICS) in it makes hibernation work again with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied.
Unbelievable, but that's what I'm seeing.
Nice find!
quoted
Now, that leads to a few questions:
- How does lockdep change the picture so it matters for hibernation?
- Why is hibernation the only piece that's affected?
- Why is RANDOMIZE_MEMORY necessary to make this breakage show up?
Thomas, any ideas?
Well, I don't think this is related.
In the meantime, I went back to my original .config and verified that
setting CONFIG_DEBUG_LOCK_ALLOC in it caused hibernation to fail (with
CONFIG_RANDOMIZE_MEMORY set and with the $subject patch applied), so
this really matters somehow.
Besides, now that I have a reproducer, I can check various other
things and for example this change (sorry for broken whitespace):
Index: linux-pm/arch/x86/mm/kaslr.c
===================================================================
I think it works because the address is fixed now (just PUD aligned).
That's exactly right.
quoted
quoted
/*
* Jump the region and add a minimum padding based on
makes hibernation work for me again in the above configuration. To
me, this means that the $subject patch works as expected and the
problem really is related to the vaddr value being too big.
I managed to debug the restoration and found that a first access
violation happens here:
So you were able to get a bit farther than I did. :-)
quoted
(gdb) x/20i 0xffffffffb20a46de
0xffffffffb20a46de <trace_suspend_resume+14>:
mov eax,DWORD PTR gs:[rip+0x4df65a4b] # 0xa130 <cpu_number>
Handled by do_async_page_fault which will fault as well on this instruction:
=> 0xffffffffb2047ca1 <do_async_page_fault+1>:
mov eax,DWORD PTR gs:[rip+0x4dfc4e58] # 0xcb00 <apf_reason+64>
So there is a problem with the gs register not being restored
correctly at this stage.
In create_image, there is tracing (trace_suspend_resume) before and
after the suspend. Except at this stage, gs was not yet restored. It
uses the old gs leading to the double fault.
Nice catch!
Thanks, got lucky.
I established that the problem happened when there was a difference between
the page_offset_base values in the restore and image kernels, so my conclusion
was that the code leaked some information related to virtual addresses from
the restore kernel back to the mage one. Which is exactly what you found. :-)
@@ -300,12 +300,12 @@ static int create_image(int platform_mode)save_processor_state();trace_suspend_resume(TPS("machine_suspend"),PM_EVENT_HIBERNATE,true);error=swsusp_arch_suspend();+/* Restore control flow magically appears here */+restore_processor_state();trace_suspend_resume(TPS("machine_suspend"),PM_EVENT_HIBERNATE,false);if(error)printk(KERN_ERR"PM: Error %d creating hibernation image\n",error);-/* Restore control flow magically appears here */-restore_processor_state();if(!in_suspend)events_check_enabled=false;
Let me know if it works for you. Note that I don't know why this issue
popup with the different config.
Yes, works like charm here (on top of 4.8-rc1 plus the $subject patch).
Please resend with a changelog and sign-off (BTW, your email client
damages whitespace in patches, any chance to avoid that?).
Will do.
I will see how I can fix the whitespace thing, that's odd. Thanks for
the heads-up.