From: Michael Ellerman <mpe@ellerman.id.au> Date: 2018-02-13 02:48:17
Andrew Morton [off-list ref] writes:
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Folks, this patch fixes a BUG and is marked for -stable. Can we please
prioritize it?
It's not crashing for me (on 4.16-rc1):
# ./huge-poison
Poisoning page...once
Poisoning page...once again
madvise: Bad address
And I guess the above is the expected behaviour?
Looking at the function trace it looks like the 2nd madvise is going
down reasonable code paths, but I don't know for sure:
8) | SyS_madvise() {
8) | capable() {
8) | ns_capable_common() {
8) 0.094 us | cap_capable();
8) 0.516 us | }
8) 1.052 us | }
8) | get_user_pages_fast() {
8) 0.354 us | gup_pgd_range();
8) | get_user_pages_unlocked() {
8) 0.050 us | down_read();
8) | __get_user_pages() {
8) | find_extend_vma() {
8) | find_vma() {
8) 0.148 us | vmacache_find();
8) 0.622 us | }
8) 1.064 us | }
8) 0.028 us | arch_vma_access_permitted();
8) | follow_hugetlb_page() {
8) | huge_pte_offset() {
8) 0.128 us | __find_linux_pte();
8) 0.580 us | }
8) 0.048 us | _raw_spin_lock();
8) | hugetlb_fault() {
8) | huge_pte_offset() {
8) 0.034 us | __find_linux_pte();
8) 0.434 us | }
8) 0.028 us | is_hugetlb_entry_migration();
8) 0.032 us | is_hugetlb_entry_hwpoisoned();
8) 2.118 us | }
8) 4.940 us | }
8) 7.468 us | }
8) 0.056 us | up_read();
8) 8.722 us | }
8) + 10.264 us | }
8) + 12.212 us | }
cheers
From: Mike Kravetz <hidden> Date: 2018-02-13 22:34:12
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
--
Mike Kravetz
quoted
Folks, this patch fixes a BUG and is marked for -stable. Can we please
prioritize it?
It's not crashing for me (on 4.16-rc1):
# ./huge-poison
Poisoning page...once
Poisoning page...once again
madvise: Bad address
And I guess the above is the expected behaviour?
Looking at the function trace it looks like the 2nd madvise is going
down reasonable code paths, but I don't know for sure:
8) | SyS_madvise() {
8) | capable() {
8) | ns_capable_common() {
8) 0.094 us | cap_capable();
8) 0.516 us | }
8) 1.052 us | }
8) | get_user_pages_fast() {
8) 0.354 us | gup_pgd_range();
8) | get_user_pages_unlocked() {
8) 0.050 us | down_read();
8) | __get_user_pages() {
8) | find_extend_vma() {
8) | find_vma() {
8) 0.148 us | vmacache_find();
8) 0.622 us | }
8) 1.064 us | }
8) 0.028 us | arch_vma_access_permitted();
8) | follow_hugetlb_page() {
8) | huge_pte_offset() {
8) 0.128 us | __find_linux_pte();
8) 0.580 us | }
8) 0.048 us | _raw_spin_lock();
8) | hugetlb_fault() {
8) | huge_pte_offset() {
8) 0.034 us | __find_linux_pte();
8) 0.434 us | }
8) 0.028 us | is_hugetlb_entry_migration();
8) 0.032 us | is_hugetlb_entry_hwpoisoned();
8) 2.118 us | }
8) 4.940 us | }
8) 7.468 us | }
8) 0.056 us | up_read();
8) 8.722 us | }
8) + 10.264 us | }
8) + 12.212 us | }
cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Wanpeng Li <hidden> Date: 2019-05-28 09:49:37
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
Regards,
Wanpeng Li
From: Mike Kravetz <hidden> Date: 2019-05-29 23:33:24
On 5/28/19 2:49 AM, Wanpeng Li wrote:
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
--
Mike Kravetz
On Wed, May 29, 2019 at 04:31:01PM -0700, Mike Kravetz wrote:
On 5/28/19 2:49 AM, Wanpeng Li wrote:
quoted
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
Thanks for forwarding this to me, I'm feeling that memory error handling
on 1GB hugepage is demanded as real use case.
quoted
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
I agree that that's better.
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
Yes, that's not practical, so we need at first establish the code base for
2GB hugetlb splitting and then extending it to 1GB next.
Thanks,
Naoya Horiguchi
From: Wanpeng Li <hidden> Date: 2019-06-11 08:41:58
On Tue, 11 Jun 2019 at 07:51, Naoya Horiguchi [off-list ref] wrote:
On Wed, May 29, 2019 at 04:31:01PM -0700, Mike Kravetz wrote:
quoted
On 5/28/19 2:49 AM, Wanpeng Li wrote:
quoted
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
Thanks for forwarding this to me, I'm feeling that memory error handling
on 1GB hugepage is demanded as real use case.
quoted
quoted
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
I agree that that's better.
quoted
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
Yes, that's not practical, so we need at first establish the code base for
2GB hugetlb splitting and then extending it to 1GB next.
I'm working on this, thanks for the inputs.
Regards,
Wanpeng Li
From: Wanpeng Li <hidden> Date: 2019-08-20 07:04:28
Cc Mel Gorman, Kirill, Dave Hansen,
On Tue, 11 Jun 2019 at 07:51, Naoya Horiguchi [off-list ref] wrote:
On Wed, May 29, 2019 at 04:31:01PM -0700, Mike Kravetz wrote:
quoted
On 5/28/19 2:49 AM, Wanpeng Li wrote:
quoted
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
Thanks for forwarding this to me, I'm feeling that memory error handling
on 1GB hugepage is demanded as real use case.
quoted
quoted
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
I agree that that's better.
quoted
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
Yes, that's not practical, so we need at first establish the code base for
2GB hugetlb splitting and then extending it to 1GB next.
I found it is not easy to split. There is a unique hugetlb page size
that is associated with a mounted hugetlbfs filesystem, file remap to
2MB/4KB will break this. How about hard offline 1GB hugetlb page as
what has already done in soft offline, replace the corrupted 1GB page
by new 1GB page through page migration, the offending/corrupted area
in the original 1GB page doesn't need to be copied into the new page,
the offending/corrupted area in new page can keep full zero just as it
is clear during hugetlb page fault, other sub-pages of the original
1GB page can be freed to buddy system. The sigbus signal is sent to
userspace w/ offending/corrupted virtual address, and signal code,
userspace should take care this.
Regards,
Wanpeng Li
On Tue, Aug 20, 2019 at 03:03:55PM +0800, Wanpeng Li wrote:
Cc Mel Gorman, Kirill, Dave Hansen,
On Tue, 11 Jun 2019 at 07:51, Naoya Horiguchi [off-list ref] wrote:
quoted
On Wed, May 29, 2019 at 04:31:01PM -0700, Mike Kravetz wrote:
quoted
On 5/28/19 2:49 AM, Wanpeng Li wrote:
quoted
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
Thanks for forwarding this to me, I'm feeling that memory error handling
on 1GB hugepage is demanded as real use case.
quoted
quoted
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
I agree that that's better.
quoted
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
Yes, that's not practical, so we need at first establish the code base for
2GB hugetlb splitting and then extending it to 1GB next.
I found it is not easy to split. There is a unique hugetlb page size
that is associated with a mounted hugetlbfs filesystem, file remap to
2MB/4KB will break this. How about hard offline 1GB hugetlb page as
what has already done in soft offline, replace the corrupted 1GB page
by new 1GB page through page migration, the offending/corrupted area
in the original 1GB page doesn't need to be copied into the new page,
the offending/corrupted area in new page can keep full zero just as it
is clear during hugetlb page fault, other sub-pages of the original
1GB page can be freed to buddy system. The sigbus signal is sent to
userspace w/ offending/corrupted virtual address, and signal code,
userspace should take care this.
Splitting hugetlb is simply hard, IMHO. THP splitting is done by years
of effort by many great kernel develpers, and I don't think doing similar
development on hugetlb is a good idea. I thought of converting hugetlb
into thp, but maybe it's not an easy task either.
"Hard offlining via soft offlining" approach sounds new and promising to me.
I guess we don't need a large patchset to do this. So, thanks for the idea!
- Naoya
From: Wanpeng Li <hidden> Date: 2019-08-21 07:15:50
On Wed, 21 Aug 2019 at 13:41, Naoya Horiguchi [off-list ref] wrote:
On Tue, Aug 20, 2019 at 03:03:55PM +0800, Wanpeng Li wrote:
quoted
Cc Mel Gorman, Kirill, Dave Hansen,
On Tue, 11 Jun 2019 at 07:51, Naoya Horiguchi [off-list ref] wrote:
quoted
On Wed, May 29, 2019 at 04:31:01PM -0700, Mike Kravetz wrote:
quoted
On 5/28/19 2:49 AM, Wanpeng Li wrote:
quoted
Cc Paolo,
Hi all,
On Wed, 14 Feb 2018 at 06:34, Mike Kravetz [off-list ref] wrote:
quoted
On 02/12/2018 06:48 PM, Michael Ellerman wrote:
quoted
Andrew Morton [off-list ref] writes:
quoted
On Thu, 08 Feb 2018 12:30:45 +0000 Punit Agrawal [off-list ref] wrote:
quoted
quoted
So I don't think that the above test result means that errors are properly
handled, and the proposed patch should help for arm64.
Although, the deviation of pud_huge() avoids a kernel crash the code
would be easier to maintain and reason about if arm64 helpers are
consistent with expectations by core code.
I'll look to update the arm64 helpers once this patch gets merged. But
it would be helpful if there was a clear expression of semantics for
pud_huge() for various cases. Is there any version that can be used as
reference?
Is that an ack or tested-by?
Mike keeps plaintively asking the powerpc developers to take a look,
but they remain steadfastly in hiding.
Cc'ing linuxppc-dev is always a good idea :)
Thanks Michael,
I was mostly concerned about use cases for soft/hard offline of huge pages
larger than PMD_SIZE on powerpc. I know that powerpc supports PGD_SIZE
huge pages, and soft/hard offline support was specifically added for this.
See, 94310cbcaa3c "mm/madvise: enable (soft|hard) offline of HugeTLB pages
at PGD level"
This patch will disable that functionality. So, at a minimum this is a
'heads up'. If there are actual use cases that depend on this, then more
work/discussions will need to happen. From the e-mail thread on PGD_SIZE
support, I can not tell if there is a real use case or this is just a
'nice to have'.
1GB hugetlbfs pages are used by DPDK and VMs in cloud deployment, we
encounter gup_pud_range() panic several times in product environment.
Is there any plan to reenable and fix arch codes?
I too am aware of slightly more interest in 1G huge pages. Suspect that as
Intel MMU capacity increases to handle more TLB entries there will be more
and more interest.
Personally, I am not looking at this issue. Perhaps Naoya will comment as
he know most about this code.
Thanks for forwarding this to me, I'm feeling that memory error handling
on 1GB hugepage is demanded as real use case.
quoted
quoted
In addition, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/mmu.c#n3213
The memory in guest can be 1GB/2MB/4K, though the host-backed memory
are 1GB hugetlbfs pages, after above PUD panic is fixed,
try_to_unmap() which is called in MCA recovery path will mark the PUD
hwpoison entry. The guest will vmexit and retry endlessly when
accessing any memory in the guest which is backed by this 1GB poisoned
hugetlbfs page. We have a plan to split this 1GB hugetblfs page by 2MB
hugetlbfs pages/4KB pages, maybe file remap to a virtual address range
which is 2MB/4KB page granularity, also split the KVM MMU 1GB SPTE
into 2MB/4KB and mark the offensive SPTE w/ a hwpoison flag, a sigbus
will be delivered to VM at page fault next time for the offensive
SPTE. Is this proposal acceptable?
I am not sure of the error handling design, but this does sound reasonable.
I agree that that's better.
quoted
That block of code which potentially dissolves a huge page on memory error
is hard to understand and I'm not sure if that is even the 'normal'
functionality. Certainly, we would hate to waste/poison an entire 1G page
for an error on a small subsection.
Yes, that's not practical, so we need at first establish the code base for
2GB hugetlb splitting and then extending it to 1GB next.
I found it is not easy to split. There is a unique hugetlb page size
that is associated with a mounted hugetlbfs filesystem, file remap to
2MB/4KB will break this. How about hard offline 1GB hugetlb page as
what has already done in soft offline, replace the corrupted 1GB page
by new 1GB page through page migration, the offending/corrupted area
in the original 1GB page doesn't need to be copied into the new page,
the offending/corrupted area in new page can keep full zero just as it
is clear during hugetlb page fault, other sub-pages of the original
1GB page can be freed to buddy system. The sigbus signal is sent to
userspace w/ offending/corrupted virtual address, and signal code,
userspace should take care this.
Splitting hugetlb is simply hard, IMHO. THP splitting is done by years
of effort by many great kernel develpers, and I don't think doing similar
development on hugetlb is a good idea. I thought of converting hugetlb
into thp, but maybe it's not an easy task either.
"Hard offlining via soft offlining" approach sounds new and promising to me.
I guess we don't need a large patchset to do this. So, thanks for the idea!
Good, I will wait a while, and start to cook the patches if there is
no opposite of voice.
Regards,
Wanpeng Li