-----Original Message-----
From: David Hildenbrand <redacted>
Sent: Wednesday, July 29, 2020 5:35 PM
To: Mike Rapoport <redacted>; Justin He <redacted>
Cc: Dan Williams <redacted>; Vishal Verma
[off-list ref]; Catalin Marinas [off-list ref];
Will Deacon [off-list ref]; Greg Kroah-Hartman
[off-list ref]; Rafael J. Wysocki [off-list ref]; Dave
Jiang [off-list ref]; Andrew Morton [off-list ref];
Steve Capper [off-list ref]; Mark Rutland [off-list ref];
Logan Gunthorpe [off-list ref]; Anshuman Khandual
[off-list ref]; Hsin-Yi Wang [off-list ref]; Jason
Gunthorpe [off-list ref]; Dave Hansen [off-list ref]; Kees
Cook [off-list ref]; linux-arm-kernel@lists.infradead.org; linux-
kernel@vger.kernel.org; linux-nvdimm@lists.01.org; linux-mm@kvack.org; Wei
Yang [off-list ref]; Pankaj Gupta
[off-list ref]; Ira Weiny [off-list ref]; Kaly Xin
[off-list ref]
Subject: Re: [RFC PATCH 0/6] decrease unnecessary gap due to pmem kmem
alignment

On 29.07.20 11:31, Mike Rapoport wrote:

quoted

Hi Justin,

On Wed, Jul 29, 2020 at 08:27:58AM +0000, Justin He wrote:

quoted

Hi David

quoted

quoted

Without this series, if qemu creates a 4G bytes nvdimm device, we can

only

quoted

use 2G bytes for dax pmem(kmem) in the worst case.
e.g.
240000000-33fdfffff : Persistent Memory
We can only use the memblock between [240000000, 2ffffffff] due to

the

quoted

quoted

quoted

hard

quoted

limitation. It wastes too much memory space.

Decreasing the SECTION_SIZE_BITS on arm64 might be an alternative,

but

quoted

quoted

quoted

there

quoted

are too many concerns from other constraints, e.g. PAGE_SIZE, hugetlb,
SPARSEMEM_VMEMMAP, page bits in struct page ...

Beside decreasing the SECTION_SIZE_BITS, we can also relax the kmem

alignment

quoted

with memory_block_size_bytes().

Tested on arm64 guest and x86 guest, qemu creates a 4G pmem device.

dax

quoted

quoted

quoted

pmem

quoted

can be used as ram with smaller gap. Also the kmem hotplug add/remove

are both

quoted

tested on arm64/x86 guest.

Hi,

I am not convinced this use case is worth such hacks (that’s what it

is)

quoted

quoted

quoted

for now. On real machines pmem is big - your example (losing 50% is
extreme).

I would much rather want to see the section size on arm64 reduced. I
remember there were patches and that at least with a base page size of

4k

quoted

quoted

quoted

it can be reduced drastically (64k base pages are more problematic due

to

quoted

quoted

quoted

the ridiculous THP size of 512M). But could be a section size of 512

is

quoted

quoted

quoted

possible on all configs right now.

Yes, I once investigated how to reduce section size on arm64

thoughtfully:

quoted

quoted

There are many constraints for reducing SECTION_SIZE_BITS
1. Given page->flags bits is limited, SECTION_SIZE_BITS can't be

reduced too

quoted

quoted

   much.
2. Once CONFIG_SPARSEMEM_VMEMMAP is enabled, section id will not be

counted

quoted

quoted

   into page->flags.
3. MAX_ORDER depends on SECTION_SIZE_BITS
 - 3.1 mmzone.h
#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
#error Allocator MAX_ORDER exceeds SECTION_SIZE
#endif
 - 3.2 hugepage_init()
MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER >= MAX_ORDER);

Hence when ARM64_4K_PAGES && CONFIG_SPARSEMEM_VMEMMAP are enabled,
SECTION_SIZE_BITS can be reduced to 27.
But when ARM64_64K_PAGES, given 3.2, MAX_ORDER > 29-16 = 13.
Given 3.1 SECTION_SIZE_BITS >= MAX_ORDER+15 > 28. So SECTION_SIZE_BITS

can not

quoted

quoted

be reduced to 27.

In one word, if we considered to reduce SECTION_SIZE_BITS on arm64, the

Kconfig

quoted

quoted

might be very complicated,e.g. we still need to consider the case for
ARM64_16K_PAGES.

It is not necessary to pollute Kconfig with that.
arch/arm64/include/asm/sparesemem.h can have something like

#ifdef CONFIG_ARM64_64K_PAGES
#define SPARSE_SECTION_SIZE 29
#elif defined(CONFIG_ARM16K_PAGES)
#define SPARSE_SECTION_SIZE 28
#elif defined(CONFIG_ARM4K_PAGES)
#define SPARSE_SECTION_SIZE 27
#else
#error
#endif

ack

Thanks, David and Mike. Will discuss it further more with arm internally about
the thoughtful section_size change

--
Cheers,
Justin (Jia He)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel