To summarize, the section size bits for each base page size config
should always

a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS

Pageblocks must also always fall completely into a section.

b. Provide minimum possible section size for a given base page config to
   have increased agility during memory hotplug operations and reduced
   vmemmap wastage for sections with holes.

OTOH, making the section size too small (e.g., 16MB) creates way to many
memory block devices in /sys/devices/system/memory/, and might consume
too many page->flags bits in the !vmemmap case.

For bigger setups, we might, similar to x86-64 (e.g., >= 64 GiB),
determine the memory_block_size_bytes() at runtime (spanning multiple
sections then), once it becomes relevant.

c. Allow 4K base page configs to have PMD based vmemmap mappings

Agreed.

Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform,
the following would always avoid the condition (a)

SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

			- 22 (11 - 1 + 12) for 4K pages
			- 24 (11 - 1 + 14) for 16K pages without THP
			- 25 (12 - 1 + 14) for 16K pages with THP
			- 26 (11 - 1 + 16) for 64K pages without THP
			- 29 (14 - 1 + 16) for 64K pages with THP

Apart from overriding 4K base page size config to have 27 as section size
bits, should not all other values be okay here ? But then wondering what
benefit 128MB (27 bits) section size for 16K config would have ? OR the
objective here is to match 16K page size config with default x86-64.

We don't want to have sections that are too small. We don't want to have
sections that are too big :)

Not sure if we really want to allow setting e.g., a section size of 4
MB. That's just going to hurt. IMHO, something in the range of 64..256
MB is quite a good choice, where possible.

quoted

(If we worry about the number of section bits in page->flags, we could
glue it to vmemmap support where that does not matter)

Could you please elaborate ? Could smaller section size bits numbers like
22 or 24 create problems in page->flags without changing other parameters
like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP

Yes, in the !vmemmap case, we have to store the section_nr in there.
IIRC, it's less of an issue with section sizes like 128 MB.

i.e 26 bits in section size, fails to boot.

26 bits would mean 64 MB, no? Not sure if that's possible even without
THP (MAX_ORDER - 1, pageblock_order ...) on 64k pages. I'd assume 512 MB
is the lowest we can go. I'd assume this would crash :)

As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB
for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER,
at least for now.

That's also what I would prefer, keeping it simple.

-- 
Thanks,

David / dhildenb