On 25-02-26, Marco Felsch wrote:

Hi,

On 24-10-08, Oreoluwa Babatunde wrote:

quoted

Reserved memory regions defined in the devicetree can be broken up into
two groups:
i) Statically-placed reserved memory regions
i.e. regions defined with a static start address and size using the
     "reg" property.
ii) Dynamically-placed reserved memory regions.
i.e. regions defined by specifying an address range where they can be
     placed in memory using the "alloc_ranges" and "size" properties.

These regions are processed and set aside at boot time.
This is done in two stages as seen below:

Stage 1:
At this stage, fdt_scan_reserved_mem() scans through the child nodes of
the reserved_memory node using the flattened devicetree and does the
following:

1) If the node represents a statically-placed reserved memory region,
   i.e. if it is defined using the "reg" property:
   - Call memblock_reserve() or memblock_mark_nomap() as needed.
   - Add the information for that region into the reserved_mem array
     using fdt_reserved_mem_save_node().
     i.e. fdt_reserved_mem_save_node(node, name, base, size).

2) If the node represents a dynamically-placed reserved memory region,
   i.e. if it is defined using "alloc-ranges" and "size" properties:
   - Add the information for that region to the reserved_mem array with
     the starting address and size set to 0.
     i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
   Note: This region is saved to the array with a starting address of 0
   because a starting address is not yet allocated for it.

Stage 2:
After iterating through all the reserved memory nodes and storing their
relevant information in the reserved_mem array,fdt_init_reserved_mem() is
called and does the following:

1) For statically-placed reserved memory regions:
   - Call the region specific init function using
     __reserved_mem_init_node().
2) For dynamically-placed reserved memory regions:
   - Call __reserved_mem_alloc_size() which is used to allocate memory
     for each of these regions, and mark them as nomap if they have the
     nomap property specified in the DT.
   - Call the region specific init function.

The current size of the resvered_mem array is 64 as is defined by
MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
how many reserved memory regions can be specified on a system.
As systems continue to grow more and more complex, the number of
reserved memory regions needed are also growing and are starting to hit
this 64 count limit, hence the need to make the reserved_mem array
dynamically sized (i.e. dynamically allocating memory for the
reserved_mem array using membock_alloc_*).

On architectures such as arm64, memory allocated using memblock is
writable only after the page tables have been setup. This means that if
the reserved_mem array is going to be dynamically allocated, it needs to
happen after the page tables have been setup, not before.

Since the reserved memory regions are currently being processed and
added to the array before the page tables are setup, there is a need to
change the order in which some of the processing is done to allow for
the reserved_mem array to be dynamically sized.

It is possible to process the statically-placed reserved memory regions
without needing to store them in the reserved_mem array until after the
page tables have been setup because all the information stored in the
array is readily available in the devicetree and can be referenced at
any time.
Dynamically-placed reserved memory regions on the other hand get
assigned a start address only at runtime, and hence need a place to be
stored once they are allocated since there is no other referrence to the
start address for these regions.

Hence this patch changes the processing order of the reserved memory
regions in the following ways:

Step 1:
fdt_scan_reserved_mem() scans through the child nodes of
the reserved_memory node using the flattened devicetree and does the
following:

1) If the node represents a statically-placed reserved memory region,
   i.e. if it is defined using the "reg" property:
   - Call memblock_reserve() or memblock_mark_nomap() as needed.

2) If the node represents a dynamically-placed reserved memory region,
   i.e. if it is defined using "alloc-ranges" and "size" properties:
   - Call __reserved_mem_alloc_size() which will:
     i) Allocate memory for the reserved region and call
     memblock_mark_nomap() as needed.
     ii) Call the region specific initialization function using
     fdt_init_reserved_mem_node().
     iii) Save the region information in the reserved_mem array using
     fdt_reserved_mem_save_node().

Step 2:
1) This stage of the reserved memory processing is now only used to add
   the statically-placed reserved memory regions into the reserved_mem
   array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
   region specific initialization functions.

2) This step has also been moved to be after the page tables are
   setup. Moving this will allow us to replace the reserved_mem
   array with a dynamically sized array before storing the rest of
   these regions.

Signed-off-by: Oreoluwa Babatunde <redacted>
---
 drivers/of/fdt.c             |   5 +-
 drivers/of/of_private.h      |   3 +-
 drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
 3 files changed, 122 insertions(+), 54 deletions(-)

this patch got into stable kernel 6.12.13++ as part of Stable-dep-of.
The stable kernel commit is: 9a0fe62f93ede02c27aaca81112af1e59c8c0979.

With the patch applied I see that the cma area pool is misplaced which
cause my 4G device to fail to activate the cma pool. Below are some
logs:

*** Good case (6.12)

root@test:~# dmesg|grep -i cma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
[    0.056915] Memory: 3695024K/4194304K available (15552K kernel code, 2510K rwdata, 5992K rodata, 6016K init, 489K bss, 231772K reserved, 262144K cma-reserved)

*** Bad (6.12.16)

root@test:~# dmesg|grep -i cma
[    0.000000] Reserved memory: created CMA memory pool at 0x00000000f2000000, size 256 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000f2000000..0x0000000101ffffff (262144 KiB) map reusable linux,cma
[    0.056968] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)
[    0.116920] cma: CMA area linux,cma could not be activated

*** Good (6.12.16, revert 9a0fe62f93ed)

root@test:~# dmesg|grep -i cma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
[    0.060976] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)

Below is our reserved-memory dts node:

reserved-memory {
	#address-cells = <2>; 
	#size-cells = <2>; 
	ranges;

	linux,cma {
		compatible = "shared-dma-pool";
		reusable;
		/*
		 * The CMA area must be in the lower 32-bit address range.
		 */
		alloc-ranges = <0x0 0x42000000 0 0xc0000000>;
		size = <0x0 0x10000000>;
		alignment = <0 0x2000>;
		linux,cma-default;
	};

	optee-core@40000000 {
		reg = <0 0x40000000 0 0x1e00000>;
		no-map;
	};

	optee-shm@41e00000 {
		reg = <0 0x41e00000 0 0x200000>;
		no-map;
	};

	m7_reserved: m7@80000000 {
		reg = <0 0x80000000 0 0x1000000>;
		no-map;
	};

	vdev0vring0: vdev0vring0@55000000 {
		reg = <0 0x55000000 0 0x8000>;
		no-map;
	};

	vdev0vring1: vdev0vring1@55008000 {
		reg = <0 0x55008000 0 0x8000>;
		no-map;
	};

	rsc_table: rsc-table@550ff000 {
		reg = <0 0x550ff000 0 0x1000>;
		no-map;
	};

	ram_console_buffer: ram-console-buffer@55100000 {
		reg = <0 0x55100000 0 0x1000>;
		no-map;
	};

	vdev0buffer: vdev0buffer@55400000 {
		compatible = "shared-dma-pool";
		reg = <0 0x55400000 0 0x100000>;
		no-map;
	};
};

My current workaround is to revert commit 9a0fe62f93ed and the
dep-chain: 2d1d620ff27b444 8de4e5a92282. But I would like to get a

It's not 2d1d620ff27b444 but e61977c71494.

proper solution without having revert commits in my downstream
patchstack.

Regards,
  Marco