Re: [PATCH v3] pseries/drmem: don't cache node id in drmem_lmb struct

On Fri, Aug 21, 2020 at 10:33:10AM +0200, Laurent Dufour wrote:

Le 11/08/2020 à 03:51, Scott Cheloha a écrit :

quoted

[...]

@@ -631,7 +638,7 @@ static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index)
  static int dlpar_add_lmb(struct drmem_lmb *lmb)
  {
  	unsigned long block_sz;
-	int rc;
+	int nid, rc;

  	if (lmb->flags & DRCONF_MEM_ASSIGNED)
  		return -EINVAL;

@@ -642,11 +649,13 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb)
  		return rc;
  	}

-	lmb_set_nid(lmb);
  	block_sz = memory_block_size_bytes();

+	/* Find the node id for this address. */
+	nid = memory_add_physaddr_to_nid(lmb->base_addr);

I think we could be more efficient here.
Here is the call stack behind memory_add_physaddr_to_nid():

  memory_add_physaddr_to_nid(lmb->base_addr)
    hot_add_scn_to_nid()
      if (of_find_node_by_path("/ibm,dynamic-reconfiguration-memory")) == true*
      then
	  hot_add_drconf_scn_to_nid()
	    for_each_drmem_lmb() to find the LMB based on lmb->base_addr
	      of_drconf_to_nid_single(found LMB)
		use lmb->aa_index to get the nid.

* that test is necessarily true when called from dlpar_add_lmb()
otherwise the call to update_lmb_associativity_index() would have
failed earlier.

Basically, we have a LMB and we later walk all the LMBs to find that lmb
again.  In the case of dlpar_add_lmb(), it would be more efficient to
directly call of_drconf_to_nid_single(). That function is not exported
from arch/powerpc/mm/numa.c but it may be good to export it through that
patch.

I've posted a patch for this:

https://lore.kernel.org/linuxppc-dev/20200910175637.2865160-1-cheloha@linux.ibm.com/T/#u (local)

The speedup is nice, especially for LMBs at the end of the array.