Re: [PATCH] mem: balanced allocation of hugepages

Hi,

On 16.02.2017 16:26, Tan, Jianfeng wrote:

Hi,

quoted

-----Original Message-----
From: Ilya Maximets [mailto:i.maximets@samsung.com]
Sent: Thursday, February 16, 2017 9:01 PM
To: dev@dpdk.org; David Marchand; Gonzalez Monroy, Sergio
Cc: Heetae Ahn; Yuanhan Liu; Tan, Jianfeng; Neil Horman; Pei, Yulong; Ilya
Maximets; stable@dpdk.org
Subject: [PATCH] mem: balanced allocation of hugepages

Currently EAL allocates hugepages one by one not paying
attention from which NUMA node allocation was done.

Such behaviour leads to allocation failure if number of
available hugepages for application limited by cgroups
or hugetlbfs and memory requested not only from the first
socket.

Example:
	# 90 x 1GB hugepages availavle in a system

	cgcreate -g hugetlb:/test
	# Limit to 32GB of hugepages
	cgset -r hugetlb.1GB.limit_in_bytes=34359738368 test
	# Request 4GB from each of 2 sockets
	cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 ...

	EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB
	EAL: 32 not 90 hugepages of size 1024 MB allocated
	EAL: Not enough memory available on socket 1!
	     Requested: 4096MB, available: 0MB
	PANIC in rte_eal_init():
	Cannot init memory

	This happens beacause all allocated pages are
	on socket 0.

For such an use case, why not just use "numactl --interleave=0,1 <DPDK app> xxx"?

Unfortunately, interleave policy doesn't work for me. I suspect kernel configuration
blocks this or I don't understand something in kernel internals.
I'm using 3.10 rt kernel from rhel7.

I tried to set up MPOL_INTERLEAVE in code and it doesn't work for me. Your example
with numactl doesn't work too:

# Limited to 8GB of hugepages
cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 

EAL: Setting up physically contiguous memory...
EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB
EAL: 8 not 90 hugepages of size 1024 MB allocated
EAL: Hugepage /dev/hugepages/rtemap_0 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_1 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_2 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_3 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_4 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_5 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_6 is on socket 0
EAL: Hugepage /dev/hugepages/rtemap_7 is on socket 0
EAL: Not enough memory available on socket 1! Requested: 4096MB, available: 0MB
PANIC in rte_eal_init():
Cannot init memory

Also, using numactl will affect all the allocations in application. This may
cause additional unexpected issues.

Do you see use case like --socket-mem 2048,1024 and only three 1GB-hugepage are allowed?

This case will work with my patch.
But the opposite one '--socket-mem=1024,2048' will fail.
To be clear, we need to allocate all required memory at first
from each numa node and then allocate all other available pages
in round-robin fashion. But such solution looks a little ugly.

What do you think?

Best regards, Ilya Maximets.