Re: [RFC] mmap(MAP_CONTIG)

On 10/04/2017 06:49 AM, Anshuman Khandual wrote:

On 10/04/2017 05:26 AM, Mike Kravetz wrote:

quoted

At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1].  The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.

When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner.  In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.

Right.

quoted

As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory.  This is via typed memory objects as described
in the posix_typed_mem_open() man page.  Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill.  Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.

Right.

quoted

mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
  pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
  will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory.  As such, the same
  privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.

Why ? May be we have memory around the edge of the existing mapping. Why
give up before trying ?

Just a simplification.  If not to complicated, we could add support for
expansion.  But, it may not be worth the cost and I do not know if there
would be any real use cases.

quoted

- At fork time, private MAP_CONTIG mappings will be converted to regular
  (non-MAP_CONTIG) mapping in the child.  As such a COW fault in the child
  will not require a contiguous allocation.

Makes sense but need to be documented as the child still knows that the buffer
came from a mmap(MAP_CONTIG) call in the parent.

quoted

Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
  than MAX_ORDER.

As I had also mentioned during the presentation at Plumbers, there should be
a fallback approach while attempting to allocate the contiguous memory.

- If order < MAX_ORDER -> alloc_pages()
- If order > MAX_ORDER -> alloc_contig_range()
- If alloc_contig_range() fails attempt a CMA based allocation scheme
  The CMA area should have been initialized at the boot exclusively for
  this purpose (may be with a CONFIG option if some one wants to go for
  this fallback at all) and use cma_alloc() on that area when we need
  to service MAP_CONTIG requests.

I am not sure about the use of CMA and requiring admin setup.  It is
something that can be considered.  However, I suspect people would want
to avoid admin interaction/requirements if possible.

quoted

- MAP_CONTIG should imply MAP_POPULATE.  At mmap time, all pages for the
  mapping must be 'pre-allocated', and they can only be used for the mapping,
  so it makes sense to 'fault in' all pages.

quoted

- Using 'pre-allocated' pages in the fault paths may be intrusive.

But we have already faulted in all of them for the mapping and they
are also locked. Hence there should not be any page faults any more
for the VMA. Am I missing something here ?

I was referring to the action of pre-populating the mapping.  Today that
is done via the normal fault paths.  So, if we use this same scheme for
MAP_CONTIG, the fault paths would need to know about pre-allocated pages.

Sorry for not being more clear as that may have been a source of confusion.

quoted

- We need to keep keep track of those pre-allocated pages until the vma is
  tore down, especially if free_contig_range() must be called

Right, probably tracking them as part of the vm_area_struct itself. 

quoted

Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?

I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time.  It is really ugly, which is why I am not
sharing the code.  Hoping for some comments/suggestions.

I am still wondering why wait till fault time not pre fault all of them
and populate the page tables.

Yes, that is the idea.  I just did not state clearly above.

-- 
Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>