From: Mike Kravetz <hidden> Date: 2017-10-03 23:57:14
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner. In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
- Using 'pre-allocated' pages in the fault paths may be intrusive.
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called.
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
[1] https://www.linuxplumbersconf.org/2017/ocw/proposals/4669
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Nazarewicz <hidden> Date: 2017-10-04 11:54:11
On Tue, Oct 03 2017, Mike Kravetz wrote:
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
Issue I have is that kind of memory needed may depend on a device. Some
may require contiguous blocks. Some may support scatter-gather. Some
may be behind IO-MMU and not care either way.
Furthermore, I feel déjà vu. Wasn’t dmabuf supposed to address this
issue?
--
Best regards
ミハウ “𝓶𝓲𝓷𝓪86” ナザレヴイツ
«If at first you don’t succeed, give up skydiving»
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner. In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
Right.
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
Right.
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
Why ? May be we have memory around the edge of the existing mapping. Why
give up before trying ?
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Makes sense but need to be documented as the child still knows that the buffer
came from a mmap(MAP_CONTIG) call in the parent.
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
As I had also mentioned during the presentation at Plumbers, there should be
a fallback approach while attempting to allocate the contiguous memory.
- If order < MAX_ORDER -> alloc_pages()
- If order > MAX_ORDER -> alloc_contig_range()
- If alloc_contig_range() fails attempt a CMA based allocation scheme
The CMA area should have been initialized at the boot exclusively for
this purpose (may be with a CONFIG option if some one wants to go for
this fallback at all) and use cma_alloc() on that area when we need
to service MAP_CONTIG requests.
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
- Using 'pre-allocated' pages in the fault paths may be intrusive.
But we have already faulted in all of them for the mapping and they
are also locked. Hence there should not be any page faults any more
for the VMA. Am I missing something here ?
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called
Right, probably tracking them as part of the vm_area_struct itself.
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
I am still wondering why wait till fault time not pre fault all of them
and populate the page tables.
From: Christopher Lameter <hidden> Date: 2017-10-04 16:05:36
On Wed, 4 Oct 2017, Anshuman Khandual wrote:
quoted
- Using 'pre-allocated' pages in the fault paths may be intrusive.
But we have already faulted in all of them for the mapping and they
are also locked. Hence there should not be any page faults any more
for the VMA. Am I missing something here ?
The PTEs may be torn down and have to reestablished through a page faults.
Page faults would not allocate memory.
I am still wondering why wait till fault time not pre fault all of them
and populate the page tables.
They are populated but some processes (swap and migration) may tear them
down.
From: Mike Kravetz <hidden> Date: 2017-10-04 17:09:17
On 10/04/2017 04:54 AM, Michal Nazarewicz wrote:
On Tue, Oct 03 2017, Mike Kravetz wrote:
quoted
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
Issue I have is that kind of memory needed may depend on a device. Some
may require contiguous blocks. Some may support scatter-gather. Some
may be behind IO-MMU and not care either way.
Furthermore, I feel déjà vu. Wasn’t dmabuf supposed to address this
issue?
Thanks Michal,
I was unaware of dmabuf and am just now looking at capabilities. The
question is whether or not the IB driver writers requesting mmap(MAP_CONTIG)
functionality could make use of dmabuf. That is out of my are of expertise,
so I will let them reply.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-04 17:36:21
On 10/04/2017 06:49 AM, Anshuman Khandual wrote:
On 10/04/2017 05:26 AM, Mike Kravetz wrote:
quoted
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner. In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
Right.
quoted
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
Right.
quoted
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
Why ? May be we have memory around the edge of the existing mapping. Why
give up before trying ?
Just a simplification. If not to complicated, we could add support for
expansion. But, it may not be worth the cost and I do not know if there
would be any real use cases.
quoted
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Makes sense but need to be documented as the child still knows that the buffer
came from a mmap(MAP_CONTIG) call in the parent.
quoted
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
As I had also mentioned during the presentation at Plumbers, there should be
a fallback approach while attempting to allocate the contiguous memory.
- If order < MAX_ORDER -> alloc_pages()
- If order > MAX_ORDER -> alloc_contig_range()
- If alloc_contig_range() fails attempt a CMA based allocation scheme
The CMA area should have been initialized at the boot exclusively for
this purpose (may be with a CONFIG option if some one wants to go for
this fallback at all) and use cma_alloc() on that area when we need
to service MAP_CONTIG requests.
I am not sure about the use of CMA and requiring admin setup. It is
something that can be considered. However, I suspect people would want
to avoid admin interaction/requirements if possible.
quoted
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
quoted
- Using 'pre-allocated' pages in the fault paths may be intrusive.
But we have already faulted in all of them for the mapping and they
are also locked. Hence there should not be any page faults any more
for the VMA. Am I missing something here ?
I was referring to the action of pre-populating the mapping. Today that
is done via the normal fault paths. So, if we use this same scheme for
MAP_CONTIG, the fault paths would need to know about pre-allocated pages.
Sorry for not being more clear as that may have been a source of confusion.
quoted
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called
Right, probably tracking them as part of the vm_area_struct itself.
quoted
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
I am still wondering why wait till fault time not pre fault all of them
and populate the page tables.
Yes, that is the idea. I just did not state clearly above.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-04 17:39:17
On 10/04/2017 09:05 AM, Christopher Lameter wrote:
On Wed, 4 Oct 2017, Anshuman Khandual wrote:
quoted
quoted
- Using 'pre-allocated' pages in the fault paths may be intrusive.
But we have already faulted in all of them for the mapping and they
are also locked. Hence there should not be any page faults any more
for the VMA. Am I missing something here ?
The PTEs may be torn down and have to reestablished through a page faults.
Page faults would not allocate memory.
quoted
I am still wondering why wait till fault time not pre fault all of them
and populate the page tables.
They are populated but some processes (swap and migration) may tear them
down.
As mentioned in my reply to Anshuman, the mention of fault paths here
may be a source of confusion. I would expect the entire mapping to be
populated at mmap time, and the pages locked. Therefore, there should
be no swap or migration.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Laura Abbott <hidden> Date: 2017-10-04 21:29:14
On 10/04/2017 10:08 AM, Mike Kravetz wrote:
On 10/04/2017 04:54 AM, Michal Nazarewicz wrote:
quoted
On Tue, Oct 03 2017, Mike Kravetz wrote:
quoted
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
Issue I have is that kind of memory needed may depend on a device. Some
may require contiguous blocks. Some may support scatter-gather. Some
may be behind IO-MMU and not care either way.
Furthermore, I feel déjà vu. Wasn’t dmabuf supposed to address this
issue?
Thanks Michal,
I was unaware of dmabuf and am just now looking at capabilities. The
question is whether or not the IB driver writers requesting mmap(MAP_CONTIG)
functionality could make use of dmabuf. That is out of my are of expertise,
so I will let them reply.
I don't think dmabuf as it exists today would help anything here.
It's designed to share buffers via fd but you still need some
place/driver to actually get the allocation and then export it
since there isn't a single interface for allocations. You could
convert drivers to take a dma_buf fd if there were appropriate
buffers available though.
Thanks,
Laura
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
Hm I didn't find slides on that link, are they available?
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner.
As Michal N. noted, the drivers might have different requirements. Is
contiguity (without extra requirements) so common that it would benefit
from a userspace API change?
Also how are the driver-specific allocations done today? mmap() on the
driver's device? Maybe we could provide some in-kernel API/library to
make them less "ad-hoc". Conversion to MAP_ANONYMOUS would at first seem
like an improvement in that userspace would be able to use a generic
allocation API and all the generic treatment of anonymous pages (LRU
aging, reclaim, migration etc), but the restrictions you listed below
eliminate most of that?
(It's likely that I just don't have enough info about how it works today
so it's difficult to judge)
In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
- Using 'pre-allocated' pages in the fault paths may be intrusive.
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called.
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
[1] https://www.linuxplumbersconf.org/2017/ocw/proposals/4669
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Guy Shattah <hidden> Date: 2017-10-05 08:58:33
I'm on vacation and having technical difficulties uploading the slides. I'll upload them once I'm back.
Sorry
Guy
Outlook ???? Android<https://aka.ms/ghei36>
________________________________
From: Vlastimil Babka <redacted>
Sent: Thursday, October 5, 2017 10:06:49 AM
To: Mike Kravetz; linux-mm@kvack.org; linux-kernel@vger.kernel.org; linux-api@vger.kernel.org
Cc: Marek Szyprowski; Michal Nazarewicz; Aneesh Kumar K.V; Joonsoo Kim; Guy Shattah; Christoph Lameter
Subject: Re: [RFC] mmap(MAP_CONTIG)
On 10/04/2017 01:56 AM, Mike Kravetz wrote:
Hi,
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
Hm I didn't find slides on that link, are they available?
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner.
As Michal N. noted, the drivers might have different requirements. Is
contiguity (without extra requirements) so common that it would benefit
from a userspace API change?
Also how are the driver-specific allocations done today? mmap() on the
driver's device? Maybe we could provide some in-kernel API/library to
make them less "ad-hoc". Conversion to MAP_ANONYMOUS would at first seem
like an improvement in that userspace would be able to use a generic
allocation API and all the generic treatment of anonymous pages (LRU
aging, reclaim, migration etc), but the restrictions you listed below
eliminate most of that?
(It's likely that I just don't have enough info about how it works today
so it's difficult to judge)
In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
- Using 'pre-allocated' pages in the fault paths may be intrusive.
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called.
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
[1] https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linuxplumbersconf.org%2F2017%2Focw%2Fproposals%2F4669&data=02%7C01%7Csguy%40mellanox.com%7Ca0ee0fe4f0f74074b69b08d50bbfa7d5%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636427840155156528&sdata=GYlJ926fwQKSUIKbP7AVI01dasvK%2F0JEWLS%2FoNwJbyU%3D&reserved=0
From: Guy Shattah <hidden> Date: 2017-10-05 12:36:40
I'm on vacation and experiencing technical difficulties uploading the slides. I'll upload them next week.
Sorry
Guy
On 10/04/2017 01:56 AM, Mike Kravetz wrote:
Hi,
quoted
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
Hm I didn't find slides on that link, are they available?
quoted
point out the performance benefits of devices that can take advantage of
larger physically contiguous areas.
When such physically contiguous allocations are done today, they are done
within drivers themselves in an ad-hoc manner.
As Michal N. noted, the drivers might have different requirements. Is
contiguity (without extra requirements) so common that it would benefit
from a userspace API change?
Also how are the driver-specific allocations done today? mmap() on the
driver's device? Maybe we could provide some in-kernel API/library to
make them less "ad-hoc". Conversion to MAP_ANONYMOUS would at first seem
like an improvement in that userspace would be able to use a generic
allocation API and all the generic treatment of anonymous pages (LRU
aging, reclaim, migration etc), but the restrictions you listed below
eliminate most of that?
(It's likely that I just don't have enough info about how it works today
so it's difficult to judge)
quoted
In addition to allocations
for DMA, allocations of this type are also performed for buffers used by
coprocessors and other acceleration engines.
As mentioned in the presentation, posix specifies an interface to obtain
physically contiguous memory. This is via typed memory objects as described
in the posix_typed_mem_open() man page. Since Linux today does not follow
the posix typed memory object model, adding infrastructure for contiguous
memory allocations seems to be overkill. Instead, a proposal was suggested
to add support via a mmap flag: MAP_CONTIG.
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Some implementation considerations:
- alloc_contig_range() or similar will be used for allocations larger
than MAX_ORDER.
- MAP_CONTIG should imply MAP_POPULATE. At mmap time, all pages for the
mapping must be 'pre-allocated', and they can only be used for the mapping,
so it makes sense to 'fault in' all pages.
- Using 'pre-allocated' pages in the fault paths may be intrusive.
- We need to keep keep track of those pre-allocated pages until the vma is
tore down, especially if free_contig_range() must be called.
Thoughts?
- Is such an interface useful?
- Any other ideas on how to achieve the same functionality?
- Any thoughts on implementation?
I have started down the path of pre-allocating contiguous pages at mmap
time and hanging those off the vma(vm_private_data) with some kludges to
use the pages at fault time. It is really ugly, which is why I am not
sharing the code. Hoping for some comments/suggestions.
[1] https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linuxplumbersconf.org%2F2017%2Focw%2Fproposals%2F4669&data=02%7C01%7Csguy%40mellanox.com%7Ca0ee0fe4f0f74074b69b08d50bbfa7d5%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636427840155156528&sdata=GYlJ926fwQKSUIKbP7AVI01dasvK%2F0JEWLS%2FoNwJbyU%3D&reserved=0
From: Christopher Lameter <hidden> Date: 2017-10-05 14:30:26
On Thu, 5 Oct 2017, Vlastimil Babka wrote:
On 10/04/2017 01:56 AM, Mike Kravetz wrote:
quoted
At Plumbers this year, Guy Shattah and Christoph Lameter gave a presentation
titled 'User space contiguous memory allocation for DMA' [1]. The slides
Hm I didn't find slides on that link, are they available?
I just added Guy's slides to the entry.
As Michal N. noted, the drivers might have different requirements. Is
contiguity (without extra requirements) so common that it would benefit
from a userspace API change?
Yes.
Also how are the driver-specific allocations done today? mmap() on the
driver's device? Maybe we could provide some in-kernel API/library to
make them less "ad-hoc". Conversion to MAP_ANONYMOUS would at first seem
like an improvement in that userspace would be able to use a generic
allocation API and all the generic treatment of anonymous pages (LRU
aging, reclaim, migration etc), but the restrictions you listed below
eliminate most of that?
(It's likely that I just don't have enough info about how it works today
so it's difficult to judge)
Contemporary devices typically can address all of memory. Moreover the
device used actually can trigger faults to page in 4k pages if they are
not present (ODP in RDMA layer). There is no need for driver specific
allocation in those drivers.
From: Mike Kravetz <hidden> Date: 2017-10-12 01:47:01
Add new MAP_CONTIG flag to mmap system call. Check for flag in normal
mmap flag processing. If present, pre-allocate a contiguous set of
pages to back the mapping. These pages will be used a fault time, and
the MAP_CONTIG flag implies populating the mapping at the mmap time.
Signed-off-by: Mike Kravetz <redacted>
---
include/uapi/asm-generic/mman.h | 1 +
mm/mmap.c | 94 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 95 insertions(+)
@@ -12,6 +12,7 @@#define MAP_NONBLOCK 0x10000 /* do not block on IO */#define MAP_STACK 0x20000 /* give out an address that is best suited for process/thread stacks */#define MAP_HUGETLB 0x40000 /* create a huge page mapping */+#define MAP_CONTIG 0x80000 /* back with contiguous pages *//* Bits [26:31] are reserved, see mman-common.h for MAP_HUGETLB usage */
@@ -1669,6 +1756,12 @@ unsigned long mmap_region(struct file *file, unsigned long addr,vma->vm_pgoff=pgoff;INIT_LIST_HEAD(&vma->anon_vma_chain);+if(vm_flags&VM_CONTIG){+error=__alloc_vma_contig_range(vma);+if(error)+gotofree_vma;+}+if(file){if(vm_flags&VM_DENYWRITE){error=deny_write_access(file);
@@ -1758,6 +1851,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if(vm_flags&VM_DENYWRITE)allow_write_access(file);free_vma:+__free_vma_contig_range(vma);kmem_cache_free(vm_area_cachep,vma);unacct_error:if(charged)
--
2.13.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -3100,7 +3100,18 @@ static int do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */if(unlikely(anon_vma_prepare(vma)))gotooom;-page=alloc_zeroed_user_highpage_movable(vma,vmf->address);++/*+*InthespecialVM_CONTIGcase,pageshavebeenpre-allocated.So,+*simplygrabtheappropriatepre-allocatedpage.+*/+if(unlikely(vma->vm_flags&VM_CONTIG)){+VM_BUG_ON(!vma->vm_private_data);+page=((structpage*)vma->vm_private_data)++((vmf->address-vma->vm_start)/PAGE_SIZE);+}else{+page=alloc_zeroed_user_highpage_movable(vma,vmf->address);+}if(!page)gotooom;
--
2.13.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-12 01:47:06
The following is a 'possible' way to add such functionality. I just
did what was easy and pre-allocated contiguous pages which are used
to populate the mapping. I did not use any of the higher order
allocators such as alloc_contig_range. Therefore, it is limited to
allocations of MAX_ORDER size. Also, the allocations should probably
be done outside mmap_sem but that was the easiest place to do it in
this quick and easy POC.
I just wanted to throw out some code to get further ideas. It is far
from complete.
Mike Kravetz (3):
mm/map_contig: Add VM_CONTIG flag to vma struct
mm/map_contig: Use pre-allocated pages for VM_CONTIG mappings
mm/map_contig: Add mmap(MAP_CONTIG) support
include/linux/mm.h | 1 +
include/uapi/asm-generic/mman.h | 1 +
kernel/fork.c | 2 +-
mm/memory.c | 13 +++++-
mm/mmap.c | 94 +++++++++++++++++++++++++++++++++++++++++
5 files changed, 109 insertions(+), 2 deletions(-)
--
2.13.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-12 01:47:11
Add the flag VM_CONTIG to vma structure to identify vmas which are
backed by contiguous memory allocations. This flag is not propogated
to child processes, so be sure to clear at fork time.
Signed-off-by: Mike Kravetz <redacted>
---
include/linux/mm.h | 1 +
kernel/fork.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
@@ -189,6 +189,7 @@ extern unsigned int kobjsize(const void *objp);#define VM_ACCOUNT 0x00100000 /* Is a VM accounted object */#define VM_NORESERVE 0x00200000 /* should the VM suppress accounting */#define VM_HUGETLB 0x00400000 /* Huge TLB Page VM */+#define VM_CONTIG 0x00800000 /* Contiguous page backing */#define VM_ARCH_1 0x01000000 /* Architecture-specific flag */#define VM_WIPEONFORK 0x02000000 /* Wipe VMA contents in child. */#define VM_DONTDUMP 0x04000000 /* Do not include in the core dump */
@@ -665,7 +665,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,gotofail_nomem_anon_vma_fork;}elseif(anon_vma_fork(tmp,mpnt))gotofail_nomem_anon_vma_fork;-tmp->vm_flags&=~(VM_LOCKED|VM_LOCKONFAULT);+tmp->vm_flags&=~(VM_LOCKED|VM_LOCKONFAULT|VM_CONTIG);tmp->vm_next=tmp->vm_prev=NULL;file=tmp->vm_file;if(file){
--
2.13.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
The following is a 'possible' way to add such functionality. I just
did what was easy and pre-allocated contiguous pages which are used
to populate the mapping. I did not use any of the higher order
allocators such as alloc_contig_range. Therefore, it is limited to
Just tried with a small prototype with an implementation similar to that
of alloc_gigantic_page() where we scan the zones (applicable zonelist)
for contiguous valid PFN range and try allocating with alloc_contig_range.
Will share it soon.
allocations of MAX_ORDER size. Also, the allocations should probably
Just did a quick test and it worked till 1UL << (MAX_ORDER - 1) numbers
of pages on a POWER system with the current RFC patches. As the pages
are allocated during VMA creation time, comparison to normal page fault
speed while accessing the buffer wont be fair.
be done outside mmap_sem but that was the easiest place to do it in
this quick and easy POC.
Why it should be done outside the mmap_sem, because it can take some
time ? But then VMA can just go away while we are allocating the big
chunks of pages (if we dont hold mmap_sem).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -3100,7 +3100,18 @@ static int do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */if(unlikely(anon_vma_prepare(vma)))gotooom;-page=alloc_zeroed_user_highpage_movable(vma,vmf->address);++/*+*InthespecialVM_CONTIGcase,pageshavebeenpre-allocated.So,+*simplygrabtheappropriatepre-allocatedpage.+*/+if(unlikely(vma->vm_flags&VM_CONTIG)){+VM_BUG_ON(!vma->vm_private_data);+page=((structpage*)vma->vm_private_data)++((vmf->address-vma->vm_start)/PAGE_SIZE);+}else{+page=alloc_zeroed_user_highpage_movable(vma,vmf->address);
vm_private_data should be fine. Seems like its getting used for HugeTLB,
special mappings and for shared memory as well. As long as we dont cross
these things (lets say while enabling this for file mapping etc with
MAP_CONTIG), we can keep using vm_private_data.
Add new MAP_CONTIG flag to mmap system call. Check for flag in normal
mmap flag processing. If present, pre-allocate a contiguous set of
pages to back the mapping. These pages will be used a fault time, and
the MAP_CONTIG flag implies populating the mapping at the mmap time.
Signed-off-by: Mike Kravetz <redacted>
---
include/uapi/asm-generic/mman.h | 1 +
mm/mmap.c | 94 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 95 insertions(+)
@@ -12,6 +12,7 @@#define MAP_NONBLOCK 0x10000 /* do not block on IO */#define MAP_STACK 0x20000 /* give out an address that is best suited for process/thread stacks */#define MAP_HUGETLB 0x40000 /* create a huge page mapping */+#define MAP_CONTIG 0x80000 /* back with contiguous pages *//* Bits [26:31] are reserved, see mman-common.h for MAP_HUGETLB usage */
Would it be GFP_HIGHUSER_MOVABLE instead ? Why __GFP_ZERO ? If its
coming from Buddy, every thing should have already been zeroed out
in there. Am I missing something ?
+ unsigned long order;
+
+ VM_BUG_ON_VMA(vma->vm_private_data != NULL, vma);
+ order = get_order(vma->vm_end - vma->vm_start);
+
+ /*
+ * FIXME - Incomplete implementation. For now, just handle
+ * allocations < MAX_ORDER in size. However, this should really
+ * handle arbitrary size allocations.
+ */
+ if (order >= MAX_ORDER)
+ return -ENOMEM;
+
+ vma->vm_private_data = alloc_pages_vma(gfp, order, vma, vma->vm_start,
+ numa_node_id(), false);
This is where I was experimenting for requests beyond MAX_ORDER
with alloc_contig_range().
+ if (!vma->vm_private_data)
+ return -ENOMEM;
+
+ /*
+ * split large allocation so it can be treated as individual
+ * pages when populating the mapping and at unmap time.
+ */
+ if (order) {
+ unsigned long vma_pages = (vma->vm_end - vma->vm_start) /
+ PAGE_SIZE;
+ unsigned long order_pages = 1 << order;
+ unsigned long i;
+ struct page *page = vma->vm_private_data;
+
+ split_page((struct page *)vma->vm_private_data, order);
+
+ /*
+ * 'order' rounds up size of vma to next power of 2. We
+ * will not need/use the extra pages so free them now.
+ */
+ for (i = vma_pages; i < order_pages; i++)
+ put_page(page + i);
Interesting and this should be kept there a little longer if we would like
to support expansion of this VMA to the next power of 2 which we originally
allocated for any way.
quoted hunk
+ }
+
+ return 0;
+}
+
+static void __free_vma_contig_range(struct vm_area_struct *vma)
+{
+ struct page *page = vma->vm_private_data;
+ unsigned long n_pages = (vma->vm_end - vma->vm_start) / PAGE_SIZE;
+ unsigned long i;
+
+ if (!page)
+ return;
+
+ for (i = 0; i < n_pages; i++)
+ put_page(page + i);
+}
+
+/*
* Some shared mappigns will want the pages marked read-only
* to track write events. If so, we'll downgrade vm_page_prot
* to the private version (using protection_map[] without the
@@ -1669,6 +1756,12 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_pgoff = pgoff; INIT_LIST_HEAD(&vma->anon_vma_chain);+ if (vm_flags & VM_CONTIG) {+ error = __alloc_vma_contig_range(vma);+ if (error)+ goto free_vma;+ }+
You wanted to have this outside of mmap_sem lock right ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
The following is a 'possible' way to add such functionality. I just
did what was easy and pre-allocated contiguous pages which are used
to populate the mapping. I did not use any of the higher order
allocators such as alloc_contig_range. Therefore, it is limited to
Just tried with a small prototype with an implementation similar to that
of alloc_gigantic_page() where we scan the zones (applicable zonelist)
for contiguous valid PFN range and try allocating with alloc_contig_range.
Will share it soon.
With this patch on top of the series can allocate little more than
twice of 1UL << (MAX_ORDER - 1) number of pages on POWER. But the
problem is it keeps on reducing every attempt till it reaches
1UL << (MAX_ORDER - 1). Will look into it.
@@ -28,5 +28,6 @@#define MAP_NONBLOCK 0x10000 /* do not block on IO */#define MAP_STACK 0x20000 /* give out an address that is best suited for process/thread stacks */#define MAP_HUGETLB 0x40000 /* create a huge page mapping */+#define MAP_CONTIG 0x80000 /* back with contiguous pages */#endif /* _UAPI_ASM_POWERPC_MMAN_H */
@@ -1588,11 +1642,19 @@ static long __alloc_vma_contig_range(struct vm_area_struct *vma)*allocations<MAX_ORDERinsize.However,thisshouldreally*handlearbitrarysizeallocations.*/++/*if(order>=MAX_ORDER)return-ENOMEM;-vma->vm_private_data=alloc_pages_vma(gfp,order,vma,vma->vm_start,-numa_node_id(),false);+*/++if(order>=MAX_ORDER)+vma->vm_private_data=alloc_pages_vma_contig(gfp,order,vma,+vma->vm_start,numa_node_id(),false);+else+vma->vm_private_data=alloc_pages_vma(gfp,order,vma,+vma->vm_start,numa_node_id(),false);if(!vma->vm_private_data)return-ENOMEM;--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-12 14:38:05
On Wed 11-10-17 18:46:11, Mike Kravetz wrote:
Add new MAP_CONTIG flag to mmap system call. Check for flag in normal
mmap flag processing. If present, pre-allocate a contiguous set of
pages to back the mapping. These pages will be used a fault time, and
the MAP_CONTIG flag implies populating the mapping at the mmap time.
I have only briefly read through the previous discussion and it is still
not clear to me _why_ we want such a interface. I didn't give it much
time yet but I do not think this is a good idea at all. Why? Do we want
any user to simply consume larger order memory blocks? What would
prevent from that? Also why should even userspace care about larger
memory blocks? We have huge pages (be it preallocated or transparent)
for that purpose already. Why should we add yet another another type
of physically contiguous memory. What is the guaratee of such a mapping.
Does the memory always stays contiguous? How much contiguous it will be?
Who is going to use such an interface? And probably many other
questions...
@@ -12,6 +12,7 @@#define MAP_NONBLOCK 0x10000 /* do not block on IO */#define MAP_STACK 0x20000 /* give out an address that is best suited for process/thread stacks */#define MAP_HUGETLB 0x40000 /* create a huge page mapping */+#define MAP_CONTIG 0x80000 /* back with contiguous pages *//* Bits [26:31] are reserved, see mman-common.h for MAP_HUGETLB usage */
@@ -1669,6 +1756,12 @@ unsigned long mmap_region(struct file *file, unsigned long addr,vma->vm_pgoff=pgoff;INIT_LIST_HEAD(&vma->anon_vma_chain);+if(vm_flags&VM_CONTIG){+error=__alloc_vma_contig_range(vma);+if(error)+gotofree_vma;+}+if(file){if(vm_flags&VM_DENYWRITE){error=deny_write_access(file);
@@ -1758,6 +1851,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,if(vm_flags&VM_DENYWRITE)allow_write_access(file);free_vma:+__free_vma_contig_range(vma);kmem_cache_free(vm_area_cachep,vma);unacct_error:if(charged)
--
2.13.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-12 17:20:49
On 10/12/2017 07:37 AM, Michal Hocko wrote:
On Wed 11-10-17 18:46:11, Mike Kravetz wrote:
quoted
Add new MAP_CONTIG flag to mmap system call. Check for flag in normal
mmap flag processing. If present, pre-allocate a contiguous set of
pages to back the mapping. These pages will be used a fault time, and
the MAP_CONTIG flag implies populating the mapping at the mmap time.
I have only briefly read through the previous discussion and it is still
not clear to me _why_ we want such a interface. I didn't give it much
time yet but I do not think this is a good idea at all.
Thanks for looking Michal. The primary use case comes from devices that can
realize performance benefits if operating on physically contiguous memory.
What sparked this effort was Christoph and Guy's plumbers presentation
where they showed RDMA performance benefits that could be realized with
contiguous memory. I also remember sitting in a presentation about
Intel's QuackAssist technology at Vault last year. The presenter mentioned
that their compression engine needed to be passed a physically contiguous
buffer. I asked how a user could obtain such a buffer. They said they
had a special driver/ioctl for that. Yuck! I'm guessing there are other
specific use cases. That is why I wanted to start the discussion as to
whether there should be an interface to provide this functionality.
Why? Do we want
any user to simply consume larger order memory blocks? What would
prevent from that?
We certainly would want to put restrictions in place for contiguous
memory allocations. Since it makes sense to pre-populate and lock
contiguous allocations, using the same restrictions as mlock is a start.
However, I can see the possible need for more restrictions.
Also why should even userspace care about larger
memory blocks? We have huge pages (be it preallocated or transparent)
for that purpose already. Why should we add yet another another type
The 'sweet spot' for the Mellanox RDMA example is 2GB. We can not
achieve that with huge pages (on x86) today.
What is the guaratee of such a mapping.
There is no guarantee. My suggestion is that mmap(MAP_CONTIG) would fail
with ENOMEM if a sufficiently sized contiguous area could not be found.
The caller would need to deal with failure.
Does the memory always stays contiguous? How much contiguous it will be?
Yes, it remains contiguous. It is locked in memory.
Who is going to use such an interface? And probably many other
questions...
Thanks for asking. I am just throwing out the idea of providing an interface
for doing contiguous memory allocations from user space. There are at least
two (and possibly more) devices that could benefit from such an interface.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-13 08:41:02
On Thu 12-10-17 10:19:16, Mike Kravetz wrote:
On 10/12/2017 07:37 AM, Michal Hocko wrote:
quoted
On Wed 11-10-17 18:46:11, Mike Kravetz wrote:
quoted
Add new MAP_CONTIG flag to mmap system call. Check for flag in normal
mmap flag processing. If present, pre-allocate a contiguous set of
pages to back the mapping. These pages will be used a fault time, and
the MAP_CONTIG flag implies populating the mapping at the mmap time.
I have only briefly read through the previous discussion and it is still
not clear to me _why_ we want such a interface. I didn't give it much
time yet but I do not think this is a good idea at all.
Thanks for looking Michal. The primary use case comes from devices that can
realize performance benefits if operating on physically contiguous memory.
What sparked this effort was Christoph and Guy's plumbers presentation
where they showed RDMA performance benefits that could be realized with
contiguous memory. I also remember sitting in a presentation about
Intel's QuackAssist technology at Vault last year. The presenter mentioned
that their compression engine needed to be passed a physically contiguous
buffer. I asked how a user could obtain such a buffer. They said they
had a special driver/ioctl for that. Yuck! I'm guessing there are other
specific use cases. That is why I wanted to start the discussion as to
whether there should be an interface to provide this functionality.
I would, quite contrary, suggest a device specific mmap implementation
which would guarantee both the best memory wrt. physical contiguous
aspect as well as the placement - what if the device have a restriction
on that as well?
quoted
any user to simply consume larger order memory blocks? What would
prevent from that?
We certainly would want to put restrictions in place for contiguous
memory allocations. Since it makes sense to pre-populate and lock
contiguous allocations, using the same restrictions as mlock is a start.
However, I can see the possible need for more restrictions.
Absolutely. mlock limit is per process (resp. mm) so a single user could
simply deplete large blocks. No good...
quoted
Does the memory always stays contiguous? How much contiguous it will be?
Yes, it remains contiguous. It is locked in memory.
Hmm, so hugetlb on steroids...
quoted
Who is going to use such an interface? And probably many other
questions...
Thanks for asking. I am just throwing out the idea of providing an interface
for doing contiguous memory allocations from user space. There are at least
two (and possibly more) devices that could benefit from such an interface.
I am not really convinced this is a good interface. You are basically
trying to bypass virtual memory abstraction and that is quite
contradicting the mmap API to me.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Would it be GFP_HIGHUSER_MOVABLE instead ? Why __GFP_ZERO ? If its
coming from Buddy, every thing should have already been zeroed out
in there. Am I missing something ?
Contiguous pages cannot and should not be moved. They will no longer be
contiguous then. Also the page migration code cannot handle this case.
From: Christopher Lameter <hidden> Date: 2017-10-13 15:20:10
On Fri, 13 Oct 2017, Michal Hocko wrote:
I would, quite contrary, suggest a device specific mmap implementation
which would guarantee both the best memory wrt. physical contiguous
aspect as well as the placement - what if the device have a restriction
on that as well?
Contemporary high end devices can handle all of memory. If someone does
not have the requirements to get all that hardware can give you in terms
of speed then they also wont need contiguous memory.
quoted
Yes, it remains contiguous. It is locked in memory.
Hmm, so hugetlb on steroids...
Its actually better because there is no requirements of allocation in
exacytly 2M chunks. The remainder can be used for regular 4k page
allocations.
quoted
quoted
Who is going to use such an interface? And probably many other
questions...
Thanks for asking. I am just throwing out the idea of providing an interface
for doing contiguous memory allocations from user space. There are at least
two (and possibly more) devices that could benefit from such an interface.
I am not really convinced this is a good interface. You are basically
trying to bypass virtual memory abstraction and that is quite
contradicting the mmap API to me.
This is a standardized posix interface as described in our presentation at
the plumbers conference. See the presentation on contiguous allocations.
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-13 15:28:08
On Fri 13-10-17 10:20:06, Cristopher Lameter wrote:
On Fri, 13 Oct 2017, Michal Hocko wrote:
[...]
quoted
I am not really convinced this is a good interface. You are basically
trying to bypass virtual memory abstraction and that is quite
contradicting the mmap API to me.
This is a standardized posix interface as described in our presentation at
the plumbers conference. See the presentation on contiguous allocations.
Are you trying to desing a generic interface with a very specific and HW
dependent usecase in mind?
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
--
Michal Hocko
SUSE Labs
From: Christopher Lameter <hidden> Date: 2017-10-13 15:42:44
On Fri, 13 Oct 2017, Michal Hocko wrote:
On Fri 13-10-17 10:20:06, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
[...]
quoted
quoted
I am not really convinced this is a good interface. You are basically
trying to bypass virtual memory abstraction and that is quite
contradicting the mmap API to me.
This is a standardized posix interface as described in our presentation at
the plumbers conference. See the presentation on contiguous allocations.
Are you trying to desing a generic interface with a very specific and HW
dependent usecase in mind?
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
quoted
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-13 15:47:52
On Fri 13-10-17 10:42:37, Cristopher Lameter wrote:
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:20:06, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
[...]
quoted
quoted
I am not really convinced this is a good interface. You are basically
trying to bypass virtual memory abstraction and that is quite
contradicting the mmap API to me.
This is a standardized posix interface as described in our presentation at
the plumbers conference. See the presentation on contiguous allocations.
Are you trying to desing a generic interface with a very specific and HW
dependent usecase in mind?
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
quoted
quoted
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Christopher Lameter <hidden> Date: 2017-10-13 15:56:29
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-14 16:48:48
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Guy Shattah <hidden> Date: 2017-10-15 07:50:40
On 13/10/2017 19:17, Michal Hocko wrote:
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize
the idea and implement
an already existing POSIX standard?
--
Guy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Guy Shattah <hidden> Date: 2017-10-15 08:07:52
On 13/10/2017 19:17, Michal Hocko wrote:
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety
of specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this
generic posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability
of the same sized contiguous physical memory range. That sounds like
the abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem.
There are other use cases.
That doesn't prevent providing a library function which could be
reused by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier
if mmap could give you the memory you need instead of havig numerous
drivers improvise on their own. This is in particular also useful for
numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have
access control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the idea and implement an already existing POSIX standard?
--
Guy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Pavel Machek <hidden> Date: 2017-10-15 21:59:12
Hi!
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
quoted
quoted
quoted
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
So you'd suggest using ioctl() for allocating memory?
That sounds quite ugly to me... mmap(MAP_CONTIG) is not nice, either, but better than
each driver inventing custom interface...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 08:18:11
On Sun 15-10-17 08:58:56, Pavel Machek wrote:
Hi!
quoted
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
quoted
quoted
quoted
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
So you'd suggest using ioctl() for allocating memory?
Why not using standard mmap on the device fd?
That sounds quite ugly to me... mmap(MAP_CONTIG) is not nice, either, but better than
each driver inventing custom interface...
As already pointed out elsewhere, I do not really see a different to
remap_pfn_range from the API point of view. A driver has some
requirements to the memory so those can be reflected in the mmap
implementation for the driver. I really do not see how that would be a
general interface without a lot of headache in future. Contiguous memory
is a hard requirement to guarantee or give out without risks.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 08:25:01
On Sun 15-10-17 10:50:29, Guy Shattah wrote:
On 13/10/2017 19:17, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
So you are saying that providing an API which fails randomly because of
the physically fragmented memory is OK? Users shouldn't really care
about the state of the physical memory. That is what we have the virtual
memory for.
quoted
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the
idea and implement
an already existing POSIX standard?
Because users shouldn't really care, really. We do have means to get
large memory and having a guaranteed large memory is a PITA. Just look
at hugetlb and all the issues it exposes. And that one is preallocated
and it requires admin to do a conscious decision about the amount of the
memory. You would like to establish something similar except without
bounds to the size and no pre-allowed amount by an admin. This sounds
just crazy to me.
On the other hand if you make this per-device mmap implementation you
can have both admin defined policy on who is allowed this memory and
moreover drivers can implement their fallback strategies which best suit
their needs. I really fail to see how this is any different from using
specialized mmap implementations.
I might be really wrong but I consider such a general purpose flag quite
dangerous and future maintenance burden. At least from the hugetlb/THP
history I do not see why this should be any different.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Guy Shattah <hidden> Date: 2017-10-16 09:11:17
On 16/10/2017 11:24, Michal Hocko wrote:
On Sun 15-10-17 10:50:29, Guy Shattah wrote:
quoted
On 13/10/2017 19:17, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
So you are saying that providing an API which fails randomly because of
the physically fragmented memory is OK? Users shouldn't really care
about the state of the physical memory. That is what we have the virtual
memory for.
Users still see and work with virtual addresses, just as before.
Users using the suggested API are aware that API might fail since it
involves current
system memory state. This won't be the first system call or the last one
to fail due to
reasons beyond user control. For example: any user app might fail due to
number of
open files, disk space, memory availability, network availability. All
beyond user control.
A smart user always has their ways to handle exceptions.
A typical user failing to allocate contiguous memory and May fallback to
allocating
non-contiguous memory. And by the way - even if each vendor implements
their own
methods to allocate contiguous memory then this vendor specific API
might fail too.
For the same reasons.
quoted
quoted
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the
idea and implement
an already existing POSIX standard?
Because users shouldn't really care, really. We do have means to get
large memory and having a guaranteed large memory is a PITA. Just look
at hugetlb and all the issues it exposes. And that one is preallocated
and it requires admin to do a conscious decision about the amount of the
memory. You would like to establish something similar except without
bounds to the size and no pre-allowed amount by an admin. This sounds
just crazy to me.
Users do care about the performance they get using devices which benefit
from contiguous memory allocation.
Assuming that user requires 700Mb of contiguous memory. Then why allocate
giant (1GB) page when you can allocate 700Mb out of the 1GB and put the
rest of the
300Mb back in the huge-pages/small-pages pool?
On the other hand if you make this per-device mmap implementation you
can have both admin defined policy on who is allowed this memory and
moreover drivers can implement their fallback strategies which best suit
their needs. I really fail to see how this is any different from using
specialized mmap implementations.
We tried doing it in the past. but the maintainer gave us a very good
argument:
" If you want to support anonymous mmaps to allocate large contiguous
pages work with the MM folks on providing that in a generic fashion."
After discussing it with people who have the same requirements as we do -
I totally agree with him
http://comments.gmane.org/gmane.linux.drivers.rdma/31467
I might be really wrong but I consider such a general purpose flag quite
dangerous and future maintenance burden. At least from the hugetlb/THP
history I do not see why this should be any different.
Could you please elaborate why is it dangerous and future maintenance
burden?
Thanks.
From: Pavel Machek <hidden> Date: 2017-10-16 09:54:51
On Mon 2017-10-16 10:18:04, Michal Hocko wrote:
On Sun 15-10-17 08:58:56, Pavel Machek wrote:
quoted
Hi!
quoted
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
quoted
quoted
quoted
The contiguous allocations are particularly useful for the RDMA API which
allows registering user space memory with devices.
then make those devices expose an implementation of an mmap which does
that. You would get both a proper access control (via fd), accounting
and others.
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
So you'd suggest using ioctl() for allocating memory?
Why not using standard mmap on the device fd?
No, sorry, that's something very different work, right? Lets say I
have a disk, and I'd like to write to it, using continguous memory for
performance.
So I mmap(MAP_CONTIG) 1GB working of working memory, prefer some data
structures there, maybe recieve from network, then decide to write
some and not write some other.
mmap(sda) does something very different... Everything you write to
that mmap will eventually go to the disk, and you don't have complete
control when.
Also, you can do mmap(MAP_CONTIG) and use that to both disk and
network. That would not work with mmap(sda) and mmap(eth0)...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
From: Michal Nazarewicz <hidden> Date: 2017-10-16 10:33:14
On Sun, Oct 15 2017, Guy Shattah wrote:
Why have several driver specific implementation if you can generalize
the idea and implement an already existing POSIX standard?
Why is there a need for contiguous allocation?
CPU cares only to the point of huge pages and there’s already an effort
in the kernel to allocate huge pages transparently without user space
being aware of it.
If not CPU than various devices all of which may have very different
needs. Some may be behind an IO MMU. Some may support DMA. Some may
indeed require physically continuous memory. How is user space to know?
Furthermore, user space does not care whether allocation is physically
contiguous or not. What it cares about is whether given allocation can
be passed as a buffer to a particular device.
If generalisation is the issue, then the solution is to define a common
API where user-space can allocate memory *in the context of* a device.
This provides a ‘give me memory I can use for this device’ request which
is what user space really wants.
So yeah, like others in this thread, the reason for this change alludes
me. On the other hand, I don’t care much so I’ll limit myself to this
one message.
--
Best regards
ミハウ “𝓶𝓲𝓷𝓪86” ナザレヴイツ
«If at first you don’t succeed, give up skydiving»
If generalisation is the issue, then the solution is to define a common
API where user-space can allocate memory *in the context of* a device.
This provides a ‘give me memory I can use for this device’ request which
is what user space really wants.
Do you suggest to add a whole new common API instead of merely adding a
flag to existing one?
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 12:18:15
On Mon 16-10-17 11:54:47, Pavel Machek wrote:
On Mon 2017-10-16 10:18:04, Michal Hocko wrote:
quoted
On Sun 15-10-17 08:58:56, Pavel Machek wrote:
[...]
quoted
quoted
So you'd suggest using ioctl() for allocating memory?
Why not using standard mmap on the device fd?
No, sorry, that's something very different work, right? Lets say I
have a disk, and I'd like to write to it, using continguous memory for
performance.
So I mmap(MAP_CONTIG) 1GB working of working memory, prefer some data
structures there, maybe recieve from network, then decide to write
some and not write some other.
Why would you want this?
--
Michal Hocko
SUSE Labs
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 12:33:21
On Mon 16-10-17 12:11:04, Guy Shattah wrote:
On 16/10/2017 11:24, Michal Hocko wrote:
quoted
On Sun 15-10-17 10:50:29, Guy Shattah wrote:
quoted
On 13/10/2017 19:17, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
So you are saying that providing an API which fails randomly because of
the physically fragmented memory is OK? Users shouldn't really care
about the state of the physical memory. That is what we have the virtual
memory for.
Users still see and work with virtual addresses, just as before.
Users using the suggested API are aware that API might fail since it
involves current system memory state. This won't be the first system
call or the last one to fail due to reasons beyond user control. For
example: any user app might fail due to number of open files, disk
space, memory availability, network availability. All beyond user
control.
But the memory fragmentation is not something that directly map to the
memory usage. As such it behaves more or less randomly to the memory
utilization (see the difference to examples mentioned above?). It
depends on many other things basically rendering such an API to be
useless unless you guarantee that the large part of the memory is
movable.
A smart user always has their ways to handle exceptions. A typical
user failing to allocate contiguous memory and May fallback to
allocating non-contiguous memory. And by the way - even if each vendor
implements their own methods to allocate contiguous memory then this
vendor specific API might fail too. For the same reasons.
yes the kernel side mmap implementation would have to care about this as
well. Nobody is questioning that part. I am just questioning such a
generic purpouse API is reasonable.
quoted
quoted
quoted
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the
idea and implement
an already existing POSIX standard?
Because users shouldn't really care, really. We do have means to get
large memory and having a guaranteed large memory is a PITA. Just look
at hugetlb and all the issues it exposes. And that one is preallocated
and it requires admin to do a conscious decision about the amount of the
memory. You would like to establish something similar except without
bounds to the size and no pre-allowed amount by an admin. This sounds
just crazy to me.
Users do care about the performance they get using devices which
benefit from contiguous memory allocation. Assuming that user
requires 700Mb of contiguous memory. Then why allocate giant (1GB)
page when you can allocate 700Mb out of the 1GB and put the rest of
the 300Mb back in the huge-pages/small-pages pool?
I believe I have explained that part. Large pages are under admin
control and responsibility. If you get a free ticket to large memory to
any user who can pin that memory then you are in serious troubles.
quoted
On the other hand if you make this per-device mmap implementation you
can have both admin defined policy on who is allowed this memory and
moreover drivers can implement their fallback strategies which best suit
their needs. I really fail to see how this is any different from using
specialized mmap implementations.
We tried doing it in the past. but the maintainer gave us a very good
argument:
" If you want to support anonymous mmaps to allocate large contiguous
pages work with the MM folks on providing that in a generic fashion."
Well, we can provide a generic library functions for your driver to use
so that you do not have to care about implementation details but I do
not think exposing this API to the userspace in a generic fashion is a
good idea. Especially when the only usecase that has been thought
through so far seems to be a very special HW optimiztion.
I might be really wrong but I consider such a general purpose flag quite
dangerous and future maintenance burden. At least from the hugetlb/THP
history I do not see why this should be any different.
Could you please elaborate why is it dangerous and future maintenance
burden?
Providing large contiguous memory ranges is not easy and we actually do
not have any reliable way to offer such a functionality for the kernel
users because we assume they are not that many. Basically anything
larger than order-3 is best effort. Even changes constant improvements
of the compaction still leaves us with something we cannot fully rely
on. And now you want to expose this to the userspace with basically
arbitrary memory sizes to be supported?
But putting that aside. Pinning a lot of memory might cause many
performance issues and misbehavior. There are still kernel users
who need high order memory to work properly. On top of that you are
basically allowing an untrusted user to deplete higher order pages very
easily unless there is a clever way to enforce per user limit on this.
That being said, the list is far from being complete, I am pretty sure
more would pop out if I thought more thoroughly. The bottom line is that
while I see many problems to actually implement this feature and
maintain it longterm I simply do not see a large benefit outside of a
very specific HW.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Christopher Lameter <hidden> Date: 2017-10-16 16:00:25
On Mon, 16 Oct 2017, Michal Hocko wrote:
But putting that aside. Pinning a lot of memory might cause many
performance issues and misbehavior. There are still kernel users
who need high order memory to work properly. On top of that you are
basically allowing an untrusted user to deplete higher order pages very
easily unless there is a clever way to enforce per user limit on this.
We already have that issue and have ways to control that by tracking
pinned and mlocked pages as well as limits on their allocations.
That being said, the list is far from being complete, I am pretty sure
more would pop out if I thought more thoroughly. The bottom line is that
while I see many problems to actually implement this feature and
maintain it longterm I simply do not see a large benefit outside of a
very specific HW.
There is not much new here in terms of problems. The hardware that
needs this seems to become more and more plentiful. That is why we need a
generic implementation.
From: Christopher Lameter <hidden> Date: 2017-10-16 16:02:29
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
So I mmap(MAP_CONTIG) 1GB working of working memory, prefer some data
structures there, maybe recieve from network, then decide to write
some and not write some other.
Why would you want this?
Because we are receiving a 1GB block of data and then wan to write it to
disk. Maybe we want to modify things a bit and may not write all that we
received.
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 17:34:06
On Mon 16-10-17 11:02:24, Cristopher Lameter wrote:
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
quoted
So I mmap(MAP_CONTIG) 1GB working of working memory, prefer some data
structures there, maybe recieve from network, then decide to write
some and not write some other.
Why would you want this?
Because we are receiving a 1GB block of data and then wan to write it to
disk. Maybe we want to modify things a bit and may not write all that we
received.
And why do you need that in a single contiguous numbers? If performance,
do you have any numbers that would clearly tell the difference?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 17:42:35
On Mon 16-10-17 11:00:19, Cristopher Lameter wrote:
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
But putting that aside. Pinning a lot of memory might cause many
performance issues and misbehavior. There are still kernel users
who need high order memory to work properly. On top of that you are
basically allowing an untrusted user to deplete higher order pages very
easily unless there is a clever way to enforce per user limit on this.
We already have that issue and have ways to control that by tracking
pinned and mlocked pages as well as limits on their allocations.
Ohh, it is very different because mlock limit is really small (64kB)
which is not even close to what this is supposed to be about. Moreover
mlock doesn't prevent from migration and so it doesn't prevent
compaction to form higher order allocations.
Really, this is just too dangerous without a deep consideration of all
the potential consequences. The more I am thinking about this the more I
am convinced that this all should be driver specific mmap based thing.
If it turns out to be too restrictive over time and there are more
experiences about the usage we can consider thinking about a more
generic API. But starting from the generic MAP_ flag is just asking for
problems.
quoted
That being said, the list is far from being complete, I am pretty sure
more would pop out if I thought more thoroughly. The bottom line is that
while I see many problems to actually implement this feature and
maintain it longterm I simply do not see a large benefit outside of a
very specific HW.
There is not much new here in terms of problems. The hardware that
needs this seems to become more and more plentiful. That is why we need a
generic implementation.
It would really help to name that HW and other potential usecases
independent on the HW because I am rather skeptical about the
_plentiful_ part. And so I really do not see any foundation to claim
the generic part. Because, fundamentally, it is the HW which requires
the specific memory placement/physically contiguous range etc. So the
generic implementation doesn't really make sense in such a context.
--
Michal Hocko
SUSE Labs
From: Mike Kravetz <hidden> Date: 2017-10-16 17:46:17
On 10/15/2017 12:50 AM, Guy Shattah wrote:
On 13/10/2017 19:17, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There is a generic posix interface that could we used for a variety of
specific hardware dependent use cases.
Yes you wrote that already and my counter argument was that this generic
posix interface shouldn't bypass virtual memory abstraction.
It does do that? In what way?
availability of the virtual address space depends on the availability of
the same sized contiguous physical memory range. That sounds like the
abstraction is gone to large part to me.
In what way? userspace users will still be working with virtual memory.
quoted
quoted
quoted
quoted
There are numerous RDMA devices that would all need the mmap
implementation. And this covers only the needs of one subsystem. There are
other use cases.
That doesn't prevent providing a library function which could be reused
by all those drivers. Nothing really too much different from
remap_pfn_range.
And then in all the other use cases as well. It would be much easier if
mmap could give you the memory you need instead of havig numerous drivers
improvise on their own. This is in particular also useful
for numerous embedded use cases where you need contiguous memory.
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the idea and implement
an already existing POSIX standard?
Just to be clear, the posix standard talks about a typed memory object.
The suggested implementation has one create a connection to the memory
object to receive a fd, then use mmap as usual to get a mapping backed
by contiguous pages/memory. Of course, this type of implementation is
not a requirement. However, this type of implementation looks quite a
bit like hugetlbfs today.
- Both require opening a special file/device, and then calling mmap on
the returned fd. You can technically use mmap(MAP_HUGETLB), but that
still ends up using hugetbfs. BTW, there was resistance to adding the
MAP_HUGETLB flag to mmap.
- Allocation of contiguous memory is much like 'on demand' allocation of
huge pages. There are some (not many) users that use this model. They
attempt to allocate huge pages on demand, and if not available fall back
to base pages. This is how contiguous allocations would need to work.
Of course, most hugetlbfs users pre-allocate pages for their use, and
this 'might' be something useful for contiguous allocations as well.
I wonder if going down the path of a separate devide/filesystem/etc for
contiguous allocations might be a better option. It would keep the
implementation somewhat separate. However, I would then be afraid that
we end up with another 'separate/special vm' as in the case of hugetlbfs
today.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Christopher Lameter <hidden> Date: 2017-10-16 17:53:48
On Mon, 16 Oct 2017, Michal Hocko wrote:
On Mon 16-10-17 11:02:24, Cristopher Lameter wrote:
quoted
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
quoted
So I mmap(MAP_CONTIG) 1GB working of working memory, prefer some data
structures there, maybe recieve from network, then decide to write
some and not write some other.
Why would you want this?
Because we are receiving a 1GB block of data and then wan to write it to
disk. Maybe we want to modify things a bit and may not write all that we
received.
And why do you need that in a single contiguous numbers? If performance,
do you have any numbers that would clearly tell the difference?
Again we have that in the presentation. Why keep asking the same question
if you already have the answer multiple times?
1G of data requires 250000 page structs to handle if the memory is not
contiguous. This is more than most controllers can support and thus the
overhead will dominate I/O. Also the scatter gather lists will cover lots
of linked 4k pages even to manage.
And in practice we already have multiple gigabytes per requests which
makes it even more severe. You cannot do a "cp" operation anymore. Instead
you need to have special code that allocates huge pages, does direct I/O
etc etc,
From: Christopher Lameter <hidden> Date: 2017-10-16 17:56:48
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
We already have that issue and have ways to control that by tracking
pinned and mlocked pages as well as limits on their allocations.
Ohh, it is very different because mlock limit is really small (64kB)
which is not even close to what this is supposed to be about. Moreover
mlock doesn't prevent from migration and so it doesn't prevent
compaction to form higher order allocations.
The mlock limit is configurable. There is a tracking of pinned pages as
well.
Really, this is just too dangerous without a deep consideration of all
the potential consequences. The more I am thinking about this the more I
am convinced that this all should be driver specific mmap based thing.
If it turns out to be too restrictive over time and there are more
experiences about the usage we can consider thinking about a more
generic API. But starting from the generic MAP_ flag is just asking for
problems.
This issue is already present with the pinning of lots of memory via the
RDMA API when in use for large gigabyte ranges. There is nothing new aside
from memory being contiguous with this approach.
quoted
There is not much new here in terms of problems. The hardware that
needs this seems to become more and more plentiful. That is why we need a
generic implementation.
It would really help to name that HW and other potential usecases
independent on the HW because I am rather skeptical about the
_plentiful_ part. And so I really do not see any foundation to claim
the generic part. Because, fundamentally, it is the HW which requires
the specific memory placement/physically contiguous range etc. So the
generic implementation doesn't really make sense in such a context.
RDMA hardware? Storage interfaces? Look at what the RDMA subsystem
and storage (NVME?) support.
This is not a hardware specific thing but a reflection of the general
limitations of the exiting 4k page struct scheme that limits performance
and causes severe pressure on I/O devices.
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 18:07:55
On Mon 16-10-17 10:43:38, Mike Kravetz wrote:
On 10/15/2017 12:50 AM, Guy Shattah wrote:
quoted
On 13/10/2017 19:17, Michal Hocko wrote:
[...]
quoted
quoted
But a generic implementation would have to deal with many issues as
already mentioned. If you make this driver specific you can have access
control based on fd etc... I really fail to see how this is any
different from remap_pfn_range.
Why have several driver specific implementation if you can generalize the idea and implement
an already existing POSIX standard?
Just to be clear, the posix standard talks about a typed memory object.
The suggested implementation has one create a connection to the memory
object to receive a fd, then use mmap as usual to get a mapping backed
by contiguous pages/memory. Of course, this type of implementation is
not a requirement.
I am not sure that POSIC standard for typed memory is easily
implementable in Linux. Does any OS actually implement this API?
However, this type of implementation looks quite a
bit like hugetlbfs today.
- Both require opening a special file/device, and then calling mmap on
the returned fd. You can technically use mmap(MAP_HUGETLB), but that
still ends up using hugetbfs. BTW, there was resistance to adding the
MAP_HUGETLB flag to mmap.
And I think we shouldn't really shape any API based on hugetlb.
- Allocation of contiguous memory is much like 'on demand' allocation of
huge pages. There are some (not many) users that use this model. They
attempt to allocate huge pages on demand, and if not available fall back
to base pages. This is how contiguous allocations would need to work.
Of course, most hugetlbfs users pre-allocate pages for their use, and
this 'might' be something useful for contiguous allocations as well.
But there is still admin configuration required to consume memory from
the pool or overcommit that pool.
I wonder if going down the path of a separate devide/filesystem/etc for
contiguous allocations might be a better option. It would keep the
implementation somewhat separate. However, I would then be afraid that
we end up with another 'separate/special vm' as in the case of hugetlbfs
today.
That depends on who is actually going to use the contiguous memory. If
we are talking about drivers to communication to the userspace then
using driver specific fd with its mmap implementation then we do not
need any special fs nor a seperate infrastructure. Well except for a
library function to handle the MM side of the thing.
If we really need a general purpose physical contiguous memory allocator
then I would agree that using MAP_ flag might be a way to go but that
would require a very careful consideration of who is allowed to allocate
and how much/large blocks. I do not see a good fit to conveying that
information to the kernel right now. Moreover, and most importantly, I
haven't heard any sound usecase for such a functionality in the first
place. There is some hand waving about performance but there are no real
numbers to back those claims AFAIK. Not to mention a serious
consideration of potential consequences of the whole MM.
--
Michal Hocko
SUSE Labs
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 18:18:02
On Mon 16-10-17 12:56:43, Cristopher Lameter wrote:
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
quoted
We already have that issue and have ways to control that by tracking
pinned and mlocked pages as well as limits on their allocations.
Ohh, it is very different because mlock limit is really small (64kB)
which is not even close to what this is supposed to be about. Moreover
mlock doesn't prevent from migration and so it doesn't prevent
compaction to form higher order allocations.
The mlock limit is configurable. There is a tracking of pinned pages as
well.
I am not aware of any such generic tracking API. The attempt by Peter
has never been merged. So what we have right now is just an adhoc
tracking...
quoted
Really, this is just too dangerous without a deep consideration of all
the potential consequences. The more I am thinking about this the more I
am convinced that this all should be driver specific mmap based thing.
If it turns out to be too restrictive over time and there are more
experiences about the usage we can consider thinking about a more
generic API. But starting from the generic MAP_ flag is just asking for
problems.
This issue is already present with the pinning of lots of memory via the
RDMA API when in use for large gigabyte ranges.
... like in those
There is nothing new aside
from memory being contiguous with this approach.
which makes a hell of a difference. Once you allow to pin larger blocks
of memory you make the whole compaction hopelessly ineffective.
quoted
quoted
There is not much new here in terms of problems. The hardware that
needs this seems to become more and more plentiful. That is why we need a
generic implementation.
It would really help to name that HW and other potential usecases
independent on the HW because I am rather skeptical about the
_plentiful_ part. And so I really do not see any foundation to claim
the generic part. Because, fundamentally, it is the HW which requires
the specific memory placement/physically contiguous range etc. So the
generic implementation doesn't really make sense in such a context.
RDMA hardware? Storage interfaces? Look at what the RDMA subsystem
and storage (NVME?) support.
This is not a hardware specific thing but a reflection of the general
limitations of the exiting 4k page struct scheme that limits performance
and causes severe pressure on I/O devices.
This is something more for storage people to comment. I expect (NVME)
storage to use DAX and it support for large and direct access. Nothing
really prevents RDMA HW to provide mmap implementation to use contiguous
pages, we already provide an API to allocate large memory.
--
Michal Hocko
SUSE Labs
From: Mike Kravetz <hidden> Date: 2017-10-16 20:33:27
On 10/16/2017 11:07 AM, Michal Hocko wrote:
On Mon 16-10-17 10:43:38, Mike Kravetz wrote:
quoted
Just to be clear, the posix standard talks about a typed memory object.
The suggested implementation has one create a connection to the memory
object to receive a fd, then use mmap as usual to get a mapping backed
by contiguous pages/memory. Of course, this type of implementation is
not a requirement.
I am not sure that POSIC standard for typed memory is easily
implementable in Linux. Does any OS actually implement this API?
A quick search only reveals Blackberry QNX and PlayBook OS.
Also somewhat related. In a earlier thread someone pointed out this
out of tree module used for contiguous allocations in SOC (and other?)
environments. It even has the option of making use of CMA.
http://processors.wiki.ti.com/index.php/CMEM_Overview
quoted
However, this type of implementation looks quite a
bit like hugetlbfs today.
- Both require opening a special file/device, and then calling mmap on
the returned fd. You can technically use mmap(MAP_HUGETLB), but that
still ends up using hugetbfs. BTW, there was resistance to adding the
MAP_HUGETLB flag to mmap.
And I think we shouldn't really shape any API based on hugetlb.
Agree. I only wanted to point out the similarities.
But, it does make me wonder how much of a benefit hugetlb 1G pages would
make in the the RDMA performance comparison. The table in the presentation
show a average speedup of something like 27% (or so) for contiguous allocation
which I assume are 2GB in size. Certainly, using hugetlb is not the ideal
case, just wondering if it does help and how much.
quoted
- Allocation of contiguous memory is much like 'on demand' allocation of
huge pages. There are some (not many) users that use this model. They
attempt to allocate huge pages on demand, and if not available fall back
to base pages. This is how contiguous allocations would need to work.
Of course, most hugetlbfs users pre-allocate pages for their use, and
this 'might' be something useful for contiguous allocations as well.
But there is still admin configuration required to consume memory from
the pool or overcommit that pool.
quoted
I wonder if going down the path of a separate devide/filesystem/etc for
contiguous allocations might be a better option. It would keep the
implementation somewhat separate. However, I would then be afraid that
we end up with another 'separate/special vm' as in the case of hugetlbfs
today.
That depends on who is actually going to use the contiguous memory. If
we are talking about drivers to communication to the userspace then
using driver specific fd with its mmap implementation then we do not
need any special fs nor a seperate infrastructure. Well except for a
library function to handle the MM side of the thing.
If we embed this functionality into device specific mmap calls it will
closely tie the usage to the devices. However, don't we still have to
worry about potential interaction with other parts of the mm as you mention
below? I guess that would be the library function and how it is used
by drivers.
--
Mike Kravetz
If we really need a general purpose physical contiguous memory allocator
then I would agree that using MAP_ flag might be a way to go but that
would require a very careful consideration of who is allowed to allocate
and how much/large blocks. I do not see a good fit to conveying that
information to the kernel right now. Moreover, and most importantly, I
haven't heard any sound usecase for such a functionality in the first
place. There is some hand waving about performance but there are no real
numbers to back those claims AFAIK. Not to mention a serious
consideration of potential consequences of the whole MM.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-16 20:58:30
On Mon 16-10-17 13:32:45, Mike Kravetz wrote:
On 10/16/2017 11:07 AM, Michal Hocko wrote:
[...]
quoted
That depends on who is actually going to use the contiguous memory. If
we are talking about drivers to communication to the userspace then
using driver specific fd with its mmap implementation then we do not
need any special fs nor a seperate infrastructure. Well except for a
library function to handle the MM side of the thing.
If we embed this functionality into device specific mmap calls it will
closely tie the usage to the devices. However, don't we still have to
worry about potential interaction with other parts of the mm as you mention
below? I guess that would be the library function and how it is used
by drivers.
Yes, those problems with pinning the amount of contiguous memory are
simply inherent. You have to be really careful when allowing to reserve large
partions of the contiguous memory. Especially if this is going to be a
very dynamic allocator. The main advantage of the per
device mmap is that it has its access control by default via file
permissions. You can simply rule the untrusted user out of the game. You
can also implement the per device usage limits. So you have some tools to
keep the usage under leash and evaluate potential costs vs. benefits.
That sounds to me much more safer than a generic API which would have
a tricky accounting and access control restrictions.
--
Michal Hocko
SUSE Labs
From: Laura Abbott <hidden> Date: 2017-10-16 21:03:24
On 10/16/2017 01:32 PM, Mike Kravetz wrote:
On 10/16/2017 11:07 AM, Michal Hocko wrote:
quoted
On Mon 16-10-17 10:43:38, Mike Kravetz wrote:
quoted
Just to be clear, the posix standard talks about a typed memory object.
The suggested implementation has one create a connection to the memory
object to receive a fd, then use mmap as usual to get a mapping backed
by contiguous pages/memory. Of course, this type of implementation is
not a requirement.
I am not sure that POSIC standard for typed memory is easily
implementable in Linux. Does any OS actually implement this API?
A quick search only reveals Blackberry QNX and PlayBook OS.
Also somewhat related. In a earlier thread someone pointed out this
out of tree module used for contiguous allocations in SOC (and other?)
environments. It even has the option of making use of CMA.
http://processors.wiki.ti.com/index.php/CMEM_Overview
If we're at the point where we're discussing CMEM, I'd like to
point out that ion (drivers/staging/android/ion) already provides an
ioctl interface to allocate CMA and other types of memory. It's
mostly used for Android as the name implies. I don't pretend the
interface is perfect but it could be useful as a discussion point
for allocation interfaces.
Thanks,
Laura
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-16 21:19:20
On 10/16/2017 02:03 PM, Laura Abbott wrote:
On 10/16/2017 01:32 PM, Mike Kravetz wrote:
quoted
On 10/16/2017 11:07 AM, Michal Hocko wrote:
quoted
On Mon 16-10-17 10:43:38, Mike Kravetz wrote:
quoted
Just to be clear, the posix standard talks about a typed memory object.
The suggested implementation has one create a connection to the memory
object to receive a fd, then use mmap as usual to get a mapping backed
by contiguous pages/memory. Of course, this type of implementation is
not a requirement.
I am not sure that POSIC standard for typed memory is easily
implementable in Linux. Does any OS actually implement this API?
A quick search only reveals Blackberry QNX and PlayBook OS.
Also somewhat related. In a earlier thread someone pointed out this
out of tree module used for contiguous allocations in SOC (and other?)
environments. It even has the option of making use of CMA.
http://processors.wiki.ti.com/index.php/CMEM_Overview
If we're at the point where we're discussing CMEM, I'd like to
point out that ion (drivers/staging/android/ion) already provides an
ioctl interface to allocate CMA and other types of memory. It's
mostly used for Android as the name implies. I don't pretend the
interface is perfect but it could be useful as a discussion point
for allocation interfaces.
Thanks Laura,
I was just pointing out other use cases where people thought contiguous
allocations were useful. And, it was useful enough that someone actually
wrote code to make it happen.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Agree. I only wanted to point out the similarities.
But, it does make me wonder how much of a benefit hugetlb 1G pages would
make in the the RDMA performance comparison. The table in the presentation
show a average speedup of something like 27% (or so) for contiguous allocation
which I assume are 2GB in size. Certainly, using hugetlb is not the ideal
case, just wondering if it does help and how much.
Good point. If somebody cares about performance benefits of contiguous
memory wrt device access, they would probably want also the TLB
performance benefits of huge pages.
From: Guy Shattah <hidden> Date: 2017-10-17 10:50:09
quoted
On 16/10/2017 11:24, Michal Hocko wrote:
quoted
On Sun 15-10-17 10:50:29, Guy Shattah wrote:
quoted
On 13/10/2017 19:17, Michal Hocko wrote:
quoted
On Fri 13-10-17 10:56:13, Cristopher Lameter wrote:
quoted
On Fri, 13 Oct 2017, Michal Hocko wrote:
quoted
quoted
There are numerous RDMA devices that would all need the
mmap implementation. And this covers only the needs of one
subsystem. There are other use cases.
That doesn't prevent providing a library function which
could be reused by all those drivers. Nothing really too
much different from remap_pfn_range.
And then in all the other use cases as well. It would be much
easier if mmap could give you the memory you need instead of
havig numerous drivers improvise on their own. This is in
particular also useful for numerous embedded use cases where you
need contiguous memory.
quoted
quoted
quoted
quoted
But a generic implementation would have to deal with many issues
as already mentioned. If you make this driver specific you can
have access control based on fd etc... I really fail to see how
this is any different from remap_pfn_range.
Why have several driver specific implementation if you can
generalize the idea and implement an already existing POSIX
standard?
Because users shouldn't really care, really. We do have means to get
large memory and having a guaranteed large memory is a PITA. Just
look at hugetlb and all the issues it exposes. And that one is
preallocated and it requires admin to do a conscious decision about
the amount of the memory. You would like to establish something
similar except without bounds to the size and no pre-allowed amount
by an admin. This sounds just crazy to me.
Users do care about the performance they get using devices which
benefit from contiguous memory allocation. Assuming that user
requires 700Mb of contiguous memory. Then why allocate giant (1GB)
page when you can allocate 700Mb out of the 1GB and put the rest of
the 300Mb back in the huge-pages/small-pages pool?
I believe I have explained that part. Large pages are under admin control and
responsibility. If you get a free ticket to large memory to any user who can
pin that memory then you are in serious troubles.
quoted
quoted
On the other hand if you make this per-device mmap implementation
you can have both admin defined policy on who is allowed this memory
and moreover drivers can implement their fallback strategies which
best suit their needs. I really fail to see how this is any
different from using specialized mmap implementations.
We tried doing it in the past. but the maintainer gave us a very good
argument:
" If you want to support anonymous mmaps to allocate large contiguous
pages work with the MM folks on providing that in a generic fashion."
Well, we can provide a generic library functions for your driver to use so that
you do not have to care about implementation details but I do not think
exposing this API to the userspace in a generic fashion is a good idea.
Especially when the only usecase that has been thought through so far seems
to be a very special HW optimiztion.
Are you going to be OK with kernel API which implements contiguous memory allocation?
Possibly with mmap style? Many drivers could utilize it instead of having their own weird
and possibly non-standard way to allocate contiguous memory.
Such API won't be available for user space.
We can begin with implementing kernel API and postpone the userspace api discussion for a future date.
if it is sufficient. We might not have to discuss it at all.
quoted
After discussing it with people who have the same requirements as we
do - I totally agree with him
I might be really wrong but I consider such a general purpose flag
quite dangerous and future maintenance burden. At least from the
hugetlb/THP history I do not see why this should be any different.
Could you please elaborate why is it dangerous and future maintenance
burden?
Providing large contiguous memory ranges is not easy and we actually do not
have any reliable way to offer such a functionality for the kernel users
because we assume they are not that many. Basically anything larger than
order-3 is best effort. Even changes constant improvements of the
compaction still leaves us with something we cannot fully rely on. And now
you want to expose this to the userspace with basically arbitrary memory
sizes to be supported?
But putting that aside. Pinning a lot of memory might cause many
performance issues and misbehavior. There are still kernel users who need
high order memory to work properly. On top of that you are basically
allowing an untrusted user to deplete higher order pages very easily unless
there is a clever way to enforce per user limit on this.
My previous suggestion prevents untrusted userspace code.
Guy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Michal Hocko <mhocko@kernel.org> Date: 2017-10-17 10:59:27
On Tue 17-10-17 10:50:02, Guy Shattah wrote:
[...]
quoted
Well, we can provide a generic library functions for your driver to use so that
you do not have to care about implementation details but I do not think
exposing this API to the userspace in a generic fashion is a good idea.
Especially when the only usecase that has been thought through so far seems
to be a very special HW optimiztion.
Are you going to be OK with kernel API which implements contiguous
memory allocation?
We already do have alloc_contig_range. It is a dumb allocator so it is
not very suitable for short term allocations.
Possibly with mmap style? Many drivers could utilize it instead of
having their own weird and possibly non-standard way to allocate
contiguous memory. Such API won't be available for user space.
Yes, an mmap helper which performs and enforces some accounting would be a
good start.
We can begin with implementing kernel API and postpone the userspace
api discussion for a future date. if it is sufficient. We might not
have to discuss it at all.
Yeah, that was my thinking as well.
--
Michal Hocko
SUSE Labs
From: Michal Nazarewicz <hidden> Date: 2017-10-17 13:22:47
On Tue, Oct 17 2017, Guy Shattah wrote:
Are you going to be OK with kernel API which implements contiguous
memory allocation? Possibly with mmap style? Many drivers could
utilize it instead of having their own weird and possibly non-standard
way to allocate contiguous memory. Such API won't be available for
user space.
What you describe sounds like CMA. It may be far from perfect but it’s
there already and drivers which need contiguous memory can allocate it.
--
Best regards
ミハウ “𝓶𝓲𝓷𝓪86” ナザレヴイツ
«If at first you don’t succeed, give up skydiving»
From: Guy Shattah <hidden> Date: 2017-10-17 14:21:02
On Tue, Oct 17 2017, Guy Shattah wrote:
quoted
Are you going to be OK with kernel API which implements contiguous
memory allocation? Possibly with mmap style? Many drivers could
utilize it instead of having their own weird and possibly non-standard
way to allocate contiguous memory. Such API won't be available for
user space.
What you describe sounds like CMA. It may be far from perfect but it’s there
already and drivers which need contiguous memory can allocate it.
1. CMA has to preconfigured. We're suggesting mechanism that works 'out of the box'
2. Due to the pre-allocation techniques CMA imposes limitation on maximum
allocated memory. RDMA users often require 1Gb or more, sometimes more.
3. CMA reserves memory in advance, our suggestion is using existing kernel memory
mechanisms (THP for example) to allocate memory.
Guy
Are you going to be OK with kernel API which implements contiguous
memory allocation? Possibly with mmap style? Many drivers could
utilize it instead of having their own weird and possibly non-standard
way to allocate contiguous memory. Such API won't be available for
user space.
What you describe sounds like CMA. It may be far from perfect but it’s there
already and drivers which need contiguous memory can allocate it.
1. CMA has to preconfigured. We're suggesting mechanism that works 'out of the box'
2. Due to the pre-allocation techniques CMA imposes limitation on maximum
allocated memory. RDMA users often require 1Gb or more, sometimes more.
3. CMA reserves memory in advance, our suggestion is using existing kernel memory
mechanisms (THP for example) to allocate memory.
You can already use THP, right? madvise(MADV_HUGEPAGE) increases your
chances to get the huge pages. Then you can mlock() them if you want.
And you get the TLB benefits. There's no guarantee of course, but you
shouldn't require a guarantee for MMAP_CONTIG anyway, because it's for
performance reasons, not functionality. So either MMAP_CONTIG would have
to fallback itself, or the userspace caller. Or would your scenario
rather fail than perform suboptimally?
Guy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-17 18:24:27
On 10/17/2017 07:20 AM, Guy Shattah wrote:
quoted
On Tue, Oct 17 2017, Guy Shattah wrote:
quoted
Are you going to be OK with kernel API which implements contiguous
memory allocation? Possibly with mmap style? Many drivers could
utilize it instead of having their own weird and possibly non-standard
way to allocate contiguous memory. Such API won't be available for
user space.
What you describe sounds like CMA. It may be far from perfect but it’s there
already and drivers which need contiguous memory can allocate it.
1. CMA has to preconfigured. We're suggesting mechanism that works 'out of the box'
2. Due to the pre-allocation techniques CMA imposes limitation on maximum
allocated memory. RDMA users often require 1Gb or more, sometimes more.
3. CMA reserves memory in advance, our suggestion is using existing kernel memory
mechanisms (THP for example) to allocate memory.
I would not totally rule out the use of CMA. I like the way that it reserves
memory, but does not prohibit use by others. In addition, there can be
device (or purpose) specific reservations.
However, since reservations need to happen quite early it is often done on
the kernel command line. IMO, this should be avoided if possible. There
are interfaces for arch specific code to make reservations. I do not know
the system initialization sequence well enough to know if it would be
possible for driver code to make CMA reservations. But, it looks doubtful.
--
Mike Kravetz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
1. CMA has to preconfigured. We're suggesting mechanism that works 'out of the box'
2. Due to the pre-allocation techniques CMA imposes limitation on maximum
allocated memory. RDMA users often require 1Gb or more, sometimes more.
3. CMA reserves memory in advance, our suggestion is using existing kernel memory
mechanisms (THP for example) to allocate memory.
I would not totally rule out the use of CMA. I like the way that it reserves
memory, but does not prohibit use by others. In addition, there can be
device (or purpose) specific reservations.
I think the use case are devices that *cannot* function without
contiguous memory, typical examples IIRC are smartphone cameras on with
Android where only single app is working with the device at given time,
so it's ok to reserve single area for the device, and allocation is done
by the driver. Here we are talking about allocations done by potentially
multiple userspace applications, so how do we reconcile that with the
reservations? How does a single flag identify which device's area to
use? How do we prevent one process depleting the area for other
processes? IMHO it's another indication that a generic interface is
infeasible and it should be driver-specific.
BTW, does RDMA need a specific NUMA node to work optimally? (one closest
to the device I presume?) Will it be the job of userspace to discover
and bind itself to that node, in addition to using MAP_CONTIG? Or would
that be another thing best handled by the driver?
However, since reservations need to happen quite early it is often done on
the kernel command line. IMO, this should be avoided if possible. There
are interfaces for arch specific code to make reservations. I do not know
the system initialization sequence well enough to know if it would be
possible for driver code to make CMA reservations. But, it looks doubtful.
From: David Nellans <hidden> Date: 2017-10-23 15:27:46
On 10/16/2017 12:42 PM, Michal Hocko wrote:
On Mon 16-10-17 11:00:19, Cristopher Lameter wrote:
quoted
On Mon, 16 Oct 2017, Michal Hocko wrote:
quoted
That being said, the list is far from being complete, I am pretty sure
more would pop out if I thought more thoroughly. The bottom line is that
while I see many problems to actually implement this feature and
maintain it longterm I simply do not see a large benefit outside of a
very specific HW.
There is not much new here in terms of problems. The hardware that
needs this seems to become more and more plentiful. That is why we need a
generic implementation.
It would really help to name that HW and other potential usecases
independent on the HW because I am rather skeptical about the
_plentiful_ part. And so I really do not see any foundation to claim
the generic part. Because, fundamentally, it is the HW which requires
the specific memory placement/physically contiguous range etc. So the
generic implementation doesn't really make sense in such a context.
There are TLB's in AMD Xen that can take advantage of contig memory to
improve TLB coverage. AFAIK contig is not functionally required, its
purely a performance optimization. Current Xen TLB implementation
doesn't support arbitrary contig lengths, page sizes, etc, but its a
start. This
type of TLB optimization can be handled on the back end by de-fragging
phys mem (when possible) now that both base and THPs can be easily
migrated; no need for up-front contig, but defrag isn't free either.
From: Dave Hansen <hidden> Date: 2017-10-23 22:10:09
On 10/03/2017 04:56 PM, Mike Kravetz wrote:
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
Do you also need to lock out the NUMA migration APIs somehow? What
about KSM (or does it already ignore VM_LOCKED)?
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Maybe we should just define it as acting as if it had MADV_DONTFORK set
on it, and also that it doesn't allow MADV_DONTFORK to be called on it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Mike Kravetz <hidden> Date: 2017-10-24 22:49:48
On 10/23/2017 03:10 PM, Dave Hansen wrote:
On 10/03/2017 04:56 PM, Mike Kravetz wrote:
quoted
mmap(MAP_CONTIG) would have the following semantics:
- The entire mapping (length size) would be backed by physically contiguous
pages.
- If 'length' physically contiguous pages can not be allocated, then mmap
will fail.
- MAP_CONTIG only works with MAP_ANONYMOUS mappings.
- MAP_CONTIG will lock the associated pages in memory. As such, the same
privileges and limits that apply to mlock will also apply to MAP_CONTIG.
- A MAP_CONTIG mapping can not be expanded.
Do you also need to lock out the NUMA migration APIs somehow? What
about KSM (or does it already ignore VM_LOCKED)?
Yes, and no.
The primary use case driving this request is RDMA. As such, the pages
can not move while being used for this purpose.
When this thread was started the thought was that generic mmap would
handle the contiguous allocations. The resulting allocated pages would
be handed to the driver for additional setup based on it's specific needs.
Since then, the thought is that the driver should handle contiguous
allocations as well. I am looking at making the existing contiguous memory
allocator more usable for driver writers.
--
Mike Kravetz
quoted
- At fork time, private MAP_CONTIG mappings will be converted to regular
(non-MAP_CONTIG) mapping in the child. As such a COW fault in the child
will not require a contiguous allocation.
Maybe we should just define it as acting as if it had MADV_DONTFORK set
on it, and also that it doesn't allow MADV_DONTFORK to be called on it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>