On Tue, Aug 18, 2020 at 09:49:24AM +0200, David Hildenbrand wrote:
On 18.08.20 01:34, Minchan Kim wrote:
quoted
On Mon, Aug 17, 2020 at 06:44:50PM +0200, David Hildenbrand wrote:
quoted
On 17.08.20 18:30, Minchan Kim wrote:
quoted
On Mon, Aug 17, 2020 at 05:45:59PM +0200, David Hildenbrand wrote:
quoted
On 17.08.20 17:27, Minchan Kim wrote:
quoted
On Sun, Aug 16, 2020 at 02:31:22PM +0200, David Hildenbrand wrote:
quoted
On 14.08.20 19:31, Minchan Kim wrote:
quoted
There is a need for special HW to require bulk allocation of
high-order pages. For example, 4800 * order-4 pages.
To meet the requirement, a option is using CMA area because
page allocator with compaction under memory pressure is
easily failed to meet the requirement and too slow for 4800
times. However, CMA has also the following drawbacks:
* 4800 of order-4 * cma_alloc is too slow
To avoid the slowness, we could try to allocate 300M contiguous
memory once and then split them into order-4 chunks.
The problem of this approach is CMA allocation fails one of the
pages in those range couldn't migrate out, which happens easily
with fs write under memory pressure.
Why not chose a value in between? Like try to allocate MAX_ORDER - 1
chunks and split them. That would already heavily reduce the call frequency.
I think you meant this:
alloc_pages(GFP_KERNEL|__GFP_NOWARN, MAX_ORDER - 1)
It would work if system has lots of non-fragmented free memory.
However, once they are fragmented, it doesn't work. That's why we have
seen even order-4 allocation failure in the field easily and that's why
CMA was there.
CMA has more logics to isolate the memory during allocation/freeing as
well as fragmentation avoidance so that it has less chance to be stealed
from others and increase high success ratio. That's why I want this API
to be used with CMA or movable zone.
I was talking about doing MAX_ORDER - 1 CMA allocations instead of one
big 300M allocation. As you correctly note, memory placed into CMA
should be movable, except for (short/long) term pinnings. In these
cases, doing allocations smaller than 300M and splitting them up should
be good enough to reduce the call frequency, no?
I should have written that. The 300M I mentioned is really minimum size.
In some scenraio, we need way bigger than 300M, up to several GB.
Furthermore, the demand would be increased in near future.
And what will the driver do with that data besides providing it to the
device? Can it be mapped to user space? I think we really need more
information / the actual user.
quoted
quoted
quoted
A usecase is device can set a exclusive CMA area up when system boots.
When device needs 4800 * order-4 pages, it could call this bulk against
of the area so that it could effectively be guaranteed to allocate
enough fast.
Just wondering
a) Why does it have to be fast?
That's because it's related to application latency, which ends up
user feel bad.
Okay, but in theory, your device-needs are very similar to
application-needs, besides you requiring order-4 pages, correct? Similar
to an application that starts up and pins 300M (or more), just with
ordr-4 pages.
Yes.
quoted
I don't get quite yet why you need a range allocator for that. Because
you intend to use CMA?
Yes, with CMA, it could be more guaranteed and fast enough with little
tweaking. Currently, CMA is too slow due to below IPI overheads.
1. set_migratetype_isolate does drain_all_pages for every pageblock.
2. __aloc_contig_migrate_range does migrate_prep
3. alloc_contig_range does lru_add_drain_all.
Thus, if we need to increase frequency of call as your suggestion,
the set up overhead is also scaling up depending on the size.
Such overhead makes sense if caller requests big contiguous memory
but it's too much for normal high-order allocations.
Maybe, we might optimize those call sites to reduce or/remove
frequency of those IPI call smarter way but that would need to deal
with success ratio vs fastness dance in the end.
Another concern to use existing cma API is it's trying to make
allocation successful at the cost of latency. For example, waiting
a page writeback.
That's the this new sematic API comes from for compromise since I believe
we need some way to separate original CMA alloc(biased to be guaranteed
but slower) with this new API(biased to be fast but less guaranteed).
Is there any idea to work without existing cma API tweaking?
Let me try to summarize:
1. Your driver needs a lot of order-4 pages. And it's needs them fast,
because of observerable lag/delay in an application. The pages will be
unmovable by the driver.
2. Your idea is to use CMA, as that avoids unmovable allocations,
theoretically allowing you to allocate all memory. But you don't
actually want a large contiguous memory area.
3. Doing a whole bunch of order-4 cma allocations is slow.
4. Doing a single large cma allocation and splitting it manually in the
caller can fail easily due to temporary page pinnings.
Regarding 4., [1] comes to mind, which has the same issues with
temporary page pinnings and solves it by simply retrying. Yeah, there
will be some lag, but maybe it's overall faster than doing separate
order-4 cma allocations?
Thanks for the pointer. However, it's not a single reason to make CMA
failure. Historically, there are various potentail problems to make
"temporal" as "non-temporal" like page write, indirect dependency
between objects.
In general, proactive compaction [2] comes to mind, does that help?
I think it makes sense if such high-order allocation are dominant in the
system workload because the benefit caused by TLB would be bigger than cost
caused by frequent migration overhead. However, it's not the our usecase.
[1]
https://lore.kernel.org/r/1596682582-29139-2-git-send-email-cgoldswo@codeaurora.org/ (local)
[2] https://nitingupta.dev/post/proactive-compaction/
I understand pfn stuff in the API is not pretty but the concept of idea
makes sense to me in that go though the *migratable area* and get possible
order pages with hard effort. It looks like GFP_NORETRY version for
kmem_cache_alloc_bulk.
How about this?
int cma_alloc(struct cma *cma, int order, unsigned int nr_elem, struct page **pages);