Re: [PATCH v1 03/14] mm: add iomem vma selection for memory migration

On Wed, Sep 01, 2021 at 07:07:34PM -0400, Felix Kuehling wrote:

On 2021-09-01 6:03 p.m., Dave Chinner wrote:

quoted

On Wed, Sep 01, 2021 at 11:40:43AM -0400, Felix Kuehling wrote:

quoted

Am 2021-09-01 um 4:29 a.m. schrieb Christoph Hellwig:

quoted

On Mon, Aug 30, 2021 at 01:04:43PM -0400, Felix Kuehling wrote:

quoted

quoted

quoted

driver code is not really involved in updating the CPU mappings. Maybe
it's something we need to do in the migration helpers.

It looks like I'm totally misunderstanding what you are adding here
then.  Why do we need any special treatment at all for memory that
has normal struct pages and is part of the direct kernel map?

The pages are like normal memory for purposes of mapping them in CPU
page tables and for coherent access from the CPU.

That's the user page tables.  What about the kernel direct map?
If there is a normal kernel struct page backing there really should
be no need for the pgmap.

I'm not sure. The physical address ranges are in the UEFI system address
map as special-purpose memory. Does Linux create the struct pages and
kernel direct map for that without a pgmap call? I didn't see that last
time I went digging through that code.

quoted

quoted

 From an application
perspective, we want file-backed and anonymous mappings to be able to
use DEVICE_PUBLIC pages with coherent CPU access. The goal is to
optimize performance for GPU heavy workloads while minimizing the need
to migrate data back-and-forth between system memory and device memory.

I don't really understand that part.  file backed pages are always
allocated by the file system using the pagecache helpers, that is
using the page allocator.  Anonymouns memory also always comes from
the page allocator.

I'm coming at this from my experience with DEVICE_PRIVATE. Both
anonymous and file-backed pages should be migrateable to DEVICE_PRIVATE
memory by the migrate_vma_* helpers for more efficient access by our
GPU. (*) It's part of the basic premise of HMM as I understand it. I
would expect the same thing to work for DEVICE_PUBLIC memory.

(*) I believe migrating file-backed pages to DEVICE_PRIVATE doesn't
currently work, but that's something I'm hoping to fix at some point.

FWIW, I'd love to see the architecture documents that define how
filesystems are supposed to interact with this device private
memory. This whole "hand filesystem controlled memory to other
devices" is a minefield that is trivial to get wrong iand very
difficult to fix - just look at the historical mess that RDMA
to/from file backed and/or DAX pages has been.

So, really, from my perspective as a filesystem engineer, I want to
see an actual specification for how this new memory type is going to
interact with filesystem and the page cache so everyone has some
idea of how this is going to work and can point out how it doesn't
work before code that simply doesn't work is pushed out into
production systems and then merged....

OK. To be clear, that's not part of this patch series. And I have no
authority to push anything in this part of the kernel, so you have nothing
to fear. ;)

I know this isn't part of the series. but this patchset is laying
the foundation for future work that will impact subsystems that
currently have zero visibility and/or knowledge of these changes.
There must be an overall architectural plan for this functionality,
regardless of the current state of implementation. It's that overall
architectural plan I'm asking about here, because I need to
understand that before I can sanely comment on the page
cache/filesystem aspect of the proposed functionality...

FWIW, we already have the ability to map file-backed system memory pages
into device page tables with HMM and interval notifiers. But we cannot
currently migrate them to ZONE_DEVICE pages.

Sure, but sharing page cache pages allocated and managed by the
filesystem is not what you are talking about. You're talking about
migrating page cache data to completely different memory allocated
by a different memory manager that the filesystems currently have no
knowledge of or have any way of interfacing with.

So I'm asking basic, fundamental questions about how these special
device based pages are going to work.  How are these pages different
to normal pages - does page_lock() still guarantee exclusive access
to the page state across all hardware that can access it? If not,
what provides access serialisation for pages that are allocated in
device memory rather than CPU memory (e.g. for truncate
serialisation)?  Does the hardware that own these pages raise page
faults on the CPU when those pages are accessed/dirtied? How does
demand paging in and out of device memory work (i.e. mapping files
larger than device memory).  How does IO to/from storage work - can
the filesystem build normal bios out of these device pages and issue
IO on them?  Are the additional constraints on IO because p2p DMA is
needed to move the data from the storage HBA directly into/out of
the GPU memory?

I can think of lots more complex questions about how filesystems are
supposed to manage remote device memory in the page cache, but these
are just some of the basic things that make file-backed mappings
different to anonymous mappings that I need to understand before I
can make head or tail of what is being proposed here.....

Beyond that, my understanding
of how filesystems and page cache work is rather superficial at this point.
I'll keep your name in mind for when I am ready to discuss this in more
detail.

If you don't know what the bigger picture is, then who does?
Somebody built the design/architecture you are working towards, and
they had to communicate it to you somehow. I'm asking for that
information to documented and made available to all the people these
changes might impact, not whether you personally know how it
works....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com