Am 2021-08-19 um 2:00 p.m. schrieb Sierra Guiza, Alejandro (Alex):

On 8/18/2021 2:28 PM, Ralph Campbell wrote:

quoted

On 8/17/21 5:35 PM, Felix Kuehling wrote:

quoted

Am 2021-08-17 um 8:01 p.m. schrieb Ralph Campbell:

quoted

On 8/12/21 11:31 PM, Alex Sierra wrote:

quoted

From: Ralph Campbell <redacted>

ZONE_DEVICE struct pages have an extra reference count that
complicates the
code for put_page() and several places in the kernel that need to
check the
reference count to see that a page is not being used (gup,
compaction,
migration, etc.). Clean up the code so the reference count doesn't
need to
be treated specially for ZONE_DEVICE.

v2:
AS: merged this patch in linux 5.11 version

v5:
AS: add condition at try_grab_page to check for the zone device type,
while
page ref counter is checked less/equal to zero. In case of device
zone, pages
ref counter are initialized to zero.

Signed-off-by: Ralph Campbell <redacted>
Signed-off-by: Alex Sierra <redacted>
---
   arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
   drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
   fs/dax.c                               |  4 +-
   include/linux/dax.h                    |  2 +-
   include/linux/memremap.h               |  7 +--
   include/linux/mm.h                     | 13 +----
   lib/test_hmm.c                         |  2 +-
   mm/internal.h                          |  8 +++
   mm/memremap.c                          | 68
+++++++-------------------
   mm/migrate.c                           |  5 --
   mm/page_alloc.c                        |  3 ++
   mm/swap.c                              | 45 ++---------------
   12 files changed, 46 insertions(+), 115 deletions(-)

I haven't seen a response to the issues I raised back at v3 of this
series.
https://lore.kernel.org/linux-mm/4f6dd918-d79b-1aa7-3a4c-caa67ddc29bc@nvidia.com/ (local)



Did I miss something?

I think part of the response was that we did more testing. Alex added
support for DEVICE_GENERIC pages to test_hmm and he ran DAX tests
recommended by Theodore Tso. In that testing he ran into a WARN_ON_ONCE
about a zero page refcount in try_get_page. The fix is in the latest
version of patch 2. But it's already obsolete because John Hubbard is
about to remove that function altogether.

I think the issues you raised were more uncertainty than known bugs. It
seems the fact that you can have DAX pages with 0 refcount is a feature
more than a bug.

Regards,
   Felix

Did you test on a system without CONFIG_ARCH_HAS_PTE_SPECIAL defined?
In that case, mmap() of a DAX device will call insert_page() which calls
get_page() which would trigger VM_BUG_ON_PAGE().

I can believe it is OK for PTE_SPECIAL page table entries to have no
struct page or that MEMORY_DEVICE_GENERIC struct pages be mapped with
a zero reference count using insert_pfn().

Hi Ralph,
We have tried the DAX tests with and without
CONFIG_ARCH_HAS_PTE_SPECIAL defined.
Apparently none of the tests touches that condition for a DAX device.
Of course,
that doesn't mean it could happen.

Regards,
Alex S.

quoted

I find it hard to believe that other MM developers don't see an issue
with a struct page with refcount == 0 and mapcount == 1.

I don't see where init_page_count() is being called for the
MEMORY_DEVICE_GENERIC or MEMORY_DEVICE_PRIVATE struct pages the AMD
driver allocates and passes to migrate_vma_setup().
Looks like svm_migrate_get_vram_page() needs to call init_page_count()
instead of get_page(). (I'm looking at branch
origin/alexsierrag/device_generic
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver.git

Yes, you're right. My bad. Thanks for catching this up. I didn't
realize I was missing
to define CONFIG_DEBUG_VM on my build. Therefore this BUG was never
caught.
It worked after I replaced get_pages by init_page_count at
svm_migrate_get_vram_page. However, I don't think this is the best way
to fix it.
Ideally, get_pages call should work for device pages with ref count
equal to 0
too. Otherwise, we could overwrite refcounter if someone else is
grabbing the page
concurrently.

I think using init_page_count in svm_migrate_get_vram_page is the right
answer. This is where the page first gets allocated and initialized
(data migrated into it). I think nobody should have or try to take a
reference to the page before that. We should probably also add a
VM_BUG_ON_PAGE(page_ref_count(page) != 0) before calling init_page_count
to make sure of that.

I was thinking to add a special condition in get_pages for dev pages.
This could
also fix the insert_page -> get_page call from a DAX device.

[+Theodore]

I got lost trying to understand how DAX counts page references and how
the PTE_SPECIAL option affects that. Theodore, can you help with this?
Is there an easy way to test without CONFIG_ARCH_HAS_PTE_SPECIAL on x86,
or do we need to test on a CPU architecture that doesn't support this
feature?

Thanks,
  Felix

Regards,
Alex S.

quoted

Also, what about the other places where is_device_private_page() is
called?
Don't they need to be updated to call is_device_page() instead?
One of my goals for this patch was to remove special casing reference
counts
for ZONE_DEVICE pages in rmap.c, etc.

Correct, is_device_private_page is still used in rmap, memcontrol and
migrate.c files
Looks like rmap and memcontrol should be replaced by is_device_page
function. However,
I still need test to validate this. For migrate.c is used in
remove_migration_pte and
migrate_vma_insert_page, however these are specific conditions for
private device type
Thanks for raise these questions, I think we're getting close.

Regards,
Alex S.

quoted

I still think this patch needs an ACK from a FS/DAX maintainer.