incoming

The remainder of the main mm/ queue.

143 patches, based on 8ca5297e7e38f2dc8c753d33a5092e7be181fff0, plus
previously sent patches.

Subsystems affected by this patch series:

  mm/pagecache
  mm/hugetlb
  mm/userfaultfd
  mm/vmscan
  mm/compaction
  mm/migration
  mm/cma
  mm/ksm
  mm/vmstat
  mm/mmap
  mm/kconfig
  mm/util
  mm/memory-hotplug
  mm/zswap
  mm/zsmalloc
  mm/highmem
  mm/cleanups
  mm/kfence

Subsystem: mm/pagecache

    "Matthew Wilcox (Oracle)" [off-list ref]:
    Patch series "Remove nrexceptional tracking", v2:
      mm: introduce and use mapping_empty()
      mm: stop accounting shadow entries
      dax: account DAX entries as nrpages
      mm: remove nrexceptional from inode

    Hugh Dickins [off-list ref]:
      mm: remove nrexceptional from inode: remove BUG_ON

Subsystem: mm/hugetlb

    Peter Xu [off-list ref]:
    Patch series "hugetlb: Disable huge pmd unshare for uffd-wp", v4:
      hugetlb: pass vma into huge_pte_alloc() and huge_pmd_share()
      hugetlb/userfaultfd: forbid huge pmd sharing when uffd enabled
      mm/hugetlb: move flush_hugetlb_tlb_range() into hugetlb.h
      hugetlb/userfaultfd: unshare all pmds for hugetlbfs when register wp

    Miaohe Lin [off-list ref]:
      mm/hugetlb: remove redundant reservation check condition in alloc_huge_page()

    Anshuman Khandual [off-list ref]:
      mm: generalize HUGETLB_PAGE_SIZE_VARIABLE

    Miaohe Lin [off-list ref]:
    Patch series "Some cleanups for hugetlb":
      mm/hugetlb: use some helper functions to cleanup code
      mm/hugetlb: optimize the surplus state transfer code in move_hugetlb_state()
      mm/hugetlb_cgroup: remove unnecessary VM_BUG_ON_PAGE in hugetlb_cgroup_migrate()
      mm/hugetlb: simplify the code when alloc_huge_page() failed in hugetlb_no_page()
      mm/hugetlb: avoid calculating fault_mutex_hash in truncate_op case
    Patch series "Cleanup and fixup for khugepaged", v2:
      khugepaged: remove unneeded return value of khugepaged_collapse_pte_mapped_thps()
      khugepaged: reuse the smp_wmb() inside __SetPageUptodate()
      khugepaged: use helper khugepaged_test_exit() in __khugepaged_enter()
      khugepaged: fix wrong result value for trace_mm_collapse_huge_page_isolate()
      mm/huge_memory.c: remove unnecessary local variable ret2
    Patch series "Some cleanups for huge_memory", v3:
      mm/huge_memory.c: rework the function vma_adjust_trans_huge()
      mm/huge_memory.c: make get_huge_zero_page() return bool
      mm/huge_memory.c: rework the function do_huge_pmd_numa_page() slightly
      mm/huge_memory.c: remove redundant PageCompound() check
      mm/huge_memory.c: remove unused macro TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG
      mm/huge_memory.c: use helper function migration_entry_to_page()

    Yanfei Xu [off-list ref]:
      mm/khugepaged.c: replace barrier() with READ_ONCE() for a selective variable

    Miaohe Lin [off-list ref]:
    Patch series "Cleanup for khugepaged":
      khugepaged: use helper function range_in_vma() in collapse_pte_mapped_thp()
      khugepaged: remove unnecessary out label in collapse_huge_page()
      khugepaged: remove meaningless !pte_present() check in khugepaged_scan_pmd()

    Zi Yan [off-list ref]:
      mm: huge_memory: a new debugfs interface for splitting THP tests
      mm: huge_memory: debugfs for file-backed THP split

    Miaohe Lin [off-list ref]:
    Patch series "Cleanup and fixup for hugetlb", v2:
      mm/hugeltb: remove redundant VM_BUG_ON() in region_add()
      mm/hugeltb: simplify the return code of __vma_reservation_common()
      mm/hugeltb: clarify (chg - freed) won't go negative in hugetlb_unreserve_pages()
      mm/hugeltb: handle the error case in hugetlb_fix_reserve_counts()
      mm/hugetlb: remove unused variable pseudo_vma in remove_inode_hugepages()

    Mike Kravetz [off-list ref]:
    Patch series "make hugetlb put_page safe for all calling contexts", v5:
      mm/cma: change cma mutex to irq safe spinlock
      hugetlb: no need to drop hugetlb_lock to call cma_release
      hugetlb: add per-hstate mutex to synchronize user adjustments
      hugetlb: create remove_hugetlb_page() to separate functionality
      hugetlb: call update_and_free_page without hugetlb_lock
      hugetlb: change free_pool_huge_page to remove_pool_huge_page
      hugetlb: make free_huge_page irq safe
      hugetlb: add lockdep_assert_held() calls for hugetlb_lock

    Oscar Salvador [off-list ref]:
    Patch series "Make alloc_contig_range handle Hugetlb pages", v10:
      mm,page_alloc: bail out earlier on -ENOMEM in alloc_contig_migrate_range
      mm,compaction: let isolate_migratepages_{range,block} return error codes
      mm,hugetlb: drop clearing of flag from prep_new_huge_page
      mm,hugetlb: split prep_new_huge_page functionality
      mm: make alloc_contig_range handle free hugetlb pages
      mm: make alloc_contig_range handle in-use hugetlb pages
      mm,page_alloc: drop unnecessary checks from pfn_range_valid_contig

Subsystem: mm/userfaultfd

    Axel Rasmussen [off-list ref]:
    Patch series "userfaultfd: add minor fault handling", v9:
      userfaultfd: add minor fault registration mode
      userfaultfd: disable huge PMD sharing for MINOR registered VMAs
      userfaultfd: hugetlbfs: only compile UFFD helpers if config enabled
      userfaultfd: add UFFDIO_CONTINUE ioctl
      userfaultfd: update documentation to describe minor fault handling
      userfaultfd/selftests: add test exercising minor fault handling

Subsystem: mm/vmscan

    Dave Hansen [off-list ref]:
      mm/vmscan: move RECLAIM* bits to uapi header
      mm/vmscan: replace implicit RECLAIM_ZONE checks with explicit checks

    Yang Shi [off-list ref]:
    Patch series "Make shrinker's nr_deferred memcg aware", v10:
      mm: vmscan: use nid from shrink_control for tracepoint
      mm: vmscan: consolidate shrinker_maps handling code
      mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation
      mm: vmscan: remove memcg_shrinker_map_size
      mm: vmscan: use kvfree_rcu instead of call_rcu
      mm: memcontrol: rename shrinker_map to shrinker_info
      mm: vmscan: add shrinker_info_protected() helper
      mm: vmscan: use a new flag to indicate shrinker is registered
      mm: vmscan: add per memcg shrinker nr_deferred
      mm: vmscan: use per memcg nr_deferred of shrinker
      mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers
      mm: memcontrol: reparent nr_deferred when memcg offline
      mm: vmscan: shrink deferred objects proportional to priority

Subsystem: mm/compaction

    Pintu Kumar [off-list ref]:
      mm/compaction: remove unused variable sysctl_compact_memory

    Charan Teja Reddy [off-list ref]:
      mm: compaction: update the COMPACT[STALL|FAIL] events properly

Subsystem: mm/migration

    Minchan Kim [off-list ref]:
      mm: disable LRU pagevec during the migration temporarily
      mm: replace migrate_[prep|finish] with lru_cache_[disable|enable]
      mm: fs: invalidate BH LRU during page migration

    Miaohe Lin [off-list ref]:
    Patch series "Cleanup and fixup for mm/migrate.c", v3:
      mm/migrate.c: make putback_movable_page() static
      mm/migrate.c: remove unnecessary rc != MIGRATEPAGE_SUCCESS check in 'else' case
      mm/migrate.c: fix potential indeterminate pte entry in migrate_vma_insert_page()
      mm/migrate.c: use helper migrate_vma_collect_skip() in migrate_vma_collect_hole()
      Revert "mm: migrate: skip shared exec THP for NUMA balancing"

Subsystem: mm/cma

    Minchan Kim [off-list ref]:
      mm: vmstat: add cma statistics

    Baolin Wang [off-list ref]:
      mm: cma: use pr_err_ratelimited for CMA warning

    Liam Mark [off-list ref]:
      mm: cma: add trace events for CMA alloc perf testing

    Minchan Kim [off-list ref]:
      mm: cma: support sysfs
      mm: cma: add the CMA instance name to cma trace events
      mm: use proper type for cma_[alloc|release]

Subsystem: mm/ksm

    Miaohe Lin [off-list ref]:
    Patch series "Cleanup and fixup for ksm":
      ksm: remove redundant VM_BUG_ON_PAGE() on stable_tree_search()
      ksm: use GET_KSM_PAGE_NOLOCK to get ksm page in remove_rmap_item_from_tree()
      ksm: remove dedicated macro KSM_FLAG_MASK
      ksm: fix potential missing rmap_item for stable_node

    Chengyang Fan [off-list ref]:
      mm/ksm: remove unused parameter from remove_trailing_rmap_items()

Subsystem: mm/vmstat

    Hugh Dickins [off-list ref]:
      mm: restore node stat checking in /proc/sys/vm/stat_refresh
      mm: no more EINVAL from /proc/sys/vm/stat_refresh
      mm: /proc/sys/vm/stat_refresh skip checking known negative stats
      mm: /proc/sys/vm/stat_refresh stop checking monotonic numa stats

    Saravanan D [off-list ref]:
      x86/mm: track linear mapping split events

Subsystem: mm/mmap

    Liam Howlett [off-list ref]:
      mm/mmap.c: don't unlock VMAs in remap_file_pages()

Subsystem: mm/kconfig

    Anshuman Khandual [off-list ref]:
    Patch series "mm: some config cleanups", v2:
      mm: generalize ARCH_HAS_CACHE_LINE_SIZE
      mm: generalize SYS_SUPPORTS_HUGETLBFS (rename as ARCH_SUPPORTS_HUGETLBFS)
      mm: generalize ARCH_ENABLE_MEMORY_[HOTPLUG|HOTREMOVE]
      mm: drop redundant ARCH_ENABLE_[HUGEPAGE|THP]_MIGRATION
      mm: drop redundant ARCH_ENABLE_SPLIT_PMD_PTLOCK
      mm: drop redundant HAVE_ARCH_TRANSPARENT_HUGEPAGE

Subsystem: mm/util

    Joe Perches [off-list ref]:
      mm/util.c: reduce mem_dump_obj() object size

    Bhaskar Chowdhury [off-list ref]:
      mm/util.c: fix typo

Subsystem: mm/memory-hotplug

    Pavel Tatashin [off-list ref]:
    Patch series "prohibit pinning pages in ZONE_MOVABLE", v11:
      mm/gup: don't pin migrated cma pages in movable zone
      mm/gup: check every subpage of a compound page during isolation
      mm/gup: return an error on migration failure
      mm/gup: check for isolation errors
      mm cma: rename PF_MEMALLOC_NOCMA to PF_MEMALLOC_PIN
      mm: apply per-task gfp constraints in fast path
      mm: honor PF_MEMALLOC_PIN for all movable pages
      mm/gup: do not migrate zero page
      mm/gup: migrate pinned pages out of movable zone
      memory-hotplug.rst: add a note about ZONE_MOVABLE and page pinning
      mm/gup: change index type to long as it counts pages
      mm/gup: longterm pin migration cleanup
      selftests/vm: gup_test: fix test flag
      selftests/vm: gup_test: test faulting in kernel, and verify pinnable pages

    Mel Gorman [off-list ref]:
      mm/memory_hotplug: remove broken locking of zone PCP structures during hot remove

    Oscar Salvador [off-list ref]:
    Patch series "Allocate memmap from hotadded memory (per device)", v10:
      drivers/base/memory: introduce memory_block_{online,offline}
      mm,memory_hotplug: relax fully spanned sections check

    David Hildenbrand [off-list ref]:
      mm,memory_hotplug: factor out adjusting present pages into adjust_present_page_count()

    Oscar Salvador [off-list ref]:
      mm,memory_hotplug: allocate memmap from the added memory range
      acpi,memhotplug: enable MHP_MEMMAP_ON_MEMORY when supported
      mm,memory_hotplug: add kernel boot option to enable memmap_on_memory
      x86/Kconfig: introduce ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE
      arm64/Kconfig: introduce ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE

Subsystem: mm/zswap

    Zhiyuan Dai [off-list ref]:
      mm/zswap.c: switch from strlcpy to strscpy

Subsystem: mm/zsmalloc

    zhouchuangao [off-list ref]:
      mm/zsmalloc: use BUG_ON instead of if condition followed by BUG.

Subsystem: mm/highmem

    Ira Weiny [off-list ref]:
    Patch series "btrfs: Convert kmap/memset/kunmap to memzero_user()":
      iov_iter: lift memzero_page() to highmem.h
      btrfs: use memzero_page() instead of open coded kmap pattern

    songqiang [off-list ref]:
      mm/highmem.c: fix coding style issue

Subsystem: mm/cleanups

    Zhiyuan Dai [off-list ref]:
      mm/mempool: minor coding style tweaks

    Zhang Yunkai [off-list ref]:
      mm/process_vm_access.c: remove duplicate include

Subsystem: mm/kfence

    Marco Elver [off-list ref]:
      kfence: zero guard page after out-of-bounds access
    Patch series "kfence: optimize timer scheduling", v2:
      kfence: await for allocation using wait_event
      kfence: maximize allocation wait timeout duration
      kfence: use power-efficient work queue to run delayed work

 Documentation/ABI/testing/sysfs-kernel-mm-cma     |   25 
 Documentation/admin-guide/kernel-parameters.txt   |   17 
 Documentation/admin-guide/mm/memory-hotplug.rst   |    9 
 Documentation/admin-guide/mm/userfaultfd.rst      |  105 +-
 arch/arc/Kconfig                                  |    9 
 arch/arm/Kconfig                                  |   10 
 arch/arm64/Kconfig                                |   34 
 arch/arm64/mm/hugetlbpage.c                       |    7 
 arch/ia64/Kconfig                                 |   14 
 arch/ia64/mm/hugetlbpage.c                        |    3 
 arch/mips/Kconfig                                 |    6 
 arch/mips/mm/hugetlbpage.c                        |    4 
 arch/parisc/Kconfig                               |    5 
 arch/parisc/mm/hugetlbpage.c                      |    2 
 arch/powerpc/Kconfig                              |   17 
 arch/powerpc/mm/hugetlbpage.c                     |    3 
 arch/powerpc/platforms/Kconfig.cputype            |   16 
 arch/riscv/Kconfig                                |    5 
 arch/s390/Kconfig                                 |   12 
 arch/s390/mm/hugetlbpage.c                        |    2 
 arch/sh/Kconfig                                   |    7 
 arch/sh/mm/Kconfig                                |    8 
 arch/sh/mm/hugetlbpage.c                          |    2 
 arch/sparc/mm/hugetlbpage.c                       |    2 
 arch/x86/Kconfig                                  |   33 
 arch/x86/mm/pat/set_memory.c                      |    8 
 drivers/acpi/acpi_memhotplug.c                    |    5 
 drivers/base/memory.c                             |  105 ++
 fs/Kconfig                                        |    5 
 fs/block_dev.c                                    |    2 
 fs/btrfs/compression.c                            |    5 
 fs/btrfs/extent_io.c                              |   22 
 fs/btrfs/inode.c                                  |   33 
 fs/btrfs/reflink.c                                |    6 
 fs/btrfs/zlib.c                                   |    5 
 fs/btrfs/zstd.c                                   |    5 
 fs/buffer.c                                       |   36 
 fs/dax.c                                          |    8 
 fs/gfs2/glock.c                                   |    3 
 fs/hugetlbfs/inode.c                              |    9 
 fs/inode.c                                        |   11 
 fs/proc/task_mmu.c                                |    3 
 fs/userfaultfd.c                                  |  149 +++
 include/linux/buffer_head.h                       |    4 
 include/linux/cma.h                               |    4 
 include/linux/compaction.h                        |    1 
 include/linux/fs.h                                |    2 
 include/linux/gfp.h                               |    2 
 include/linux/highmem.h                           |    7 
 include/linux/huge_mm.h                           |    3 
 include/linux/hugetlb.h                           |   37 
 include/linux/memcontrol.h                        |   27 
 include/linux/memory.h                            |    8 
 include/linux/memory_hotplug.h                    |   15 
 include/linux/memremap.h                          |    2 
 include/linux/migrate.h                           |   11 
 include/linux/mm.h                                |   28 
 include/linux/mmzone.h                            |   20 
 include/linux/pagemap.h                           |    5 
 include/linux/pgtable.h                           |   12 
 include/linux/sched.h                             |    2 
 include/linux/sched/mm.h                          |   27 
 include/linux/shrinker.h                          |    7 
 include/linux/swap.h                              |   21 
 include/linux/userfaultfd_k.h                     |   55 +
 include/linux/vm_event_item.h                     |    8 
 include/trace/events/cma.h                        |   92 +-
 include/trace/events/migrate.h                    |   25 
 include/trace/events/mmflags.h                    |    7 
 include/uapi/linux/mempolicy.h                    |    7 
 include/uapi/linux/userfaultfd.h                  |   36 
 init/Kconfig                                      |    5 
 kernel/sysctl.c                                   |    2 
 lib/Kconfig.kfence                                |    1 
 lib/iov_iter.c                                    |    8 
 mm/Kconfig                                        |   28 
 mm/Makefile                                       |    6 
 mm/cma.c                                          |   70 +
 mm/cma.h                                          |   25 
 mm/cma_debug.c                                    |    8 
 mm/cma_sysfs.c                                    |  112 ++
 mm/compaction.c                                   |  113 ++
 mm/filemap.c                                      |   24 
 mm/frontswap.c                                    |   12 
 mm/gup.c                                          |  264 +++---
 mm/gup_test.c                                     |   29 
 mm/gup_test.h                                     |    3 
 mm/highmem.c                                      |   11 
 mm/huge_memory.c                                  |  326 +++++++-
 mm/hugetlb.c                                      |  843 ++++++++++++++--------
 mm/hugetlb_cgroup.c                               |    9 
 mm/internal.h                                     |   10 
 mm/kfence/core.c                                  |   61 +
 mm/khugepaged.c                                   |   63 -
 mm/ksm.c                                          |   17 
 mm/list_lru.c                                     |    6 
 mm/memcontrol.c                                   |  137 ---
 mm/memory_hotplug.c                               |  220 +++++
 mm/mempolicy.c                                    |   16 
 mm/mempool.c                                      |    2 
 mm/migrate.c                                      |  103 --
 mm/mlock.c                                        |    4 
 mm/mmap.c                                         |   18 
 mm/oom_kill.c                                     |    2 
 mm/page_alloc.c                                   |   83 +-
 mm/process_vm_access.c                            |    1 
 mm/shmem.c                                        |    2 
 mm/sparse.c                                       |    4 
 mm/swap.c                                         |   69 +
 mm/swap_state.c                                   |    4 
 mm/swapfile.c                                     |    4 
 mm/truncate.c                                     |   19 
 mm/userfaultfd.c                                  |   39 -
 mm/util.c                                         |   26 
 mm/vmalloc.c                                      |    2 
 mm/vmscan.c                                       |  543 +++++++++-----
 mm/vmstat.c                                       |   45 -
 mm/workingset.c                                   |    1 
 mm/zsmalloc.c                                     |    6 
 mm/zswap.c                                        |    2 
 tools/testing/selftests/vm/.gitignore             |    1 
 tools/testing/selftests/vm/Makefile               |    1 
 tools/testing/selftests/vm/gup_test.c             |   38 
 tools/testing/selftests/vm/split_huge_page_test.c |  400 ++++++++++
 tools/testing/selftests/vm/userfaultfd.c          |  164 ++++
 125 files changed, 3596 insertions(+), 1668 deletions(-)

From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: mm: introduce and use mapping_empty()

Patch series "Remove nrexceptional tracking", v2.

We actually use nrexceptional for very little these days.  It's a minor
pain to keep in sync with nrpages, but the pain becomes much bigger with
the THP patches because we don't know how many indices a shadow entry
occupies.  It's easier to just remove it than keep it accurate.

Also, we save 8 bytes per inode which is nothing to sneeze at; on my
laptop, it would improve shmem_inode_cache from 22 to 23 objects per
16kB, and inode_cache from 26 to 27 objects.  Combined, that saves
a megabyte of memory from a combined usage of 25MB for both caches.
Unfortunately, ext4 doesn't cross a magic boundary, so it doesn't save
any memory for ext4.


This patch (of 4):

Instead of checking the two counters (nrpages and nrexceptional), we can
just check whether i_pages is empty.

Link: https://lkml.kernel.org/r/20201026151849.24232-1-willy@infradead.org
Link: https://lkml.kernel.org/r/20201026151849.24232-2-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Tested-by: Vishal Verma <vishal.l.verma@intel.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/block_dev.c          |    2 +-
 fs/dax.c                |    2 +-
 fs/gfs2/glock.c         |    3 +--
 include/linux/pagemap.h |    5 +++++
 mm/truncate.c           |   18 +++---------------
 5 files changed, 11 insertions(+), 19 deletions(-)

--- a/fs/block_dev.c~mm-introduce-and-use-mapping_empty
+++ a/fs/block_dev.c

@@ -79,7 +79,7 @@ static void kill_bdev(struct block_devic
 {
 	struct address_space *mapping = bdev->bd_inode->i_mapping;
 
-	if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
+	if (mapping_empty(mapping))
 		return;
 
 	invalidate_bh_lrus();
--- a/fs/dax.c~mm-introduce-and-use-mapping_empty
+++ a/fs/dax.c

@@ -965,7 +965,7 @@ int dax_writeback_mapping_range(struct a
 	if (WARN_ON_ONCE(inode->i_blkbits != PAGE_SHIFT))
 		return -EIO;
 
-	if (!mapping->nrexceptional || wbc->sync_mode != WB_SYNC_ALL)
+	if (mapping_empty(mapping) || wbc->sync_mode != WB_SYNC_ALL)
 		return 0;
 
 	trace_dax_writeback_range(inode, xas.xa_index, end_index);
--- a/fs/gfs2/glock.c~mm-introduce-and-use-mapping_empty
+++ a/fs/gfs2/glock.c

@@ -273,8 +273,7 @@ static void __gfs2_glock_put(struct gfs2
 	if (mapping) {
 		truncate_inode_pages_final(mapping);
 		if (!gfs2_withdrawn(sdp))
-			GLOCK_BUG_ON(gl, mapping->nrpages ||
-				     mapping->nrexceptional);
+			GLOCK_BUG_ON(gl, !mapping_empty(mapping));
 	}
 	trace_gfs2_glock_put(gl);
 	sdp->sd_lockstruct.ls_ops->lm_put_lock(gl);
--- a/include/linux/pagemap.h~mm-introduce-and-use-mapping_empty
+++ a/include/linux/pagemap.h

@@ -18,6 +18,11 @@
 
 struct pagevec;
 
+static inline bool mapping_empty(struct address_space *mapping)
+{
+	return xa_empty(&mapping->i_pages);
+}
+
 /*
  * Bits in mapping->flags.
  */
--- a/mm/truncate.c~mm-introduce-and-use-mapping_empty
+++ a/mm/truncate.c

@@ -295,7 +295,7 @@ void truncate_inode_pages_range(struct a
 	pgoff_t		index;
 	int		i;
 
-	if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
+	if (mapping_empty(mapping))
 		goto out;
 
 	/* Offsets within partial pages */

@@ -440,9 +440,6 @@ EXPORT_SYMBOL(truncate_inode_pages);
  */
 void truncate_inode_pages_final(struct address_space *mapping)
 {
-	unsigned long nrexceptional;
-	unsigned long nrpages;
-
 	/*
 	 * Page reclaim can not participate in regular inode lifetime
 	 * management (can't call iput()) and thus can race with the

@@ -452,16 +449,7 @@ void truncate_inode_pages_final(struct a
 	 */
 	mapping_set_exiting(mapping);
 
-	/*
-	 * When reclaim installs eviction entries, it increases
-	 * nrexceptional first, then decreases nrpages.  Make sure we see
-	 * this in the right order or we might miss an entry.
-	 */
-	nrpages = mapping->nrpages;
-	smp_rmb();
-	nrexceptional = mapping->nrexceptional;
-
-	if (nrpages || nrexceptional) {
+	if (!mapping_empty(mapping)) {
 		/*
 		 * As truncation uses a lockless tree lookup, cycle
 		 * the tree lock to make sure any ongoing tree

@@ -633,7 +621,7 @@ int invalidate_inode_pages2_range(struct
 	int ret2 = 0;
 	int did_range_unmap = 0;
 
-	if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
+	if (mapping_empty(mapping))
 		goto out;
 
 	pagevec_init(&pvec);

_

From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: mm: stop accounting shadow entries

We no longer need to keep track of how many shadow entries are present in
a mapping.  This saves a few writes to the inode and memory barriers.

Link: https://lkml.kernel.org/r/20201026151849.24232-3-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Tested-by: Vishal Verma <vishal.l.verma@intel.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/filemap.c    |   13 -------------
 mm/swap_state.c |    4 ----
 mm/truncate.c   |    1 -
 mm/workingset.c |    1 -
 4 files changed, 19 deletions(-)

--- a/mm/filemap.c~mm-stop-accounting-shadow-entries
+++ a/mm/filemap.c

@@ -142,17 +142,6 @@ static void page_cache_delete(struct add
 
 	page->mapping = NULL;
 	/* Leave page->index set: truncation lookup relies upon it */
-
-	if (shadow) {
-		mapping->nrexceptional += nr;
-		/*
-		 * Make sure the nrexceptional update is committed before
-		 * the nrpages update so that final truncate racing
-		 * with reclaim does not see both counters 0 at the
-		 * same time and miss a shadow entry.
-		 */
-		smp_wmb();
-	}
 	mapping->nrpages -= nr;
 }
 

@@ -925,8 +914,6 @@ noinline int __add_to_page_cache_locked(
 		if (xas_error(&xas))
 			goto unlock;
 
-		if (old)
-			mapping->nrexceptional--;
 		mapping->nrpages++;
 
 		/* hugetlb pages do not participate in page cache accounting */
--- a/mm/swap_state.c~mm-stop-accounting-shadow-entries
+++ a/mm/swap_state.c

@@ -132,7 +132,6 @@ int add_to_swap_cache(struct page *page,
 			xas_store(&xas, page);
 			xas_next(&xas);
 		}
-		address_space->nrexceptional -= nr_shadows;
 		address_space->nrpages += nr;
 		__mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr);
 		__mod_lruvec_page_state(page, NR_SWAPCACHE, nr);

@@ -172,8 +171,6 @@ void __delete_from_swap_cache(struct pag
 		xas_next(&xas);
 	}
 	ClearPageSwapCache(page);
-	if (shadow)
-		address_space->nrexceptional += nr;
 	address_space->nrpages -= nr;
 	__mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, -nr);
 	__mod_lruvec_page_state(page, NR_SWAPCACHE, -nr);

@@ -275,7 +272,6 @@ void clear_shadow_from_swap_cache(int ty
 			xas_store(&xas, NULL);
 			nr_shadows++;
 		}
-		address_space->nrexceptional -= nr_shadows;
 		xa_unlock_irq(&address_space->i_pages);
 
 		/* search the next swapcache until we meet end */
--- a/mm/truncate.c~mm-stop-accounting-shadow-entries
+++ a/mm/truncate.c

@@ -40,7 +40,6 @@ static inline void __clear_shadow_entry(
 	if (xas_load(&xas) != entry)
 		return;
 	xas_store(&xas, NULL);
-	mapping->nrexceptional--;
 }
 
 static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
--- a/mm/workingset.c~mm-stop-accounting-shadow-entries
+++ a/mm/workingset.c

@@ -554,7 +554,6 @@ static enum lru_status shadow_lru_isolat
 		goto out_invalid;
 	if (WARN_ON_ONCE(node->count != node->nr_values))
 		goto out_invalid;
-	mapping->nrexceptional -= node->nr_values;
 	xa_delete_node(node, workingset_update_node);
 	__inc_lruvec_kmem_state(node, WORKINGSET_NODERECLAIM);
 

_

From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: dax: account DAX entries as nrpages

Simplify mapping_needs_writeback() by accounting DAX entries as pages
instead of exceptional entries.

Link: https://lkml.kernel.org/r/20201026151849.24232-4-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Tested-by: Vishal Verma <vishal.l.verma@intel.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/dax.c     |    6 +++---
 mm/filemap.c |    3 ---
 2 files changed, 3 insertions(+), 6 deletions(-)

--- a/fs/dax.c~dax-account-dax-entries-as-nrpages
+++ a/fs/dax.c

@@ -525,7 +525,7 @@ retry:
 		dax_disassociate_entry(entry, mapping, false);
 		xas_store(xas, NULL);	/* undo the PMD join */
 		dax_wake_entry(xas, entry, true);
-		mapping->nrexceptional--;
+		mapping->nrpages -= PG_PMD_NR;
 		entry = NULL;
 		xas_set(xas, index);
 	}

@@ -541,7 +541,7 @@ retry:
 		dax_lock_entry(xas, entry);
 		if (xas_error(xas))
 			goto out_unlock;
-		mapping->nrexceptional++;
+		mapping->nrpages += 1UL << order;
 	}
 
 out_unlock:

@@ -661,7 +661,7 @@ static int __dax_invalidate_entry(struct
 		goto out;
 	dax_disassociate_entry(entry, mapping, trunc);
 	xas_store(&xas, NULL);
-	mapping->nrexceptional--;
+	mapping->nrpages -= 1UL << dax_entry_order(entry);
 	ret = 1;
 out:
 	put_unlocked_entry(&xas, entry);
--- a/mm/filemap.c~dax-account-dax-entries-as-nrpages
+++ a/mm/filemap.c

@@ -618,9 +618,6 @@ EXPORT_SYMBOL(filemap_fdatawait_keep_err
 /* Returns true if writeback might be needed or already in progress. */
 static bool mapping_needs_writeback(struct address_space *mapping)
 {
-	if (dax_mapping(mapping))
-		return mapping->nrexceptional;
-
 	return mapping->nrpages;
 }
 

_

From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: mm: remove nrexceptional from inode

We no longer track anything in nrexceptional, so remove it, saving 8 bytes
per inode.

Link: https://lkml.kernel.org/r/20201026151849.24232-5-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Tested-by: Vishal Verma <vishal.l.verma@intel.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/inode.c         |    2 +-
 include/linux/fs.h |    2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

--- a/fs/inode.c~mm-remove-nrexceptional-from-inode
+++ a/fs/inode.c

@@ -529,7 +529,7 @@ void clear_inode(struct inode *inode)
 	 */
 	xa_lock_irq(&inode->i_data.i_pages);
 	BUG_ON(inode->i_data.nrpages);
-	BUG_ON(inode->i_data.nrexceptional);
+	BUG_ON(!mapping_empty(&inode->i_data));
 	xa_unlock_irq(&inode->i_data.i_pages);
 	BUG_ON(!list_empty(&inode->i_data.private_list));
 	BUG_ON(!(inode->i_state & I_FREEING));
--- a/include/linux/fs.h~mm-remove-nrexceptional-from-inode
+++ a/include/linux/fs.h

@@ -442,7 +442,6 @@ int pagecache_write_end(struct file *, s
  * @i_mmap: Tree of private and shared mappings.
  * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable.
  * @nrpages: Number of page entries, protected by the i_pages lock.
- * @nrexceptional: Shadow or DAX entries, protected by the i_pages lock.
  * @writeback_index: Writeback starts here.
  * @a_ops: Methods.
  * @flags: Error bits and flags (AS_*).

@@ -463,7 +462,6 @@ struct address_space {
 	struct rb_root_cached	i_mmap;
 	struct rw_semaphore	i_mmap_rwsem;
 	unsigned long		nrpages;
-	unsigned long		nrexceptional;
 	pgoff_t			writeback_index;
 	const struct address_space_operations *a_ops;
 	unsigned long		flags;

_

From: Hugh Dickins <hughd@google.com>
Subject: mm: remove nrexceptional from inode: remove BUG_ON

clear_inode()'s BUG_ON(!mapping_empty(&inode->i_data)) is unsafe: we know
of two ways in which nodes can and do (on rare occasions) get left behind.
Until those are fixed, do not BUG_ON() nor even WARN_ON().  Yes, this
will then leak those nodes (or the next user of the struct inode may use
them); but this has been happening for years, and the new
BUG_ON(!mapping_empty) was only guilty of revealing that.  A proper fix
will follow, but no hurry.

Link: https://lkml.kernel.org/r/alpine.LSU.2.11.2104292229380.16080@eggly.anvils
Signed-off-by: Hugh Dickins <hughd@google.com>
Cc: Matthew Wilcox <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/inode.c |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

--- a/fs/inode.c~mm-remove-nrexceptional-from-inode-remove-bug_on
+++ a/fs/inode.c

@@ -529,7 +529,14 @@ void clear_inode(struct inode *inode)
 	 */
 	xa_lock_irq(&inode->i_data.i_pages);
 	BUG_ON(inode->i_data.nrpages);
-	BUG_ON(!mapping_empty(&inode->i_data));
+	/*
+	 * Almost always, mapping_empty(&inode->i_data) here; but there are
+	 * two known and long-standing ways in which nodes may get left behind
+	 * (when deep radix-tree node allocation failed partway; or when THP
+	 * collapse_file() failed). Until those two known cases are cleaned up,
+	 * or a cleanup function is called here, do not BUG_ON(!mapping_empty),
+	 * nor even WARN_ON(!mapping_empty).
+	 */
 	xa_unlock_irq(&inode->i_data.i_pages);
 	BUG_ON(!list_empty(&inode->i_data.private_list));
 	BUG_ON(!(inode->i_state & I_FREEING));

_

From: Peter Xu <peterx@redhat.com>
Subject: hugetlb: pass vma into huge_pte_alloc() and huge_pmd_share()

Patch series "hugetlb: Disable huge pmd unshare for uffd-wp", v4.

This series tries to disable huge pmd unshare of hugetlbfs backed memory
for uffd-wp.  Although uffd-wp of hugetlbfs is still during rfc stage, the
idea of this series may be needed for multiple tasks (Axel's uffd minor
fault series, and Mike's soft dirty series), so I picked it out from the
larger series.


This patch (of 4):

It is a preparation work to be able to behave differently in the per
architecture huge_pte_alloc() according to different VMA attributes.

Pass it deeper into huge_pmd_share() so that we can avoid the find_vma() call.

[peterx@redhat.com: build fix]
  Link: https://lkml.kernel.org/r/20210304164653.GB397383@xz-x1Link: https://lkml.kernel.org/r/20210218230633.15028-1-peterx@redhat.com
Link: https://lkml.kernel.org/r/20210218230633.15028-2-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Suggested-by: Mike Kravetz <redacted>
Cc: Adam Ruprecht <redacted>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexey Dobriyan <redacted>
Cc: Andrea Arcangeli <redacted>
Cc: Anshuman Khandual <redacted>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Cannon Matthews <redacted>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chinwen Chang <redacted>
Cc: David Rientjes <rientjes@google.com>
Cc: "Dr . David Alan Gilbert" <redacted>
Cc: Huang Ying <redacted>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jerome Glisse <redacted>
Cc: Kirill A. Shutemov <redacted>
Cc: Lokesh Gidra <redacted>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: "Michal Koutn" <mkoutny@suse.com>
Cc: Michel Lespinasse <redacted>
Cc: Mike Rapoport <redacted>
Cc: Mina Almasry <redacted>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oliver Upton <redacted>
Cc: Shaohua Li <redacted>
Cc: Shawn Anastasio <redacted>
Cc: Steven Price <steven.price@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Vlastimil Babka <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 arch/arm64/mm/hugetlbpage.c   |    4 ++--
 arch/ia64/mm/hugetlbpage.c    |    3 ++-
 arch/mips/mm/hugetlbpage.c    |    4 ++--
 arch/parisc/mm/hugetlbpage.c  |    2 +-
 arch/powerpc/mm/hugetlbpage.c |    3 ++-
 arch/s390/mm/hugetlbpage.c    |    2 +-
 arch/sh/mm/hugetlbpage.c      |    2 +-
 arch/sparc/mm/hugetlbpage.c   |    2 +-
 include/linux/hugetlb.h       |    5 +++--
 mm/hugetlb.c                  |   15 ++++++++-------
 mm/userfaultfd.c              |    2 +-
 11 files changed, 24 insertions(+), 20 deletions(-)

--- a/arch/arm64/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/arm64/mm/hugetlbpage.c

@@ -252,7 +252,7 @@ void set_huge_swap_pte_at(struct mm_stru
 		set_pte(ptep, pte);
 }
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 		      unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgdp;

@@ -286,7 +286,7 @@ pte_t *huge_pte_alloc(struct mm_struct *
 	} else if (sz == PMD_SIZE) {
 		if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) &&
 		    pud_none(READ_ONCE(*pudp)))
-			ptep = huge_pmd_share(mm, addr, pudp);
+			ptep = huge_pmd_share(mm, vma, addr, pudp);
 		else
 			ptep = (pte_t *)pmd_alloc(mm, pudp, addr);
 	} else if (sz == (CONT_PMD_SIZE)) {
--- a/arch/ia64/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/ia64/mm/hugetlbpage.c

@@ -25,7 +25,8 @@ unsigned int hpage_shift = HPAGE_SHIFT_D
 EXPORT_SYMBOL(hpage_shift);
 
 pte_t *
-huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
+huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
+	       unsigned long addr, unsigned long sz)
 {
 	unsigned long taddr = htlbpage_to_page(addr);
 	pgd_t *pgd;
--- a/arch/mips/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/mips/mm/hugetlbpage.c

@@ -21,8 +21,8 @@
 #include <asm/tlb.h>
 #include <asm/tlbflush.h>
 
-pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr,
-		      unsigned long sz)
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
+		      unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgd;
 	p4d_t *p4d;
--- a/arch/parisc/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/parisc/mm/hugetlbpage.c

@@ -44,7 +44,7 @@ hugetlb_get_unmapped_area(struct file *f
 }
 
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgd;
--- a/arch/powerpc/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/powerpc/mm/hugetlbpage.c

@@ -106,7 +106,8 @@ static int __hugepte_alloc(struct mm_str
  * At this point we do the placement change only for BOOK3S 64. This would
  * possibly work on other subarchs.
  */
-pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
+		      unsigned long addr, unsigned long sz)
 {
 	pgd_t *pg;
 	p4d_t *p4;
--- a/arch/s390/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/s390/mm/hugetlbpage.c

@@ -189,7 +189,7 @@ pte_t huge_ptep_get_and_clear(struct mm_
 	return pte;
 }
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgdp;
--- a/arch/sh/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/sh/mm/hugetlbpage.c

@@ -21,7 +21,7 @@
 #include <asm/tlbflush.h>
 #include <asm/cacheflush.h>
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgd;
--- a/arch/sparc/mm/hugetlbpage.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/arch/sparc/mm/hugetlbpage.c

@@ -279,7 +279,7 @@ unsigned long pud_leaf_size(pud_t pud) {
 unsigned long pmd_leaf_size(pmd_t pmd) { return 1UL << tte_to_shift(*(pte_t *)&pmd); }
 unsigned long pte_leaf_size(pte_t pte) { return 1UL << tte_to_shift(pte); }
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgd;
--- a/include/linux/hugetlb.h~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/include/linux/hugetlb.h

@@ -152,7 +152,8 @@ void hugetlb_fix_reserve_counts(struct i
 extern struct mutex *hugetlb_fault_mutex_table;
 u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
 
-pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud);
+pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
+		      unsigned long addr, pud_t *pud);
 
 struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
 

@@ -161,7 +162,7 @@ extern struct list_head huge_boot_pages;
 
 /* arch callbacks */
 
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz);
 pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
--- a/mm/hugetlb.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/mm/hugetlb.c

@@ -3795,7 +3795,7 @@ int copy_hugetlb_page_range(struct mm_st
 		src_pte = huge_pte_offset(src, addr, sz);
 		if (!src_pte)
 			continue;
-		dst_pte = huge_pte_alloc(dst, addr, sz);
+		dst_pte = huge_pte_alloc(dst, vma, addr, sz);
 		if (!dst_pte) {
 			ret = -ENOMEM;
 			break;

@@ -4563,7 +4563,7 @@ vm_fault_t hugetlb_fault(struct mm_struc
 	 */
 	mapping = vma->vm_file->f_mapping;
 	i_mmap_lock_read(mapping);
-	ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
+	ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h));
 	if (!ptep) {
 		i_mmap_unlock_read(mapping);
 		return VM_FAULT_OOM;

@@ -5370,9 +5370,9 @@ void adjust_range_if_pmd_sharing_possibl
  * if !vma_shareable check at the beginning of the routine. i_mmap_rwsem is
  * only required for subsequent processing.
  */
-pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
+pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
+		      unsigned long addr, pud_t *pud)
 {
-	struct vm_area_struct *vma = find_vma(mm, addr);
 	struct address_space *mapping = vma->vm_file->f_mapping;
 	pgoff_t idx = ((addr - vma->vm_start) >> PAGE_SHIFT) +
 			vma->vm_pgoff;

@@ -5450,7 +5450,8 @@ int huge_pmd_unshare(struct mm_struct *m
 }
 #define want_pmd_share()	(1)
 #else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
-pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
+pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
+		      unsigned long addr, pud_t *pud)
 {
 	return NULL;
 }

@@ -5469,7 +5470,7 @@ void adjust_range_if_pmd_sharing_possibl
 #endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
 
 #ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB
-pte_t *huge_pte_alloc(struct mm_struct *mm,
+pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long addr, unsigned long sz)
 {
 	pgd_t *pgd;

@@ -5488,7 +5489,7 @@ pte_t *huge_pte_alloc(struct mm_struct *
 		} else {
 			BUG_ON(sz != PMD_SIZE);
 			if (want_pmd_share() && pud_none(*pud))
-				pte = huge_pmd_share(mm, addr, pud);
+				pte = huge_pmd_share(mm, vma, addr, pud);
 			else
 				pte = (pte_t *)pmd_alloc(mm, pud, addr);
 		}
--- a/mm/userfaultfd.c~hugetlb-pass-vma-into-huge_pte_alloc-and-huge_pmd_share
+++ a/mm/userfaultfd.c

@@ -290,7 +290,7 @@ retry:
 		mutex_lock(&hugetlb_fault_mutex_table[hash]);
 
 		err = -ENOMEM;
-		dst_pte = huge_pte_alloc(dst_mm, dst_addr, vma_hpagesize);
+		dst_pte = huge_pte_alloc(dst_mm, dst_vma, dst_addr, vma_hpagesize);
 		if (!dst_pte) {
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
 			i_mmap_unlock_read(mapping);

_

From: Peter Xu <peterx@redhat.com>
Subject: hugetlb/userfaultfd: forbid huge pmd sharing when uffd enabled

Huge pmd sharing could bring problem to userfaultfd.  The thing is that
userfaultfd is running its logic based on the special bits on page table
entries, however the huge pmd sharing could potentially share page table
entries for different address ranges.  That could cause issues on either:

  - When sharing huge pmd page tables for an uffd write protected range, the
    newly mapped huge pmd range will also be write protected unexpectedly, or,

  - When we try to write protect a range of huge pmd shared range, we'll first
    do huge_pmd_unshare() in hugetlb_change_protection(), however that also
    means the UFFDIO_WRITEPROTECT could be silently skipped for the shared
    region, which could lead to data loss.

Since at it, a few other things are done altogether:

  - Move want_pmd_share() from mm/hugetlb.c into linux/hugetlb.h, because
    that's definitely something that arch code would like to use too

  - ARM64 currently directly check against CONFIG_ARCH_WANT_HUGE_PMD_SHARE when
    trying to share huge pmd.  Switch to the want_pmd_share() helper.

Since at it, move vma_shareable() from huge_pmd_share() into want_pmd_share().

[peterx@redhat.com: fix build with !ARCH_WANT_HUGE_PMD_SHARE]
  Link: https://lkml.kernel.org/r/20210310185359.88297-1-peterx@redhat.com
Link: https://lkml.kernel.org/r/20210218231202.15426-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mike Kravetz <redacted>
Reviewed-by: Axel Rasmussen <axelrasmussen@google.com>
Tested-by: Naresh Kamboju <redacted>
Cc: Adam Ruprecht <redacted>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexey Dobriyan <redacted>
Cc: Andrea Arcangeli <redacted>
Cc: Anshuman Khandual <redacted>
Cc: Cannon Matthews <redacted>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chinwen Chang <redacted>
Cc: David Rientjes <rientjes@google.com>
Cc: "Dr . David Alan Gilbert" <redacted>
Cc: Huang Ying <redacted>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jerome Glisse <redacted>
Cc: Kirill A. Shutemov <redacted>
Cc: Lokesh Gidra <redacted>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: "Michal Koutn" <mkoutny@suse.com>
Cc: Michel Lespinasse <redacted>
Cc: Mike Rapoport <redacted>
Cc: Mina Almasry <redacted>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oliver Upton <redacted>
Cc: Shaohua Li <redacted>
Cc: Shawn Anastasio <redacted>
Cc: Steven Price <steven.price@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Vlastimil Babka <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 arch/arm64/mm/hugetlbpage.c   |    3 +--
 include/linux/hugetlb.h       |    2 ++
 include/linux/userfaultfd_k.h |    9 +++++++++
 mm/hugetlb.c                  |   22 ++++++++++++++++------
 4 files changed, 28 insertions(+), 8 deletions(-)

--- a/arch/arm64/mm/hugetlbpage.c~hugetlb-userfaultfd-forbid-huge-pmd-sharing-when-uffd-enabled
+++ a/arch/arm64/mm/hugetlbpage.c

@@ -284,8 +284,7 @@ pte_t *huge_pte_alloc(struct mm_struct *
 		 */
 		ptep = pte_alloc_map(mm, pmdp, addr);
 	} else if (sz == PMD_SIZE) {
-		if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) &&
-		    pud_none(READ_ONCE(*pudp)))
+		if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp)))
 			ptep = huge_pmd_share(mm, vma, addr, pudp);
 		else
 			ptep = (pte_t *)pmd_alloc(mm, pudp, addr);
--- a/include/linux/hugetlb.h~hugetlb-userfaultfd-forbid-huge-pmd-sharing-when-uffd-enabled
+++ a/include/linux/hugetlb.h

@@ -1040,4 +1040,6 @@ static inline __init void hugetlb_cma_ch
 }
 #endif
 
+bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
+
 #endif /* _LINUX_HUGETLB_H */
--- a/include/linux/userfaultfd_k.h~hugetlb-userfaultfd-forbid-huge-pmd-sharing-when-uffd-enabled
+++ a/include/linux/userfaultfd_k.h

@@ -52,6 +52,15 @@ static inline bool is_mergeable_vm_userf
 	return vma->vm_userfaultfd_ctx.ctx == vm_ctx.ctx;
 }
 
+/*
+ * Never enable huge pmd sharing on uffd-wp registered vmas, because uffd-wp
+ * protect information is per pgtable entry.
+ */
+static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)
+{
+	return vma->vm_flags & VM_UFFD_WP;
+}
+
 static inline bool userfaultfd_missing(struct vm_area_struct *vma)
 {
 	return vma->vm_flags & VM_UFFD_MISSING;
--- a/mm/hugetlb.c~hugetlb-userfaultfd-forbid-huge-pmd-sharing-when-uffd-enabled
+++ a/mm/hugetlb.c

@@ -5326,6 +5326,15 @@ static bool vma_shareable(struct vm_area
 	return false;
 }
 
+bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr)
+{
+#ifdef CONFIG_USERFAULTFD
+	if (uffd_disable_huge_pmd_share(vma))
+		return false;
+#endif
+	return vma_shareable(vma, addr);
+}
+
 /*
  * Determine if start,end range within vma could be mapped by shared pmd.
  * If yes, adjust start and end to cover range associated with possible

@@ -5382,9 +5391,6 @@ pte_t *huge_pmd_share(struct mm_struct *
 	pte_t *pte;
 	spinlock_t *ptl;
 
-	if (!vma_shareable(vma, addr))
-		return (pte_t *)pmd_alloc(mm, pud, addr);
-
 	i_mmap_assert_locked(mapping);
 	vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
 		if (svma == vma)

@@ -5448,7 +5454,7 @@ int huge_pmd_unshare(struct mm_struct *m
 	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;
 }
-#define want_pmd_share()	(1)
+
 #else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
 pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 		      unsigned long addr, pud_t *pud)

@@ -5466,7 +5472,11 @@ void adjust_range_if_pmd_sharing_possibl
 				unsigned long *start, unsigned long *end)
 {
 }
-#define want_pmd_share()	(0)
+
+bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr)
+{
+	return false;
+}
 #endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
 
 #ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB

@@ -5488,7 +5498,7 @@ pte_t *huge_pte_alloc(struct mm_struct *
 			pte = (pte_t *)pud;
 		} else {
 			BUG_ON(sz != PMD_SIZE);
-			if (want_pmd_share() && pud_none(*pud))
+			if (want_pmd_share(vma, addr) && pud_none(*pud))
 				pte = huge_pmd_share(mm, vma, addr, pud);
 			else
 				pte = (pte_t *)pmd_alloc(mm, pud, addr);

_

From: Peter Xu <peterx@redhat.com>
Subject: mm/hugetlb: move flush_hugetlb_tlb_range() into hugetlb.h

Prepare for it to be called outside of mm/hugetlb.c.

Link: https://lkml.kernel.org/r/20210218231204.15474-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mike Kravetz <redacted>
Reviewed-by: Axel Rasmussen <axelrasmussen@google.com>
Cc: Adam Ruprecht <redacted>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexey Dobriyan <redacted>
Cc: Andrea Arcangeli <redacted>
Cc: Anshuman Khandual <redacted>
Cc: Cannon Matthews <redacted>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chinwen Chang <redacted>
Cc: David Rientjes <rientjes@google.com>
Cc: "Dr . David Alan Gilbert" <redacted>
Cc: Huang Ying <redacted>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jerome Glisse <redacted>
Cc: Kirill A. Shutemov <redacted>
Cc: Lokesh Gidra <redacted>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: "Michal Koutn" <mkoutny@suse.com>
Cc: Michel Lespinasse <redacted>
Cc: Mike Rapoport <redacted>
Cc: Mina Almasry <redacted>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oliver Upton <redacted>
Cc: Shaohua Li <redacted>
Cc: Shawn Anastasio <redacted>
Cc: Steven Price <steven.price@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Vlastimil Babka <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 include/linux/hugetlb.h |    8 ++++++++
 mm/hugetlb.c            |    8 --------
 2 files changed, 8 insertions(+), 8 deletions(-)

--- a/include/linux/hugetlb.h~mm-hugetlb-move-flush_hugetlb_tlb_range-into-hugetlbh
+++ a/include/linux/hugetlb.h

@@ -1042,4 +1042,12 @@ static inline __init void hugetlb_cma_ch
 
 bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
 
+#ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
+/*
+ * ARCHes with special requirements for evicting HUGETLB backing TLB entries can
+ * implement this.
+ */
+#define flush_hugetlb_tlb_range(vma, addr, end)	flush_tlb_range(vma, addr, end)
+#endif
+
 #endif /* _LINUX_HUGETLB_H */
--- a/mm/hugetlb.c~mm-hugetlb-move-flush_hugetlb_tlb_range-into-hugetlbh
+++ a/mm/hugetlb.c

@@ -4996,14 +4996,6 @@ long follow_hugetlb_page(struct mm_struc
 	return i ? i : err;
 }
 
-#ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
-/*
- * ARCHes with special requirements for evicting HUGETLB backing TLB entries can
- * implement this.
- */
-#define flush_hugetlb_tlb_range(vma, addr, end)	flush_tlb_range(vma, addr, end)
-#endif
-
 unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 		unsigned long address, unsigned long end, pgprot_t newprot)
 {

_

From: Peter Xu <peterx@redhat.com>
Subject: hugetlb/userfaultfd: unshare all pmds for hugetlbfs when register wp

Huge pmd sharing for hugetlbfs is racy with userfaultfd-wp because
userfaultfd-wp is always based on pgtable entries, so they cannot be
shared.

Walk the hugetlb range and unshare all such mappings if there is, right
before UFFDIO_REGISTER will succeed and return to userspace.

This will pair with want_pmd_share() in hugetlb code so that huge pmd
sharing is completely disabled for userfaultfd-wp registered range.

Link: https://lkml.kernel.org/r/20210218231206.15524-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mike Kravetz <redacted>
Cc: Peter Xu <peterx@redhat.com>
Cc: Andrea Arcangeli <redacted>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Mike Rapoport <redacted>
Cc: Kirill A. Shutemov <redacted>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Adam Ruprecht <redacted>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alexey Dobriyan <redacted>
Cc: Anshuman Khandual <redacted>
Cc: Cannon Matthews <redacted>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chinwen Chang <redacted>
Cc: David Rientjes <rientjes@google.com>
Cc: "Dr . David Alan Gilbert" <redacted>
Cc: Huang Ying <redacted>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Jerome Glisse <redacted>
Cc: Lokesh Gidra <redacted>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: "Michal Koutn" <mkoutny@suse.com>
Cc: Michel Lespinasse <redacted>
Cc: Mina Almasry <redacted>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oliver Upton <redacted>
Cc: Shaohua Li <redacted>
Cc: Shawn Anastasio <redacted>
Cc: Steven Price <steven.price@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Vlastimil Babka <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/userfaultfd.c        |    4 ++
 include/linux/hugetlb.h |    3 ++
 mm/hugetlb.c            |   51 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+)

--- a/fs/userfaultfd.c~hugetlb-userfaultfd-unshare-all-pmds-for-hugetlbfs-when-register-wp
+++ a/fs/userfaultfd.c

@@ -15,6 +15,7 @@
 #include <linux/sched/signal.h>
 #include <linux/sched/mm.h>
 #include <linux/mm.h>
+#include <linux/mmu_notifier.h>
 #include <linux/poll.h>
 #include <linux/slab.h>
 #include <linux/seq_file.h>

@@ -1449,6 +1450,9 @@ static int userfaultfd_register(struct u
 		vma->vm_flags = new_flags;
 		vma->vm_userfaultfd_ctx.ctx = ctx;
 
+		if (is_vm_hugetlb_page(vma) && uffd_disable_huge_pmd_share(vma))
+			hugetlb_unshare_all_pmds(vma);
+
 	skip:
 		prev = vma;
 		start = vma->vm_end;
--- a/include/linux/hugetlb.h~hugetlb-userfaultfd-unshare-all-pmds-for-hugetlbfs-when-register-wp
+++ a/include/linux/hugetlb.h

@@ -188,6 +188,7 @@ unsigned long hugetlb_change_protection(
 		unsigned long address, unsigned long end, pgprot_t newprot);
 
 bool is_hugetlb_entry_migration(pte_t pte);
+void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
 
 #else /* !CONFIG_HUGETLB_PAGE */
 

@@ -369,6 +370,8 @@ static inline vm_fault_t hugetlb_fault(s
 	return 0;
 }
 
+static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }
+
 #endif /* !CONFIG_HUGETLB_PAGE */
 /*
  * hugepages at page global directory. If arch support
--- a/mm/hugetlb.c~hugetlb-userfaultfd-unshare-all-pmds-for-hugetlbfs-when-register-wp
+++ a/mm/hugetlb.c

@@ -5691,6 +5691,57 @@ void move_hugetlb_state(struct page *old
 	}
 }
 
+/*
+ * This function will unconditionally remove all the shared pmd pgtable entries
+ * within the specific vma for a hugetlbfs memory range.
+ */
+void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
+{
+	struct hstate *h = hstate_vma(vma);
+	unsigned long sz = huge_page_size(h);
+	struct mm_struct *mm = vma->vm_mm;
+	struct mmu_notifier_range range;
+	unsigned long address, start, end;
+	spinlock_t *ptl;
+	pte_t *ptep;
+
+	if (!(vma->vm_flags & VM_MAYSHARE))
+		return;
+
+	start = ALIGN(vma->vm_start, PUD_SIZE);
+	end = ALIGN_DOWN(vma->vm_end, PUD_SIZE);
+
+	if (start >= end)
+		return;
+
+	/*
+	 * No need to call adjust_range_if_pmd_sharing_possible(), because
+	 * we have already done the PUD_SIZE alignment.
+	 */
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
+				start, end);
+	mmu_notifier_invalidate_range_start(&range);
+	i_mmap_lock_write(vma->vm_file->f_mapping);
+	for (address = start; address < end; address += PUD_SIZE) {
+		unsigned long tmp = address;
+
+		ptep = huge_pte_offset(mm, address, sz);
+		if (!ptep)
+			continue;
+		ptl = huge_pte_lock(h, mm, ptep);
+		/* We don't want 'address' to be changed */
+		huge_pmd_unshare(mm, vma, &tmp, ptep);
+		spin_unlock(ptl);
+	}
+	flush_hugetlb_tlb_range(vma, start, end);
+	i_mmap_unlock_write(vma->vm_file->f_mapping);
+	/*
+	 * No need to call mmu_notifier_invalidate_range(), see
+	 * Documentation/vm/mmu_notifier.rst.
+	 */
+	mmu_notifier_invalidate_range_end(&range);
+}
+
 #ifdef CONFIG_CMA
 static bool cma_reserve_called __initdata;
 

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb: remove redundant reservation check condition in alloc_huge_page()

vma_resv_map(vma) checks if a reserve map is associated with the vma.  The
routine vma_needs_reservation() will check vma_resv_map(vma) and return 1
if no reserv map is present.  map_chg is set to the return value of
vma_needs_reservation().  Therefore, !vma_resv_map(vma) is redundant in
the expression:

	map_chg || avoid_reserve || !vma_resv_map(vma);

Remove the redundant check.

[Thanks Mike Kravetz for reshaping this commit message!]

Link: https://lkml.kernel.org/r/20210301104726.45159-1-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Reviewed-by: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/hugetlb.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/mm/hugetlb.c~mm-hugetlb-remove-redundant-reservation-check-condition-in-alloc_huge_page
+++ a/mm/hugetlb.c

@@ -2316,7 +2316,7 @@ struct page *alloc_huge_page(struct vm_a
 
 	/* If this allocation is not consuming a reservation, charge it now.
 	 */
-	deferred_reserve = map_chg || avoid_reserve || !vma_resv_map(vma);
+	deferred_reserve = map_chg || avoid_reserve;
 	if (deferred_reserve) {
 		ret = hugetlb_cgroup_charge_cgroup_rsvd(
 			idx, pages_per_huge_page(h), &h_cg);

_

From: Anshuman Khandual <redacted>
Subject: mm: generalize HUGETLB_PAGE_SIZE_VARIABLE

HUGETLB_PAGE_SIZE_VARIABLE need not be defined for each individual
platform subscribing it.  Instead just make it generic.

Link: https://lkml.kernel.org/r/1614914928-22039-1-git-send-email-anshuman.khandual@arm.com
Signed-off-by: Anshuman Khandual <redacted>
Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>	[powerpc]
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <redacted>
Cc: Christophe Leroy <redacted>
Cc: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 arch/ia64/Kconfig    |    6 +-----
 arch/powerpc/Kconfig |    6 +-----
 mm/Kconfig           |    7 +++++++
 3 files changed, 9 insertions(+), 10 deletions(-)

--- a/arch/ia64/Kconfig~mm-generalize-hugetlb_page_size_variable
+++ a/arch/ia64/Kconfig

@@ -32,6 +32,7 @@ config IA64
 	select TTY
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_VIRT_CPU_ACCOUNTING
+	select HUGETLB_PAGE_SIZE_VARIABLE if HUGETLB_PAGE
 	select VIRT_TO_BUS
 	select GENERIC_IRQ_PROBE
 	select GENERIC_PENDING_IRQ if SMP

@@ -82,11 +83,6 @@ config STACKTRACE_SUPPORT
 config GENERIC_LOCKBREAK
 	def_bool n
 
-config HUGETLB_PAGE_SIZE_VARIABLE
-	bool
-	depends on HUGETLB_PAGE
-	default y
-
 config GENERIC_CALIBRATE_DELAY
 	bool
 	default y
--- a/arch/powerpc/Kconfig~mm-generalize-hugetlb_page_size_variable
+++ a/arch/powerpc/Kconfig

@@ -232,6 +232,7 @@ config PPC
 	select HAVE_HARDLOCKUP_DETECTOR_PERF	if PERF_EVENTS && HAVE_PERF_EVENTS_NMI && !HAVE_HARDLOCKUP_DETECTOR_ARCH
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
+	select HUGETLB_PAGE_SIZE_VARIABLE	if PPC_BOOK3S_64 && HUGETLB_PAGE
 	select MMU_GATHER_RCU_TABLE_FREE
 	select MMU_GATHER_PAGE_SIZE
 	select HAVE_REGS_AND_STACK_ACCESS_API

@@ -416,11 +417,6 @@ config HIGHMEM
 
 source "kernel/Kconfig.hz"
 
-config HUGETLB_PAGE_SIZE_VARIABLE
-	bool
-	depends on HUGETLB_PAGE && PPC_BOOK3S_64
-	default y
-
 config MATH_EMULATION
 	bool "Math emulation"
 	depends on 4xx || PPC_8xx || PPC_MPC832x || BOOKE
--- a/mm/Kconfig~mm-generalize-hugetlb_page_size_variable
+++ a/mm/Kconfig

@@ -273,6 +273,13 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
 config ARCH_ENABLE_THP_MIGRATION
 	bool
 
+config HUGETLB_PAGE_SIZE_VARIABLE
+	def_bool n
+	help
+	  Allows the pageblock_order value to be dynamic instead of just standard
+	  HUGETLB_PAGE_ORDER when there are multiple HugeTLB page sizes available
+	  on a platform.
+
 config CONTIG_ALLOC
 	def_bool (MEMORY_ISOLATION && COMPACTION) || CMA
 

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb: use some helper functions to cleanup code

Patch series "Some cleanups for hugetlb".

This series contains cleanups to remove unnecessary VM_BUG_ON_PAGE, use
helper function and so on.  I also collect some previous patches into this
series in case they are forgotten.


This patch (of 5):

We could use pages_per_huge_page to get the number of pages per hugepage,
use get_hstate_idx to calculate hstate index, and use hstate_is_gigantic
to check if a hstate is gigantic to make code more succinct.

Link: https://lkml.kernel.org/r/20210308112809.26107-1-linmiaohe@huawei.com
Link: https://lkml.kernel.org/r/20210308112809.26107-2-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Reviewed-by: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/hugetlbfs/inode.c |    2 +-
 mm/hugetlb.c         |    6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

--- a/fs/hugetlbfs/inode.c~mm-hugetlb-use-some-helper-functions-to-cleanup-code
+++ a/fs/hugetlbfs/inode.c

@@ -1435,7 +1435,7 @@ static int get_hstate_idx(int page_size_
 
 	if (!h)
 		return -1;
-	return h - hstates;
+	return hstate_index(h);
 }
 
 /*
--- a/mm/hugetlb.c~mm-hugetlb-use-some-helper-functions-to-cleanup-code
+++ a/mm/hugetlb.c

@@ -1273,7 +1273,7 @@ static void free_gigantic_page(struct pa
 static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask,
 		int nid, nodemask_t *nodemask)
 {
-	unsigned long nr_pages = 1UL << huge_page_order(h);
+	unsigned long nr_pages = pages_per_huge_page(h);
 	if (nid == NUMA_NO_NODE)
 		nid = numa_mem_id();
 

@@ -3267,10 +3267,10 @@ static int __init hugepages_setup(char *
 
 	/*
 	 * Global state is always initialized later in hugetlb_init.
-	 * But we need to allocate >= MAX_ORDER hstates here early to still
+	 * But we need to allocate gigantic hstates here early to still
 	 * use the bootmem allocator.
 	 */
-	if (hugetlb_max_hstate && parsed_hstate->order >= MAX_ORDER)
+	if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate))
 		hugetlb_hstate_alloc_pages(parsed_hstate);
 
 	last_mhp = mhp;

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb_cgroup: remove unnecessary VM_BUG_ON_PAGE in hugetlb_cgroup_migrate()

!PageHuge(oldhpage) is implicitly checked in page_hstate() above, so we
remove this explicit one.

Link: https://lkml.kernel.org/r/20210308112809.26107-4-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/hugetlb_cgroup.c |    1 -
 1 file changed, 1 deletion(-)

--- a/mm/hugetlb_cgroup.c~hugetlb_cgroup-remove-unnecessary-vm_bug_on_page-in-hugetlb_cgroup_migrate
+++ a/mm/hugetlb_cgroup.c

@@ -784,7 +784,6 @@ void hugetlb_cgroup_migrate(struct page
 	if (hugetlb_cgroup_disabled())
 		return;
 
-	VM_BUG_ON_PAGE(!PageHuge(oldhpage), oldhpage);
 	spin_lock(&hugetlb_lock);
 	h_cg = hugetlb_cgroup_from_page(oldhpage);
 	h_cg_rsvd = hugetlb_cgroup_from_page_rsvd(oldhpage);

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb: optimize the surplus state transfer code in move_hugetlb_state()

We should not transfer the per-node surplus state when we do not cross the
node in order to save some cpu cycles

Link: https://lkml.kernel.org/r/20210308112809.26107-3-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Reviewed-by: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/hugetlb.c |    6 ++++++
 1 file changed, 6 insertions(+)

--- a/mm/hugetlb.c~mm-hugetlb-optimize-the-surplus-state-transfer-code-in-move_hugetlb_state
+++ a/mm/hugetlb.c

@@ -5682,6 +5682,12 @@ void move_hugetlb_state(struct page *old
 		SetHPageTemporary(oldpage);
 		ClearHPageTemporary(newpage);
 
+		/*
+		 * There is no need to transfer the per-node surplus state
+		 * when we do not cross the node.
+		 */
+		if (new_nid == old_nid)
+			return;
 		spin_lock(&hugetlb_lock);
 		if (h->surplus_huge_pages_node[old_nid]) {
 			h->surplus_huge_pages_node[old_nid]--;

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb: simplify the code when alloc_huge_page() failed in hugetlb_no_page()

Rework the error handling code when alloc_huge_page() failed to remove
some duplicated code and simplify the code slightly.

Link: https://lkml.kernel.org/r/20210308112809.26107-5-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/hugetlb.c |    9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

--- a/mm/hugetlb.c~mm-hugetlb-simplify-the-code-when-alloc_huge_page-failed-in-hugetlb_no_page
+++ a/mm/hugetlb.c

@@ -4395,13 +4395,10 @@ retry:
 			 * sure there really is no pte entry.
 			 */
 			ptl = huge_pte_lock(h, mm, ptep);
-			if (!huge_pte_none(huge_ptep_get(ptep))) {
-				ret = 0;
-				spin_unlock(ptl);
-				goto out;
-			}
+			ret = 0;
+			if (huge_pte_none(huge_ptep_get(ptep)))
+				ret = vmf_error(PTR_ERR(page));
 			spin_unlock(ptl);
-			ret = vmf_error(PTR_ERR(page));
 			goto out;
 		}
 		clear_huge_page(page, address, pages_per_huge_page(h));

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: mm/hugetlb: avoid calculating fault_mutex_hash in truncate_op case

The fault_mutex hashing overhead can be avoided in truncate_op case
because page faults can not race with truncation in this routine.  So
calculate hash for fault_mutex only in !truncate_op case to save some cpu
cycles.

Link: https://lkml.kernel.org/r/20210308112809.26107-6-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/hugetlbfs/inode.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

--- a/fs/hugetlbfs/inode.c~mm-hugetlb-avoid-calculating-fault_mutex_hash-in-truncate_op-case
+++ a/fs/hugetlbfs/inode.c

@@ -482,10 +482,9 @@ static void remove_inode_hugepages(struc
 
 		for (i = 0; i < pagevec_count(&pvec); ++i) {
 			struct page *page = pvec.pages[i];
-			u32 hash;
+			u32 hash = 0;
 
 			index = page->index;
-			hash = hugetlb_fault_mutex_hash(mapping, index);
 			if (!truncate_op) {
 				/*
 				 * Only need to hold the fault mutex in the

@@ -493,6 +492,7 @@ static void remove_inode_hugepages(struc
 				 * page faults.  Races are not possible in the
 				 * case of truncation.
 				 */
+				hash = hugetlb_fault_mutex_hash(mapping, index);
 				mutex_lock(&hugetlb_fault_mutex_table[hash]);
 			}
 

_

From: Miaohe Lin <linmiaohe@huawei.com>
Subject: khugepaged: remove unneeded return value of khugepaged_collapse_pte_mapped_thps()

Patch series "Cleanup and fixup for khugepaged", v2.

This series contains cleanups to remove unneeded return value, use helper
function and so on.  And there is one fix to correct the wrong result
value for trace_mm_collapse_huge_page_isolate().


This patch (of 4):

The return value of khugepaged_collapse_pte_mapped_thps() is never checked
since it's introduced.  We should remove such unneeded return value.

Link: https://lkml.kernel.org/r/20210306032947.35921-1-linmiaohe@huawei.com
Link: https://lkml.kernel.org/r/20210306032947.35921-2-linmiaohe@huawei.com
Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
Acked-by: Kirill A. Shutemov <redacted>
Cc: Rik van Riel <redacted>
Cc: Ebru Akagunduz <redacted>
Cc: Dan Carpenter <redacted>
Cc: Mike Kravetz <redacted>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/khugepaged.c |   10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

--- a/mm/khugepaged.c~khugepaged-remove-unneeded-return-value-of-khugepaged_collapse_pte_mapped_thps
+++ a/mm/khugepaged.c

@@ -1533,16 +1533,16 @@ abort:
 	goto drop_hpage;
 }
 
-static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot)
+static void khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot)
 {
 	struct mm_struct *mm = mm_slot->mm;
 	int i;
 
 	if (likely(mm_slot->nr_pte_mapped_thp == 0))
-		return 0;
+		return;
 
 	if (!mmap_write_trylock(mm))
-		return -EBUSY;
+		return;
 
 	if (unlikely(khugepaged_test_exit(mm)))
 		goto out;