Re: [PATCH v5 00/38] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL

On Fri, 09 Oct 2026 19:03:54 +0100 "Lorenzo Stoakes (ARM)" [off-list ref] wrote:

The VM_SPECIAL / VMA_SPECIAL_FLAGS mask conflates several unrelated
properties:

* Is this managed by core mm, or by a driver mapping MMIO, kernel-allocated
  pages, or even ordinary pages itself?
* Can it be expanded or merged?
* Is this a 'weird' case like mlock where migration might race and we
  'have' to set invalid flags to notify?
* Is it another 'weird' case where we just want to stop GUP from touching
  it?

Driver writers have often been confused about this, and who can blame them?

It also interacts badly with the eternal edgecase known as hugetlb - which
sets VMA_DONTEXPAND_BIT but doesn't also want to be treated like a
'special' flag.

Another issue is that we cannot make sensible assumptions about flag
use. It's not possible to assume VMA_IO_BIT means iommu because drivers
abuse it and mlock abuses it.

Special is also an overloaded term in mm. VDSO and VVAR mappings are also
called 'special' but they're special in a... special way.

Sometimes things are called special that are a subset of
VMA_SPECIAL_FLAGS (VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT for instance when it
comes to zapping or vm_normal_folio()).

There's a specific kind of special for THP too, which considers
PFN map, mixed map 'special' but DAX not.

It's all rather a mess.

This series brings some order to things by both limiting what drivers can
do with VMA flags and switching to using predicates that describe
behaviour, not arbitrary flags.

Thanks, I've updated mm-unstable to this version.

For some reason, new versions of this series aren't applying when I
drop the new series in place of the old.  I made the same old bunch of
reject-fixups, so the below interdiff will show if I messed any of them up.

v5:
* Applied tags (thanks everybody!)
* Updated 3/38 to fix a bug in __mmap_new_file_vma() by zapping + closing
  the VMA before setting vma->vm_file NULL to avoid an i_mmap_writable
  imbalance. A repro was tested before/after and confirmed fixed.
* Updated 7/38 to make __map_kernel_pages_prepare() use
  mmap_assert_write_locked() rather than the weird
  assert-on-mmap_read_trylock() pattern, as per Suren.
* Also update vm_insert_page(), vm_insert_pages() to fix the same pattern
  and to use the new VMA API, and converted a BUG_ON to a
  VM_WARN_ON_ONCE().

Here's how v5 altered mm.git:

 mm/memory.c                       |   20 +++++++++++---------
 mm/vma.c                          |   16 ++++++++--------
 tools/testing/vma/include/stubs.h |    4 ++++
 3 files changed, 23 insertions(+), 17 deletions(-)

--- a/mm/memory.c~b
+++ a/mm/memory.c

@@ -2611,10 +2611,11 @@ int vm_insert_pages(struct vm_area_struc
 
 	if (!range_in_vma(vma, addr, end))
 		return -EFAULT;
-	if (!(vma->vm_flags & VM_MIXEDMAP)) {
-		VM_WARN_ON_ONCE(mmap_read_trylock(vma->vm_mm));
-		VM_WARN_ON_ONCE(vma->vm_flags & VM_PFNMAP);
-		vm_flags_set(vma, VM_MIXEDMAP);
+	if (!vma_test(vma, VMA_MIXEDMAP_BIT)) {
+		mmap_assert_write_locked(vma->vm_mm);
+		VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT));
+		vma_start_write(vma);
+		vma_set_flags(vma, VMA_MIXEDMAP_BIT);
 	}
 	/* Defer page refcount checking till we're about to map that page. */
 	return insert_pages(vma, addr, pages, num, vma->vm_page_prot);

@@ -2626,7 +2627,7 @@ static void __map_kernel_pages_prepare(s
 	if (vma_desc_test(desc, VMA_MIXEDMAP_BIT))
 		return;
 
-	VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm));
+	mmap_assert_write_locked(desc->mm);
 	VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT));
 	vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT);
 }

@@ -2785,10 +2786,11 @@ int vm_insert_page(struct vm_area_struct
 {
 	if (addr < vma->vm_start || addr >= vma->vm_end)
 		return -EFAULT;
-	if (!(vma->vm_flags & VM_MIXEDMAP)) {
-		BUG_ON(mmap_read_trylock(vma->vm_mm));
-		BUG_ON(vma->vm_flags & VM_PFNMAP);
-		vm_flags_set(vma, VM_MIXEDMAP);
+	if (!vma_test(vma, VMA_MIXEDMAP_BIT)) {
+		mmap_assert_write_locked(vma->vm_mm);
+		VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT));
+		vma_start_write(vma);
+		vma_set_flags(vma, VMA_MIXEDMAP_BIT);
 	}
 	return insert_page(vma, addr, page, vma->vm_page_prot, false);
 }
--- a/mm/vma.c~b
+++ a/mm/vma.c

@@ -2631,21 +2631,21 @@ static int __mmap_new_file_vma(struct mm
 	map->vm_file = vma->vm_file;
 
 	if (error) {
-		struct file *file = vma->vm_file;
 		UNMAP_STATE(unmap, vmi, vma, vma->vm_start, vma->vm_end,
 			    map->prev, map->next);
 
+		if (vma->vm_ops->close) {
+			zap_vma(vma);
+			/* Safe, because VMA zapped. */
+			vma_close(vma);
+		}
+		if (map_same_file(map))
+			fput(map->vm_file);
+
 		vma->vm_file = NULL;
 		vma_iter_set(vmi, vma->vm_end);
 		/* Undo any partial mapping done by a device driver. */
 		unmap_region(&unmap);
-		vma->vm_file = file;
-		/* Only safe once unmapped. */
-		vma_close(vma);
-
-		if (map_same_file(map))
-			fput(map->vm_file);
-		vma->vm_file = NULL;
 		return error;
 	}
 
--- a/tools/testing/vma/include/stubs.h~b
+++ a/tools/testing/vma/include/stubs.h

@@ -329,6 +329,10 @@ static inline void vma_close(struct vm_a
 {
 }
 
+static inline void zap_vma(struct vm_area_struct *vma)
+{
+}
+
 static inline int mmap_file(struct file *file, struct vm_area_struct *vma)
 {
 	return 0;

_