From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:21
Based on mm/mm-new.
In the future, as we decouple "struct page" from "struct folio", pages
that support "non-lru page migration" -- movable_ops page migration
such as memory balloons and zsmalloc -- will no longer be folios. They
will not have ->mapping, ->lru, and likely no refcount and no
page lock. But they will have flags :)
This is the first part (other parts not written yet) of decoupling
movable_ops page migration from folio migration.
In this series, we get rid of the ->mapping usage, and start cleaning up
the code + separating it from folio migration.
Migration core will have to be further reworked to not treat movable_ops
pages like folios. This is the first step into that direction.
RFC because I only did some light testing with virtio-balloon + zsmalloc
page migration, and have to polish some patch descriptions.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <redacted>
Cc: Jerrin Shaji George <jerrin.shaji-george@broadcom.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: David Hildenbrand <redacted>
Cc: Jason Wang <redacted>
Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: "Eugenio Pérez" <eperezma@redhat.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Lorenzo Stoakes <redacted>
Cc: "Liam R. Howlett" <redacted>
Cc: Vlastimil Babka <redacted>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Brendan Jackman <redacted>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Xu Xin <xu.xin16@zte.com.cn>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Rik van Riel <riel@surriel.com>
Cc: Harry Yoo <redacted>
Cc: Qi Zheng <redacted>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
David Hildenbrand (29):
mm/balloon_compaction: we cannot have isolated pages in the balloon
list
mm/balloon_compaction: convert balloon_page_delete() to
balloon_page_finalize()
mm/zsmalloc: drop PageIsolated() related VM_BUG_ONs
mm/page_alloc: allow for making page types sticky until freed
mm/balloon_compaction: make PageOffline sticky
mm/zsmalloc: make PageZsmalloc() sticky
mm/migrate: rename isolate_movable_page() to
isolate_movable_ops_page()
mm/migrate: rename putback_movable_folio() to
putback_movable_ops_page()
mm/migrate: factor out movable_ops page handling into
migrate_movable_ops_page()
mm/migrate: remove folio_test_movable() and folio_movable_ops()
mm/migrate: move movable_ops page handling out of move_to_new_folio()
mm/zsmalloc: stop using __ClearPageMovable()
mm/balloon_compaction: stop using __ClearPageMovable()
mm/migrate: remove __ClearPageMovable()
mm/migration: remove PageMovable()
mm: rename __PageMovable() to page_has_movable_ops()
mm/page_isolation: drop __folio_test_movable() check for large folios
mm: remove __folio_test_movable()
mm: stop storing migration_ops in page->mapping
mm: convert "movable" flag in page->mapping to a page flag
mm: rename PG_isolated to PG_movable_ops_isolated
mm/page-flags: rename PAGE_MAPPING_MOVABLE to PAGE_MAPPING_ANON_KSM
mm/page-alloc: remove PageMappingFlags()
mm/page-flags: remove folio_mapping_flags()
mm: simplify folio_expected_ref_count()
mm: rename PAGE_MAPPING_* to FOLIO_MAPPING_*
docs/mm: convert from "Non-LRU page migration" to "movable_ops page
migration"
mm/balloon_compaction: "movable_ops" doc updates
mm/balloon_compaction: provide single balloon_page_insert() and
balloon_mapping_gfp_mask()
Documentation/mm/page_migration.rst | 39 ++--
arch/powerpc/platforms/pseries/cmm.c | 2 +-
drivers/misc/vmw_balloon.c | 3 +-
drivers/virtio/virtio_balloon.c | 4 +-
fs/proc/page.c | 4 +-
include/linux/balloon_compaction.h | 90 ++++-----
include/linux/fs.h | 2 +-
include/linux/migrate.h | 42 +----
include/linux/mm.h | 4 +-
include/linux/mm_types.h | 1 -
include/linux/page-flags.h | 104 +++++++----
include/linux/pagemap.h | 2 +-
include/linux/zsmalloc.h | 2 +
mm/balloon_compaction.c | 21 ++-
mm/compaction.c | 44 +----
mm/gup.c | 4 +-
mm/internal.h | 2 +-
mm/ksm.c | 4 +-
mm/memory-failure.c | 4 +-
mm/memory_hotplug.c | 8 +-
mm/migrate.c | 269 +++++++++++++++------------
mm/page_alloc.c | 12 +-
mm/page_isolation.c | 12 +-
mm/rmap.c | 16 +-
mm/util.c | 6 +-
mm/vmscan.c | 6 +-
mm/zpdesc.h | 15 +-
mm/zsmalloc.c | 29 ++-
28 files changed, 361 insertions(+), 390 deletions(-)
base-commit: 21f39cc0346ea2081a83f0469995144d75d23075
--
2.49.0
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:26
Let's move the removal of the page from the balloon list into the single
caller, to remove the dependency on the PG_isolated flag and clarify
locking requirements.
We'll shuffle the operations a bit such that they logically make more sense
(e.g., remove from the list before clearing flags).
In balloon migration functions we can now move the balloon_page_finalize()
out of the balloon lock and perform the finalization just before dropping
the balloon reference.
Document that the page lock is currently required when modifying the
movability aspects of a page; hopefully we can soon decouple this from the
page lock.
Signed-off-by: David Hildenbrand <redacted>
---
arch/powerpc/platforms/pseries/cmm.c | 2 +-
drivers/misc/vmw_balloon.c | 3 +-
drivers/virtio/virtio_balloon.c | 4 +--
include/linux/balloon_compaction.h | 43 +++++++++++-----------------
mm/balloon_compaction.c | 3 +-
5 files changed, 21 insertions(+), 34 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:29
The core will set PG_isolated only after mops->isolate_page() was
called. In case of the balloon, that is where we will remove it from
the balloon list. So we cannot have isolated pages in the balloon list.
Let's drop this unnecessary check.
Signed-off-by: David Hildenbrand <redacted>
---
mm/balloon_compaction.c | 6 ------
1 file changed, 6 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:29
Let's drop these checks; these are conditions the core migration code
must make sure will hold either way, no need to double check.
Signed-off-by: David Hildenbrand <redacted>
---
mm/zpdesc.h | 5 -----
mm/zsmalloc.c | 5 -----
2 files changed, 10 deletions(-)
@@ -1739,8 +1737,6 @@ static int zs_page_migrate(struct page *newpage, struct page *page,unsignedlongold_obj,new_obj;unsignedintobj_idx;-VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc),zpdesc_page(zpdesc));-/* The page is locked, so this pointer must remain valid */zspage=get_zspage(zpdesc);pool=zspage->pool;
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:31
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
@@ -140,7 +140,7 @@ static inline void balloon_page_finalize(struct page *page)__ClearPageMovable(page);set_page_private(page,0);}-__ClearPageOffline(page);+/* PageOffline is sticky until the page is freed to the buddy. */}/*
@@ -880,7 +880,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc)ClearPagePrivate(page);zpdesc->zspage=NULL;zpdesc->next=NULL;-__ClearPageZsmalloc(page);+/* PageZsmalloc is sticky until the page is freed to the buddy. */}staticinttrylock_zspage(structzspage*zspage)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:39
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 4 ++--
mm/compaction.c | 2 +-
mm/migrate.c | 39 +++++++++++++++++++++++++++++----------
3 files changed, 32 insertions(+), 13 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:46
Let's factor it out, simplifying the calling code.
The assumption is that flush_dcache_page() is not required for
movable_ops pages: as documented for flush_dcache_folio(), it really
only applies when the kernel wrote to pagecache pages / pages in
highmem. movable_ops callbacks should be handling flushing
caches if ever required.
Note that we can now change folio_mapping_flags() to folio_test_anon()
to make it clearer, because movable_ops pages will never take that path.
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 82 ++++++++++++++++++++++++++++------------------------
1 file changed, 45 insertions(+), 37 deletions(-)
@@ -159,6 +159,45 @@ static void putback_movable_ops_page(struct page *page)folio_put(folio);}+/**+*migrate_movable_ops_page-migrateanisolatedmovable_opspage+*@page:Theisolatedpage.+*+*Migrateanisolatedmovable_opspage.+*+*Ifthesrcpagewasalreadyreleasedbyitsowner,thesrcpageis+*un-isolated(putback)andmigrationsucceeds;themigrationcorewillbethe+*ownerofbothpages.+*+*Ifthesrcpagewasnotreleasedbyitsownerandthemigrationwas+*successful,theownerofthesrcpageandthedstpageareswappedand+*thesrcpageisun-isolated.+*+*Ifmigrationfails,theownershipstaysunmodifiedandthesrcpage+*remainsisolated:migrationmayberetriedlaterorthepagecanbeputback.+*+*TODO:migrationcorewilltreatbothpagesasfoliosandlockthembefore+*thiscalltounlockthemafterthiscall.Further,thefoliorefcountson+*srcanddstarealsoreleasedbymigrationcore.Thesepageswillnotbe+*foliosinthefuture,sothatmustbereworked.+*+*ReturnsMIGRATEPAGE_SUCCESSonsuccess,otherwiseanegativeerror+*code.+*/+staticintmigrate_movable_ops_page(structpage*dst,structpage*src,+enummigrate_modemode)+{+intrc=MIGRATEPAGE_SUCCESS;++VM_WARN_ON_ONCE_PAGE(!PageIsolated(src),src);+/* If the page was released by it's owner, there is nothing to do. */+if(PageMovable(src))+rc=page_movable_ops(src)->migrate_page(dst,src,mode);+if(rc==MIGRATEPAGE_SUCCESS)+ClearPageIsolated(src);+returnrc;+}+/**Putpreviouslyisolatedpagesbackontotheappropriatelists*fromwheretheywereoncetakenoffforcompaction/migration.
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:47
Folios will have nothing to do with movable_ops page migration. These
functions are now unused, so let's remove them.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 14 --------------
1 file changed, 14 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:49
... and factor the complete handling of movable_ops pages out.
Convert it similar to isolate_movable_ops_page().
While at it, convert the VM_BUG_ON_FOLIO() into a VM_WARN_ON_PAGE().
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 37 ++++++++++++++++++++++++-------------
1 file changed, 24 insertions(+), 13 deletions(-)
@@ -133,12 +133,30 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)returnfalse;}-staticvoidputback_movable_folio(structfolio*folio)+/**+*putback_movable_ops_page-putbackanisolatedmovable_opspage+*@page:Theisolatedpage.+*+*Putbackanisolatedmovable_opspage.+*+*Afterthepagewasputback,itmightgetfreedinstantly.+*/+staticvoidputback_movable_ops_page(structpage*page){-conststructmovable_operations*mops=folio_movable_ops(folio);--mops->putback_page(&folio->page);-folio_clear_isolated(folio);+/*+*TODO:thesepageswillnotbefoliosinthefuture.All+*foliodependencieswillhavetoberemoved.+*/+structfolio*folio=page_folio(page);++VM_WARN_ON_ONCE_PAGE(!PageIsolated(page),page);+folio_lock(folio);+/* If the page was released by it's owner, there is nothing to do. */+if(PageMovable(page))+page_movable_ops(page)->putback_page(page);+ClearPageIsolated(page);+folio_unlock(folio);+folio_put(folio);}/*
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:50
Let's move that handling directly into migrate_folio_move(), so we can
simplify move_to_new_folio(). While at it, fixup the documentation a
bit.
Note that unmap_and_move_huge_page() does not care, because it only
deals with actual folios. (we only support migration of
individual movable_ops pages)
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 61 ++++++++++++++++++++++++----------------------------
1 file changed, 28 insertions(+), 33 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:53
Instead, let's check in the callbacks if the page was already destroyed,
which can be checked by looking at zpdesc->zspage (see reset_zpdesc()).
If we detect that the page was destroyed:
(1) Fail isolation, just like the migration core would
(2) Fake migration success just like the migration core would
In the putback case there is nothing to do, as we don't do anything just
like the migration core would do.
In the future, we should look into not letting these pages get destroyed
while they are isolated -- and instead delaying that to the
putback/migration call. Add a TODO for that.
Signed-off-by: David Hildenbrand <redacted>
---
mm/zsmalloc.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
@@ -1736,6 +1736,13 @@ static int zs_page_migrate(struct page *newpage, struct page *page,unsignedlongold_obj,new_obj;unsignedintobj_idx;+/*+*TODO:nothingpreventsazspagefromgettingdestroyedwhile+*isolated:weshoulddisallowthatanddeferit.+*/+if(!zpdesc->zspage)+returnMIGRATEPAGE_SUCCESS;+/* The page is locked, so this pointer must remain valid */zspage=get_zspage(zpdesc);pool=zspage->pool;
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:55
We can just look at the balloon device (stored in page->private), to see
of the page is still part of the balloon.
As isolated balloon pages cannot get released (they are taken off the
balloon list while isolated), we don't have to worry about this case in
the putback and migration callback. Add a WARN_ON_ONCE for now.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 4 +---
mm/balloon_compaction.c | 11 +++++++++++
2 files changed, 12 insertions(+), 3 deletions(-)
@@ -136,10 +136,8 @@ static inline gfp_t balloon_mapping_gfp_mask(void)*/staticinlinevoidballoon_page_finalize(structpage*page){-if(IS_ENABLED(CONFIG_BALLOON_COMPACTION)){-__ClearPageMovable(page);+if(IS_ENABLED(CONFIG_BALLOON_COMPACTION))set_page_private(page,0);-}/* PageOffline is sticky until the page is freed to the buddy. */}
From: David Hildenbrand <hidden> Date: 2025-06-18 17:40:58
Unused, let's remove it.
The Chinese docs in Documentation/translations/zh_CN/mm/page_migration.rst
still mention it, but that whole docs is destined to get outdated and
updated by somebody that actually speaks that language.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 4 ----
mm/compaction.c | 11 -----------
2 files changed, 15 deletions(-)
@@ -137,17 +137,6 @@ void __SetPageMovable(struct page *page, const struct movable_operations *mops)}EXPORT_SYMBOL(__SetPageMovable);-void__ClearPageMovable(structpage*page)-{-VM_BUG_ON_PAGE(!PageMovable(page),page);-/*-*Thispagestillhasthetypeofamovablepage,butit's-*actuallynotmovableanymore.-*/-page->mapping=(void*)PAGE_MAPPING_MOVABLE;-}-EXPORT_SYMBOL(__ClearPageMovable);-/* Do not skip compaction more than 64 times */#define COMPACT_MAX_DEFER_SHIFT 6
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:04
As __ClearPageMovable() is gone that would have only made
PageMovable()==false but still __PageMovable()==true, now
PageMovable() == __PageMovable().
So we can replace PageMovable() checks by __PageMovable(). In fact,
__PageMovable() cannot change until a page is freed, so we can turn
some PageMovable() into sanity checks for __PageMovable().
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 2 --
mm/compaction.c | 15 ---------------
mm/migrate.c | 18 ++++++++++--------
3 files changed, 10 insertions(+), 25 deletions(-)
@@ -149,11 +153,10 @@ static void putback_movable_ops_page(struct page *page)*/structfolio*folio=page_folio(page);+VM_WARN_ON_ONCE_PAGE(!__PageMovable(page),page);VM_WARN_ON_ONCE_PAGE(!PageIsolated(page),page);folio_lock(folio);-/* If the page was released by it's owner, there is nothing to do. */-if(PageMovable(page))-page_movable_ops(page)->putback_page(page);+page_movable_ops(page)->putback_page(page);ClearPageIsolated(page);folio_unlock(folio);folio_put(folio);
@@ -189,10 +192,9 @@ static int migrate_movable_ops_page(struct page *dst, struct page *src,{intrc=MIGRATEPAGE_SUCCESS;+VM_WARN_ON_ONCE_PAGE(!__PageMovable(src),src);VM_WARN_ON_ONCE_PAGE(!PageIsolated(src),src);-/* If the page was released by it's owner, there is nothing to do. */-if(PageMovable(src))-rc=page_movable_ops(src)->migrate_page(dst,src,mode);+rc=page_movable_ops(src)->migrate_page(dst,src,mode);if(rc==MIGRATEPAGE_SUCCESS)ClearPageIsolated(src);returnrc;
@@ -1759,9 +1759,7 @@ static int scan_movable_pages(unsigned long start, unsigned long end,structfolio*folio;page=pfn_to_page(pfn);-if(PageLRU(page))-gotofound;-if(__PageMovable(page))+if(PageLRU(page)||page_has_movable_ops(page))gotofound;/*
@@ -133,7 +133,7 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long eif((mode==PB_ISOLATE_MODE_MEM_OFFLINE)&&PageOffline(page))continue;-if(__PageMovable(page)||PageLRU(page))+if(PageLRU(page)||page_has_movable_ops(page))continue;/*
@@ -421,7 +421,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn,*properfreeandsplithandlingforthem.*/VM_WARN_ON_ONCE_PAGE(PageLRU(page),page);-VM_WARN_ON_ONCE_PAGE(__PageMovable(page),page);+VM_WARN_ON_ONCE_PAGE(page_has_movable_ops(page),page);gotofailed;}
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:08
Currently, we only support migration of individual non-folio pages, so
we can not run into that.
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_isolation.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:08
Convert to page_has_movable_ops(). While at it, cleanup relevant code
a bit.
The data_race() in migrate_folio_unmap() is questionable: we already
hold a page reference, and concurrent modifications can no longer
happen (iow: __ClearPageMovable() no longer exists). Drop it for now,
we'll rework page_has_movable_ops() soon either way to no longer
rely on page->mapping.
Wherever we cast from folio to page now is a clear sign that this
code has to be decoupled.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 6 ------
mm/migrate.c | 43 ++++++++++++--------------------------
mm/vmscan.c | 6 ++++--
3 files changed, 17 insertions(+), 38 deletions(-)
@@ -237,26 +232,20 @@ void putback_movable_pages(struct list_head *l)/* Must be called with an elevated refcount on the non-hugetlb folio */boolisolate_folio_to_list(structfolio*folio,structlist_head*list){-boolisolated,lru;-if(folio_test_hugetlb(folio))returnfolio_isolate_hugetlb(folio,list);-lru=!__folio_test_movable(folio);-if(lru)-isolated=folio_isolate_lru(folio);-else-isolated=isolate_movable_ops_page(&folio->page,-ISOLATE_UNEVICTABLE);--if(!isolated)-returnfalse;--list_add(&folio->lru,list);-if(lru)+if(page_has_movable_ops(&folio->page)){+if(!isolate_movable_ops_page(&folio->page,+ISOLATE_UNEVICTABLE))+returnfalse;+}else{+if(!folio_isolate_lru(folio))+returnfalse;node_stat_add_folio(folio,NR_ISOLATED_ANON+folio_is_file_lru(folio));-+}+list_add(&folio->lru,list);returntrue;}
@@ -1651,9 +1651,11 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,unsignedintnoreclaim_flag;list_for_each_entry_safe(folio,next,folio_list,lru){+/* TODO: these pages should not even appear in this list. */+if(page_has_movable_ops(&folio->page))+continue;if(!folio_test_hugetlb(folio)&&folio_is_file_lru(folio)&&-!folio_test_dirty(folio)&&!__folio_test_movable(folio)&&-!folio_test_unevictable(folio)){+!folio_test_dirty(folio)&&!folio_test_unevictable(folio)){folio_clear_active(folio);list_move(&folio->lru,&clean_folios);}
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:11
... instead, look them up statically based on the page type. Maybe in the
future we want a registration interface? At least for now, it can be
easily handled using the two page types that actually support page
migration.
The remaining usage of page->mapping is to flag such pages as actually
being movable (having movable_ops), which we will change next.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 2 +-
include/linux/migrate.h | 14 ++------------
include/linux/zsmalloc.h | 2 ++
mm/balloon_compaction.c | 1 -
mm/compaction.c | 5 ++---
mm/migrate.c | 23 +++++++++++++++++++++++
mm/zpdesc.h | 5 ++---
mm/zsmalloc.c | 8 +++-----
8 files changed, 35 insertions(+), 25 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:19
Let's rename the flag to make it clearer where it applies (not folios
...).
While at it, define the flag only with CONFIG_MIGRATION.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 16 +++++++++++-----
mm/compaction.c | 2 +-
mm/migrate.c | 14 +++++++-------
3 files changed, 19 insertions(+), 13 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:19
Instead, let's use a page flag. As the page flag can result in
false-positives, glue it to the page types for which we
support/implement movable_ops page migration.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 2 +-
include/linux/migrate.h | 8 -----
include/linux/page-flags.h | 52 ++++++++++++++++++++++++------
mm/compaction.c | 6 ----
mm/zpdesc.h | 2 +-
5 files changed, 44 insertions(+), 26 deletions(-)
@@ -170,6 +170,11 @@ enum pageflags {/* non-lru isolated movable page */PG_isolated=PG_reclaim,+#ifdef CONFIG_MIGRATION+/* this is a movable_ops page (for selected typed pages only) */+PG_movable_ops=PG_uptodate,+#endif+/* Only valid for buddy pages. Used to track pages that are reported */PG_reported=PG_uptodate,
@@ -114,12 +114,6 @@ static unsigned long release_free_list(struct list_head *freepages)}#ifdef CONFIG_COMPACTION-void__SetPageMovable(structpage*page)-{-VM_BUG_ON_PAGE(!PageLocked(page),page);-page->mapping=(void*)(PAGE_MAPPING_MOVABLE);-}-EXPORT_SYMBOL(__SetPageMovable);/* Do not skip compaction more than 64 times */#define COMPACT_MAX_DEFER_SHIFT 6
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:20
KSM is the only remaining user, let's rename the flag. While at it,
adjust to remaining page -> folio in the doc.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:25
It's unused and the page counterpart is gone, so let's remove it.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 5 -----
1 file changed, 5 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:31
Now that PAGE_MAPPING_MOVABLE is gone, we can simplify and rely on the
folio_test_anon() test only.
... but staring at the users, this function should never even have been
called on movable_ops pages. E.g.,
* __buffer_migrate_folio() does not make sense for them
* folio_migrate_mapping() does not make sense for them
* migrate_huge_page_move_mapping() does not make sense for them
* __migrate_folio() does not make sense for them
* ... and khugepaged should never stumble over them
Let's simply refuse typed pages (which includes slab) except hugetlb,
and WARN.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/mm.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
@@ -2167,13 +2167,13 @@ static inline int folio_expected_ref_count(const struct folio *folio)constintorder=folio_order(folio);intref_count=0;-if(WARN_ON_ONCE(folio_test_slab(folio)))+if(WARN_ON_ONCE(page_has_type(&folio->page)&&!folio_test_hugetlb(folio)))return0;if(folio_test_anon(folio)){/* One reference per page from the swapcache. */ref_count+=folio_test_swapcache(folio)<<order;-}elseif(!((unsignedlong)folio->mapping&PAGE_MAPPING_FLAGS)){+}else{/* One reference per page from the pagecache. */ref_count+=!!folio->mapping<<order;/* One reference from PG_private. */
@@ -524,7 +524,7 @@ struct address_space {/**Onmostarchitecturesthatalignmentisalreadythecase;but*mustbeenforcedhereforCRIS,tolettheleastsignificantbit-*ofstructpage's"mapping"pointerbeusedforPAGE_MAPPING_ANON.+*ofstructfolio's"mapping"pointerbeusedforFOLIO_MAPPING_ANON.*//* XArray tags, for tagging dirty and writeback pages in the pagecache. */
@@ -146,18 +146,33 @@ Steps:18. The new page is moved to the LRU and can be scanned by the swapper, etc. again.-Non-LRU page migration-======================--Although migration originally aimed for reducing the latency of memory-accesses for NUMA, compaction also uses migration to create high-order-pages. For compaction purposes, it is also useful to be able to move-non-LRU pages, such as zsmalloc and virtio-balloon pages.--If a driver wants to make its pages movable, it should define a struct-movable_operations. It then needs to call __SetPageMovable() on each-page that it may be able to move. This uses the ``page->mapping`` field,-so this field is not available for the driver to use for other purposes.+movable_ops page migration+==========================++Selected typed, non-folio pages (e.g., pages inflated in a memory balloon,+zsmalloc pages) can be migrated using the movable_ops migration framework.++The "struct movable_operations" provide callbacks specific to a page type+for isolating, migrating and un-isolating (putback) these pages.++Once a page is indicated as having movable_ops, that condition must not+change until the page was freed back to the buddy. This includes not+changing/clearing the page type and not changing/clearing the+PG_movable_ops page flag.++Arbitrary drivers cannot currently make use of this framework, as it+requires:++(a) a page type+(b) indicating them as possibly having movable_ops in page_has_movable_ops()+ based on the page type+(c) returning the movable_ops from page_has_movable_ops() based on the page+ type+(d) not reusing the PG_movable_ops and PG_movable_ops_isolated page flags+ for other purposes++For example, balloon drivers can make use of this framework through the+balloon-compaction framework residing in the core kernel. Monitoring Migration =====================
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:36
Let's bring the docs up-to-date. Setting PG_movable_ops + page->private
very likely still requires to be performed under documented locks:
it's complicated.
We will rework this in the future, as we will try avoiding using the
page lock.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:41:40
Let's just special-case based on IS_ENABLED(CONFIG_BALLOON_COMPACTION
like we did for balloon_page_finalize().
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 42 +++++++++++-------------------
1 file changed, 15 insertions(+), 27 deletions(-)
From: David Hildenbrand <hidden> Date: 2025-06-18 17:42:49
We can now simply check for PageAnon() and remove PageMappingFlags().
... and while at it, use the folio instead and operate on
folio->mapping.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 5 -----
mm/page_alloc.c | 7 +++----
2 files changed, 3 insertions(+), 9 deletions(-)
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:04:26
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted hunk
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
From: Matthew Wilcox <willy@infradead.org> Date: 2025-06-18 18:07:11
On Wed, Jun 18, 2025 at 02:04:18PM -0400, Zi Yan wrote:
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
quoted
+ if (unlikely(page_has_type(page)))
+ page->page_type = UINT_MAX;
+
if (is_check_pages_enabled()) {
if (free_page_is_bad(page))
bad++;
--
2.49.0
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing
page_type?
The point is that the _caller_ used to have to clear the page type.
This patch allows the caller to free the page without clearing
the page type first.
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:08:15
On 18 Jun 2025, at 14:04, Zi Yan wrote:
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing
page_type?
OK, next patch explains it. free_pages_prepare() clears page_type,
so that caller does not need to.
I think the message is better to be
mm/page_alloc: clear page_type at page free time
page_type is no longer needed to be cleared before a page is freed, as
page free code does that.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:09:15
On 18 Jun 2025, at 14:06, Matthew Wilcox wrote:
On Wed, Jun 18, 2025 at 02:04:18PM -0400, Zi Yan wrote:
quoted
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
quoted
quoted
+ if (unlikely(page_has_type(page)))
+ page->page_type = UINT_MAX;
+
if (is_check_pages_enabled()) {
if (free_page_is_bad(page))
bad++;
--
2.49.0
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing
page_type?
The point is that the _caller_ used to have to clear the page type.
This patch allows the caller to free the page without clearing
the page type first.
Yep, find that out when I read the next patch.
I think the change is fine, but the commit message is unclear.
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:14:22
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for
folios? Is there any fundamental reason preventing movable_ops from
being used on folios?
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:32:27
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
The core will set PG_isolated only after mops->isolate_page() was
called. In case of the balloon, that is where we will remove it from
the balloon list. So we cannot have isolated pages in the balloon list.
Let's drop this unnecessary check.
Signed-off-by: David Hildenbrand <redacted>
---
mm/balloon_compaction.c | 6 ------
1 file changed, 6 deletions(-)
Acked-by: Zi Yan <ziy@nvidia.com>
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:38:07
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
Let's drop these checks; these are conditions the core migration code
must make sure will hold either way, no need to double check.
Signed-off-by: David Hildenbrand <redacted>
---
mm/zpdesc.h | 5 -----
mm/zsmalloc.c | 5 -----
2 files changed, 10 deletions(-)
Acked-by: Zi Yan <ziy@nvidia.com>
Best Regards,
Yan, Zi
From: Matthew Wilcox <willy@infradead.org> Date: 2025-06-18 18:39:24
On Wed, Jun 18, 2025 at 02:14:15PM -0400, Zi Yan wrote:
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for
folios? Is there any fundamental reason preventing movable_ops from
being used on folios?
folios either belong to a filesystem or they are anonymous memory, and
so either the filesystem knows how to migrate them (through its a_ops)
or the migration code knows how to handle anon folios directly.
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:43:36
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted hunk
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
Should we be pedantic to only do this for PageOffline and PageZsmalloc
and warn for the rest page types?
Something like:
if (unlikely(page_has_type(page))) {
if (PageOffline(page) || PageZsmalloc(page))
page->page_type = UINT_MAX;
else
VM_WARN_ONCE_PAGE(1, page);
}
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 18:48:27
On 18 Jun 2025, at 14:39, Matthew Wilcox wrote:
On Wed, Jun 18, 2025 at 02:14:15PM -0400, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for
folios? Is there any fundamental reason preventing movable_ops from
being used on folios?
folios either belong to a filesystem or they are anonymous memory, and
so either the filesystem knows how to migrate them (through its a_ops)
or the migration code knows how to handle anon folios directly.
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is
only for anon and fs.
Best Regards,
Yan, Zi
@@ -140,7 +140,7 @@ static inline void balloon_page_finalize(struct page *page)__ClearPageMovable(page);set_page_private(page,0);}-__ClearPageOffline(page);+/* PageOffline is sticky until the page is freed to the buddy. */}/*
--
2.49.0
Acked-by: Zi Yan <ziy@nvidia.com>
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 19:10:19
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted hunk
... and factor the complete handling of movable_ops pages out.
Convert it similar to isolate_movable_ops_page().
While at it, convert the VM_BUG_ON_FOLIO() into a VM_WARN_ON_PAGE().
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 37 ++++++++++++++++++++++++-------------
1 file changed, 24 insertions(+), 13 deletions(-)
@@ -133,12 +133,30 @@ bool isolate_movable_ops_page(struct page *page, isolate_mode_t mode)returnfalse;}-staticvoidputback_movable_folio(structfolio*folio)+/**+*putback_movable_ops_page-putbackanisolatedmovable_opspage+*@page:Theisolatedpage.+*+*Putbackanisolatedmovable_opspage.+*+*Afterthepagewasputback,itmightgetfreedinstantly.+*/+staticvoidputback_movable_ops_page(structpage*page){-conststructmovable_operations*mops=folio_movable_ops(folio);--mops->putback_page(&folio->page);-folio_clear_isolated(folio);+/*+*TODO:thesepageswillnotbefoliosinthefuture.All+*foliodependencieswillhavetoberemoved.+*/+structfolio*folio=page_folio(page);++VM_WARN_ON_ONCE_PAGE(!PageIsolated(page),page);+folio_lock(folio);+/* If the page was released by it's owner, there is nothing to do. */+if(PageMovable(page))+page_movable_ops(page)->putback_page(page);+ClearPageIsolated(page);+folio_unlock(folio);+folio_put(folio);
Why not use page version of lock, unlock, and put? Especially you are
thinking about not using folio for these pages. Just a question,
I am OK with current patch.
From: Matthew Wilcox <willy@infradead.org> Date: 2025-06-18 19:18:16
On Wed, Jun 18, 2025 at 03:10:10PM -0400, Zi Yan wrote:
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
+ /*
+ * TODO: these pages will not be folios in the future. All
+ * folio dependencies will have to be removed.
+ */
+ struct folio *folio = page_folio(page);
+
+ VM_WARN_ON_ONCE_PAGE(!PageIsolated(page), page);
+ folio_lock(folio);
+ /* If the page was released by it's owner, there is nothing to do. */
+ if (PageMovable(page))
+ page_movable_ops(page)->putback_page(page);
+ ClearPageIsolated(page);
+ folio_unlock(folio);
+ folio_put(folio);
Why not use page version of lock, unlock, and put? Especially you are
thinking about not using folio for these pages. Just a question,
I am OK with current patch.
That would reintroduce unnecessary calls to compound_head().
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-18 19:26:03
On 18 Jun 2025, at 15:18, Matthew Wilcox wrote:
On Wed, Jun 18, 2025 at 03:10:10PM -0400, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
+ /*
+ * TODO: these pages will not be folios in the future. All
+ * folio dependencies will have to be removed.
+ */
+ struct folio *folio = page_folio(page);
+
+ VM_WARN_ON_ONCE_PAGE(!PageIsolated(page), page);
+ folio_lock(folio);
+ /* If the page was released by it's owner, there is nothing to do. */
+ if (PageMovable(page))
+ page_movable_ops(page)->putback_page(page);
+ ClearPageIsolated(page);
+ folio_unlock(folio);
+ folio_put(folio);
Why not use page version of lock, unlock, and put? Especially you are
thinking about not using folio for these pages. Just a question,
I am OK with current patch.
That would reintroduce unnecessary calls to compound_head().
Got it. But here page is not folio, so it cannot be a compound page.
Then, we will need page versions without compound_head() for
non compound pages. Could that happen in the future when only folio
can be compound and page is only order-0?
Best Regards,
Yan, Zi
From: Matthew Wilcox <willy@infradead.org> Date: 2025-06-18 20:04:12
On Wed, Jun 18, 2025 at 03:25:46PM -0400, Zi Yan wrote:
On 18 Jun 2025, at 15:18, Matthew Wilcox wrote:
quoted
quoted
Why not use page version of lock, unlock, and put? Especially you are
thinking about not using folio for these pages. Just a question,
I am OK with current patch.
That would reintroduce unnecessary calls to compound_head().
Got it. But here page is not folio, so it cannot be a compound page.
Then, we will need page versions without compound_head() for
non compound pages. Could that happen in the future when only folio
can be compound and page is only order-0?
I think the assumption that we'll only see compound pages as part of
folios is untrue. For example, slabs will still allocate multiple
pages (though slabs aren't migratable at this point). The sketch at
https://kernelnewbies.org/MatthewWilcox/Memdescs supports "misc pages"
with an order stored in bits 12-17 of the memdesc. I don't know
how useful that will turn out to be; maybe we'll never implement that.
Let's drop these checks; these are conditions the core migration code
must make sure will hold either way, no need to double check.
Signed-off-by: David Hildenbrand <redacted>
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 19:30:21
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
Let's factor it out, simplifying the calling code.
The assumption is that flush_dcache_page() is not required for
movable_ops pages: as documented for flush_dcache_folio(), it really
only applies when the kernel wrote to pagecache pages / pages in
highmem. movable_ops callbacks should be handling flushing
caches if ever required.
Note that we can now change folio_mapping_flags() to folio_test_anon()
to make it clearer, because movable_ops pages will never take that path.
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 82 ++++++++++++++++++++++++++++------------------------
1 file changed, 45 insertions(+), 37 deletions(-)
LGTM. Reviewed-by: Zi Yan [off-list ref]
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 19:37:02
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
Folios will have nothing to do with movable_ops page migration. These
functions are now unused, so let's remove them.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 14 --------------
1 file changed, 14 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:06:53
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
Let's move that handling directly into migrate_folio_move(), so we can
simplify move_to_new_folio(). While at it, fixup the documentation a
bit.
Note that unmap_and_move_huge_page() does not care, because it only
deals with actual folios. (we only support migration of
individual movable_ops pages)
Signed-off-by: David Hildenbrand <redacted>
---
mm/migrate.c | 61 ++++++++++++++++++++++++----------------------------
1 file changed, 28 insertions(+), 33 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:15:32
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
Unused, let's remove it.
The Chinese docs in Documentation/translations/zh_CN/mm/page_migration.rst
still mention it, but that whole docs is destined to get outdated and
updated by somebody that actually speaks that language.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 4 ----
mm/compaction.c | 11 -----------
2 files changed, 15 deletions(-)
The comment for struct movable_operations needs an update too.
"
If page migration is successful, the driver should call
__ClearPageMovable(@src) and return MIGRATEPAGE_SUCCESS.
"
Otherwise, Reviewed-by: Zi Yan [off-list ref]
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:19:20
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
As __ClearPageMovable() is gone that would have only made
PageMovable()==false but still __PageMovable()==true, now
PageMovable() == __PageMovable().
So we can replace PageMovable() checks by __PageMovable(). In fact,
__PageMovable() cannot change until a page is freed, so we can turn
some PageMovable() into sanity checks for __PageMovable().
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 2 --
mm/compaction.c | 15 ---------------
mm/migrate.c | 18 ++++++++++--------
3 files changed, 10 insertions(+), 25 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*Skipanyothertypeofpage*/if(!PageLRU(page)){-/*-*__PageMovablecanreturnfalsepositivesoweneed-*toverifyitunderpage_lock.-*/-if(unlikely(__PageMovable(page))&&+/* Isolation will grab the page lock. */
I feel that the removed comment should stay, since the current comment
makes no sense when I read it alone.
In addition, why is __PageMovable() is renamed to page_has_movable_ops() but
__SetPageMovable() stays the same? page_has_movable_ops() and __SetPageMovable()
are functions for checking and setting PAGE_MAPPING_MOVABLE. The naming just
does not look symmetric.
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:38:26
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
Currently, we only support migration of individual non-folio pages, so
we can not run into that.
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_isolation.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:41:46
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
Convert to page_has_movable_ops(). While at it, cleanup relevant code
a bit.
The data_race() in migrate_folio_unmap() is questionable: we already
hold a page reference, and concurrent modifications can no longer
happen (iow: __ClearPageMovable() no longer exists). Drop it for now,
we'll rework page_has_movable_ops() soon either way to no longer
rely on page->mapping.
Wherever we cast from folio to page now is a clear sign that this
code has to be decoupled.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 6 ------
mm/migrate.c | 43 ++++++++++++--------------------------
mm/vmscan.c | 6 ++++--
3 files changed, 17 insertions(+), 38 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-20 20:45:52
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
... instead, look them up statically based on the page type. Maybe in the
future we want a registration interface? At least for now, it can be
easily handled using the two page types that actually support page
migration.
The remaining usage of page->mapping is to flag such pages as actually
being movable (having movable_ops), which we will change next.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 2 +-
include/linux/migrate.h | 14 ++------------
include/linux/zsmalloc.h | 2 ++
mm/balloon_compaction.c | 1 -
mm/compaction.c | 5 ++---
mm/migrate.c | 23 +++++++++++++++++++++++
mm/zpdesc.h | 5 ++---
mm/zsmalloc.c | 8 +++-----
8 files changed, 35 insertions(+), 25 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:14:23
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
Instead, let's use a page flag. As the page flag can result in
false-positives, glue it to the page types for which we
support/implement movable_ops page migration.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/balloon_compaction.h | 2 +-
include/linux/migrate.h | 8 -----
include/linux/page-flags.h | 52 ++++++++++++++++++++++++------
mm/compaction.c | 6 ----
mm/zpdesc.h | 2 +-
5 files changed, 44 insertions(+), 26 deletions(-)
<snip>
+
+/**
+ * page_has_movable_ops - test for a movable_ops page
+ * @page The page to test.
+ *
+ * Test whether this is a movable_ops page. Such pages will stay that
+ * way until freed.
+ *
+ * Returns true if this is a movable_ops page, otherwise false.
+ */
+static inline bool page_has_movable_ops(const struct page *page)
+{
+ return PageMovableOps(page) &&
+ (PageOffline(page) || PageZsmalloc(page));
+}
+
Should we do the code below in case PageMovableOps is set on pages
other than PageOffline and PageZsmalloc?
return PageMovableOps(page) &&
!VM_WARN_ON_ONCE_PAGE(!(PageOffline(page) || PageZsmalloc(page)),
page);
quoted hunk
static __always_inline int PageAnonExclusive(const struct page *page)
{
VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
@@ -114,12 +114,6 @@ static unsigned long release_free_list(struct list_head *freepages)}#ifdef CONFIG_COMPACTION-void__SetPageMovable(structpage*page)-{-VM_BUG_ON_PAGE(!PageLocked(page),page);-page->mapping=(void*)(PAGE_MAPPING_MOVABLE);-}-EXPORT_SYMBOL(__SetPageMovable);/* Do not skip compaction more than 64 times */#define COMPACT_MAX_DEFER_SHIFT 6
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:16:19
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
Let's rename the flag to make it clearer where it applies (not folios
...).
While at it, define the flag only with CONFIG_MIGRATION.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 16 +++++++++++-----
mm/compaction.c | 2 +-
mm/migrate.c | 14 +++++++-------
3 files changed, 19 insertions(+), 13 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:17:13
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
KSM is the only remaining user, let's rename the flag. While at it,
adjust to remaining page -> folio in the doc.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:20:22
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
We can now simply check for PageAnon() and remove PageMappingFlags().
... and while at it, use the folio instead and operate on
folio->mapping.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 5 -----
mm/page_alloc.c | 7 +++----
2 files changed, 3 insertions(+), 9 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:20:46
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
It's unused and the page counterpart is gone, so let's remove it.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/page-flags.h | 5 -----
1 file changed, 5 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 14:23:58
On 18 Jun 2025, at 13:40, David Hildenbrand wrote:
Now that PAGE_MAPPING_MOVABLE is gone, we can simplify and rely on the
folio_test_anon() test only.
... but staring at the users, this function should never even have been
called on movable_ops pages. E.g.,
* __buffer_migrate_folio() does not make sense for them
* folio_migrate_mapping() does not make sense for them
* migrate_huge_page_move_mapping() does not make sense for them
* __migrate_folio() does not make sense for them
* ... and khugepaged should never stumble over them
Let's simply refuse typed pages (which includes slab) except hugetlb,
and WARN.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/mm.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
@@ -146,18 +146,33 @@ Steps:18. The new page is moved to the LRU and can be scanned by the swapper, etc. again.-Non-LRU page migration-======================--Although migration originally aimed for reducing the latency of memory-accesses for NUMA, compaction also uses migration to create high-order-pages. For compaction purposes, it is also useful to be able to move-non-LRU pages, such as zsmalloc and virtio-balloon pages.--If a driver wants to make its pages movable, it should define a struct-movable_operations. It then needs to call __SetPageMovable() on each-page that it may be able to move. This uses the ``page->mapping`` field,-so this field is not available for the driver to use for other purposes.+movable_ops page migration+==========================++Selected typed, non-folio pages (e.g., pages inflated in a memory balloon,
This is a great clarification. Thanks.
+zsmalloc pages) can be migrated using the movable_ops migration framework.
+
+The "struct movable_operations" provide callbacks specific to a page type
+for isolating, migrating and un-isolating (putback) these pages.
+
+Once a page is indicated as having movable_ops, that condition must not
+change until the page was freed back to the buddy. This includes not
+changing/clearing the page type and not changing/clearing the
+PG_movable_ops page flag.
+
+Arbitrary drivers cannot currently make use of this framework, as it
+requires:
+
+(a) a page type
+(b) indicating them as possibly having movable_ops in page_has_movable_ops()
+ based on the page type
+(c) returning the movable_ops from page_has_movable_ops() based on the page
+ type
+(d) not reusing the PG_movable_ops and PG_movable_ops_isolated page flags
+ for other purposes
+
+For example, balloon drivers can make use of this framework through the
+balloon-compaction framework residing in the core kernel.
Monitoring Migration
=====================
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: David Hildenbrand <hidden> Date: 2025-06-23 15:26:13
On 18.06.25 20:08, Zi Yan wrote:
On 18 Jun 2025, at 14:04, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing
page_type?
OK, next patch explains it. free_pages_prepare() clears page_type,
so that caller does not need to.
I think the message is better to be
mm/page_alloc: clear page_type at page free time
page_type is no longer needed to be cleared before a page is freed, as
page free code does that.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
I'll change it to
mm/page_alloc: let page freeing clear any set page type
Currently, any user of page types must clear that type before freeing
a page back to the buddy, otherwise we'll run into mapcount related
sanity checks (because the page type currently overlays the page
mapcount).
Let's allow for not clearing the page type by page type users by letting
the buddy handle it instead.
We'll focus on having a page type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
--
Cheers,
David / dhildenb
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 15:28:49
On 23 Jun 2025, at 11:26, David Hildenbrand wrote:
On 18.06.25 20:08, Zi Yan wrote:
quoted
On 18 Jun 2025, at 14:04, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
How does this preserve page type? Isn’t page->page_type = UINT_MAX clearing
page_type?
OK, next patch explains it. free_pages_prepare() clears page_type,
so that caller does not need to.
I think the message is better to be
mm/page_alloc: clear page_type at page free time
page_type is no longer needed to be cleared before a page is freed, as
page free code does that.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
I'll change it to
mm/page_alloc: let page freeing clear any set page type
Currently, any user of page types must clear that type before freeing
a page back to the buddy, otherwise we'll run into mapcount related
sanity checks (because the page type currently overlays the page
mapcount).
Let's allow for not clearing the page type by page type users by letting
the buddy handle it instead.
We'll focus on having a page type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Thanks.
Reviewed-by: Zi Yan <ziy@nvidia.com>
--
Best Regards,
Yan, Zi
From: David Hildenbrand <hidden> Date: 2025-06-23 15:28:58
On 18.06.25 20:43, Zi Yan wrote:
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
Let's allow for not clearing a page type before freeing a page to the
buddy.
We'll focus on having a type set on the first page of a larger
allocation only.
With this change, we can reliably identify typed folios even though
they might be in the process of getting freed, which will come in handy
in migration code (at least in the transition phase).
Signed-off-by: David Hildenbrand <redacted>
---
mm/page_alloc.c | 3 +++
1 file changed, 3 insertions(+)
Should we be pedantic to only do this for PageOffline and PageZsmalloc
and warn for the rest page types?
I think we should just allow any page types. Limiting it to specific
types sounds like some use-after-free check that probably shouldn't be
handled that way.
--
Cheers,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2025-06-23 15:33:22
On 18.06.25 20:48, Zi Yan wrote:
On 18 Jun 2025, at 14:39, Matthew Wilcox wrote:
quoted
On Wed, Jun 18, 2025 at 02:14:15PM -0400, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for
folios? Is there any fundamental reason preventing movable_ops from
being used on folios?
folios either belong to a filesystem or they are anonymous memory, and
so either the filesystem knows how to migrate them (through its a_ops)
or the migration code knows how to handle anon folios directly.
Right, migration of folios will be handled by migration core.
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is
only for anon and fs.
I assume they are proper folios, so yes. Just like they are handled
today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we
sure these things will be folios even before they are assigned to a
filesystem? I recall the answer was "yes".
So we don't (and will not) support movable_ops for folios.
--
Cheers,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2025-06-23 15:38:00
On 18.06.25 22:04, Matthew Wilcox wrote:
On Wed, Jun 18, 2025 at 03:25:46PM -0400, Zi Yan wrote:
quoted
On 18 Jun 2025, at 15:18, Matthew Wilcox wrote:
quoted
quoted
Why not use page version of lock, unlock, and put? Especially you are
thinking about not using folio for these pages. Just a question,
I am OK with current patch.
That would reintroduce unnecessary calls to compound_head().
Right. And I want to make it clear that everything that uses "folio"
here must be reworked: not necessarily switching to the "page" variant,
but actually by implementing it entirely differently.
(e.g., store them in an array instead of a list, get rid of the lock
bit, try getting rid of the refcount as well)
quoted
Got it. But here page is not folio, so it cannot be a compound page.
Then, we will need page versions without compound_head() for
non compound pages. Could that happen in the future when only folio
can be compound and page is only order-0?
I think the assumption that we'll only see compound pages as part of
folios is untrue. For example, slabs will still allocate multiple
pages (though slabs aren't migratable at this point). The sketch at
https://kernelnewbies.org/MatthewWilcox/Memdescs supports "misc pages"
with an order stored in bits 12-17 of the memdesc. I don't know
how useful that will turn out to be; maybe we'll never implement that.
Once we have to support that for movable_ops, it will be an interesting
question how to handle that. Most certainly, these things will not be
folios. :)
--
Cheers,
David / dhildenb
From: Zi Yan <ziy@nvidia.com> Date: 2025-06-23 15:42:14
On 23 Jun 2025, at 11:33, David Hildenbrand wrote:
On 18.06.25 20:48, Zi Yan wrote:
quoted
On 18 Jun 2025, at 14:39, Matthew Wilcox wrote:
quoted
On Wed, Jun 18, 2025 at 02:14:15PM -0400, Zi Yan wrote:
quoted
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
... and start moving back to per-page things that will absolutely not be
folio things in the future. Add documentation and a comment that the
remaining folio stuff (lock, refcount) will have to be reworked as well.
While at it, convert the VM_BUG_ON() into a WARN_ON_ONCE() and handle
it gracefully (relevant with further changes), and convert a
WARN_ON_ONCE() into a VM_WARN_ON_ONCE_PAGE().
The reason is that there is no upstream code, which use movable_ops for
folios? Is there any fundamental reason preventing movable_ops from
being used on folios?
folios either belong to a filesystem or they are anonymous memory, and
so either the filesystem knows how to migrate them (through its a_ops)
or the migration code knows how to handle anon folios directly.
Right, migration of folios will be handled by migration core.
quoted
for device private pages, to support migrating >0 order anon or fs folios
to device, how should we represent them for devices? if you think folio is
only for anon and fs.
I assume they are proper folios, so yes. Just like they are handled today (-> folios)
I was asking a related question at LSF/MM in Alistair's session: are we sure these things will be folios even before they are assigned to a filesystem? I recall the answer was "yes".
So we don't (and will not) support movable_ops for folios.
Got it. (I was abusing it to help develop alloc_contig_range() at pageblock
granularity, since it was easy to write a driver to allocate a compound page
at a specific PFN and claim the page is movable, then do page online/offline
the range containing the PFNs. :) )
For the patch, Reviewed-by: Zi Yan [off-list ref]
--
Best Regards,
Yan, Zi
From: David Hildenbrand <hidden> Date: 2025-06-23 15:43:49
On 20.06.25 22:15, Zi Yan wrote:
On 18 Jun 2025, at 13:39, David Hildenbrand wrote:
quoted
Unused, let's remove it.
The Chinese docs in Documentation/translations/zh_CN/mm/page_migration.rst
still mention it, but that whole docs is destined to get outdated and
updated by somebody that actually speaks that language.
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/migrate.h | 4 ----
mm/compaction.c | 11 -----------
2 files changed, 15 deletions(-)
The comment for struct movable_operations needs an update too.
"
If page migration is successful, the driver should call
__ClearPageMovable(@src) and return MIGRATEPAGE_SUCCESS.
> "
Ah, thanks, it will simply be "should return MIGRATEPAGE_SUCCESS".
There is more magic to it (the driver must setup the dst page as
movable), but probably that might all be reworked soon.
--
Cheers,
David / dhildenb
@@ -1056,11 +1056,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*Skipanyothertypeofpage*/if(!PageLRU(page)){-/*-*__PageMovablecanreturnfalsepositivesoweneed-*toverifyitunderpage_lock.-*/-if(unlikely(__PageMovable(page))&&+/* Isolation will grab the page lock. */
I feel that the removed comment should stay, since the current comment
makes no sense when I read it alone.
Well, talking about the page lock is moot either way. The thing is,
anything can change while we don't hold a page reference. So should we
change the comment to
/* isolation code will deal with any races. */
... or drop it completely?
In addition, why is __PageMovable() is renamed to page_has_movable_ops() but
__SetPageMovable() stays the same? page_has_movable_ops() and __SetPageMovable()
are functions for checking and setting PAGE_MAPPING_MOVABLE. The naming just
does not look symmetric.
See follow-up commits where __SetPageMovable() is cleaned up.
--
Cheers,
David / dhildenb