[RFC PATCH 20/57] mm/collapse: put the sources back
From: Kiryl Shutsemau <hidden>
Date: 2026-08-16 22:46:57
Also in:
bpf, linux-kselftest, linux-mm, lkml
Subsystem:
memory management, the rest · Maintainers:
Andrew Morton, Linus Torvalds
From: "Kiryl Shutsemau (Meta)" <kas@kernel.org> Fill in the putback: for every installed candidate, lower the barriers the freeze raised, span by span. This is also what wakes the faulters the collapse held up. They sleep on a source folio's lock; once it is dropped they refault and find present PTEs pointing at the new folio. The order within a span is important: - Unfreeze first. Rmap removal munlocks under VM_LOCKED, and munlock_folio() takes a reference a frozen folio forbids. - Then drop the rmap. Until it is gone the expected count still holds the span's mapping references; afterwards they belong to the round, so every folio_remove_rmap_ptes() is paired with a folio_put_refs() for the same slots. - Then unlock, which is the wake. Holding the lock until here keeps lock-taking rmap walkers out, and the window it leaves -- a live folio with no PTEs -- is one any teardown of a mapped folio passes through. - Drop the references strictly last, the round's included. Waiters wait without a reference of their own, so the round's has to outlive the unlock. The stale swapcache entry goes too: the copy has replaced what it described. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org> --- mm/collapse.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
diff --git a/mm/collapse.c b/mm/collapse.c
index ab7476471b8d..f65f413339bf 100644
--- a/mm/collapse.c
+++ b/mm/collapse.c@@ -1439,6 +1439,56 @@ static void collapse_install(struct vm_area_struct *vma, static void collapse_putback(struct vm_area_struct *vma, struct collapse_control *cc) { + unsigned int i; + + for (i = 0; i < cc->nr_candidates; i++) { + struct collapse_candidate *cand = &cc->candidates[i]; + const unsigned int nr_pages = candidate_nr_pages(cand); + unsigned int k = 0; + + if (cand->state != CAND_INSTALLED) + continue; + + while (k < nr_pages) { + struct folio *folio; + unsigned int nr; + + /* A slot with no source has nothing to put back */ + if (pte_none_or_zero(cand->saved_ptes[k])) { + k++; + continue; + } + + folio = pte_folio(cand->saved_ptes[k]); + nr = collapse_saved_span_len(cand, k, nr_pages); + + /* + * Unfreeze before the rmap drop: rmap removal munlocks + * under VM_LOCKED, and munlock_folio() takes a reference + * a frozen folio forbids. The expected count still + * holds the span's mapping references; once the rmap is + * gone they are ours to drop, so every + * folio_remove_rmap_ptes() is paired with a + * folio_put_refs() for the same slots. The folio lock + * is held until the wake below, so lock-taking rmap + * walkers stay excluded, and the stale-rmap window this + * leaves -- live folio, no PTEs -- is one any teardown of + * a mapped folio passes through. + */ + folio_ref_unfreeze(folio, + folio_expected_ref_count(folio) + 1); + folio_remove_rmap_ptes(folio, + pte_page(cand->saved_ptes[k]), + nr, vma); + folio_unlock(folio); + + /* The copy replaced it; drop the stale swap entry */ + free_swap_cache(folio); + folio_put_refs(folio, nr + 1); + + k += nr; + } + } } /*
--
2.54.0