Commit 3dfde978 authored by David Hildenbrand's avatar David Hildenbrand Committed by Andrew Morton
Browse files

mm: add get_and_clear_ptes() and clear_ptes() 

Patch series "Optimizations for khugepaged", v4.

If the underlying folio mapped by the ptes is large, we can process those
ptes in a batch using folio_pte_batch().

For arm64 specifically, this results in a 16x reduction in the number of
ptep_get() calls, since on a contig block, ptep_get() on arm64 will
iterate through all 16 entries to collect a/d bits.  Next, ptep_clear()
will cause a TLBI for every contig block in the range via
contpte_try_unfold().  Instead, use clear_ptes() to only do the TLBI at
the first and last contig block of the range.

For split folios, there will be no pte batching; the batch size returned
by folio_pte_batch() will be 1.  For pagetable split folios, the ptes will
still point to the same large folio; for arm64, this results in the
optimization described above, and for other arches, a minor improvement is
expected due to a reduction in the number of function calls and batching
atomic operations.


This patch (of 3):

Let's add variants to be used where "full" does not apply -- which will
be the majority of cases in the future. "full" really only applies if
we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will
be added next.

Link: https://lkml.kernel.org/r/20250724052301.23844-2-dev.jain@arm.com


Signed-off-by: default avatarDavid Hildenbrand <david@redhat.com>
Signed-off-by: default avatarDev Jain <dev.jain@arm.com>
Reviewed-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: default avatarBarry Song <baohua@kernel.org>
Reviewed-by: default avatarLorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: default avatarZi Yan <ziy@nvidia.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Mariano Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
parent 1623717b

arch/arm64/mm/mmu.c

+1 −1
Original line number Diff line number Diff line
@@ -1528,7 +1528,7 @@ early_initcall(prevent_bootmem_remove_init); 
pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,

			     pte_t *ptep, unsigned int nr)

{

	pte_t pte = get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full = */ 0);

	pte_t pte = get_and_clear_ptes(vma->vm_mm, addr, ptep, nr);



	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) {

		/*

include/linux/pgtable.h

+45 −0
Original line number Diff line number Diff line
@@ -736,6 +736,29 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, 
}

#endif



/**

 * get_and_clear_ptes - Clear present PTEs that map consecutive pages of

 *			the same folio, collecting dirty/accessed bits.

 * @mm: Address space the pages are mapped into.

 * @addr: Address the first page is mapped at.

 * @ptep: Page table pointer for the first entry.

 * @nr: Number of entries to clear.

 *

 * Use this instead of get_and_clear_full_ptes() if it is known that we don't

 * need to clear the full mm, which is mostly the case.

 *

 * Note that PTE bits in the PTE range besides the PFN can differ. For example,

 * some PTEs might be write-protected.

 *

 * Context: The caller holds the page table lock.  The PTEs map consecutive

 * pages that belong to the same folio.  The PTEs are all in the same PMD.

 */

static inline pte_t get_and_clear_ptes(struct mm_struct *mm, unsigned long addr,

		pte_t *ptep, unsigned int nr)

{

	return get_and_clear_full_ptes(mm, addr, ptep, nr, 0);

}



#ifndef clear_full_ptes

/**

 * clear_full_ptes - Clear present PTEs that map consecutive pages of the same
@@ -768,6 +791,28 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, 
}

#endif



/**

 * clear_ptes - Clear present PTEs that map consecutive pages of the same folio.

 * @mm: Address space the pages are mapped into.

 * @addr: Address the first page is mapped at.

 * @ptep: Page table pointer for the first entry.

 * @nr: Number of entries to clear.

 *

 * Use this instead of clear_full_ptes() if it is known that we don't need to

 * clear the full mm, which is mostly the case.

 *

 * Note that PTE bits in the PTE range besides the PFN can differ. For example,

 * some PTEs might be write-protected.

 *

 * Context: The caller holds the page table lock.  The PTEs map consecutive

 * pages that belong to the same folio.  The PTEs are all in the same PMD.

 */

static inline void clear_ptes(struct mm_struct *mm, unsigned long addr,

		pte_t *ptep, unsigned int nr)

{

	clear_full_ptes(mm, addr, ptep, nr, 0);

}



/*

 * If two threads concurrently fault at the same page, the thread that

 * won the race updates the PTE and its local TLB/Cache. The other thread

mm/mremap.c

+1 −1
Original line number Diff line number Diff line
@@ -280,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc, 
							 old_pte, max_nr_ptes);

			force_flush = true;

		}

		pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0);

		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);

		pte = move_pte(pte, old_addr, new_addr);

		pte = move_soft_dirty_pte(pte);

mm/rmap.c

+1 −1
Original line number Diff line number Diff line
@@ -2036,7 +2036,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, 
			flush_cache_range(vma, address, end_addr);



			/* Nuke the page table entry. */

			pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0);

			pteval = get_and_clear_ptes(mm, address, pvmw.pte, nr_pages);

			/*

			 * We clear the PTE but do not flush so potentially

			 * a remote CPU could still be writing to the folio.