Commit c320dbb7 authored by Dev Jain's avatar Dev Jain Committed by Catalin Marinas
Browse files

arm64/mm: Elide TLB flush in certain pte protection transitions 



Currently arm64 does an unconditional TLB flush in mprotect(). This is not
required for some cases, for example, when changing from PROT_NONE to
PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate
growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already
implemented by some other arches as well.

Running a userspace program changing permissions back and forth between
PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken
for the none->rw transition, I get a reduction from 3.2 microseconds to
2.85 microseconds, giving a 12.3% improvement.

Reviewed-by: default avatarKefeng Wang <wangkefeng.wang@huawei.com>
Signed-off-by: default avatarDev Jain <dev.jain@arm.com>
Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
parent 1b214452

arch/arm64/include/asm/tlbflush.h

+27 −0
Original line number Diff line number Diff line
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b 
{

	__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);

}



static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)

{

	ptdesc_t diff = oldval ^ newval;



	/* invalid to valid transition requires no flush */

	if (!(oldval & PTE_VALID))

		return false;



	/* Transition in the SW bits requires no flush */

	diff &= ~PTE_SWBITS_MASK;



	return diff;

}



static inline bool pte_needs_flush(pte_t oldpte, pte_t newpte)

{

	return __pte_flags_need_flush(pte_val(oldpte), pte_val(newpte));

}

#define pte_needs_flush pte_needs_flush



static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd)

{

	return __pte_flags_need_flush(pmd_val(oldpmd), pmd_val(newpmd));

}

#define huge_pmd_needs_flush huge_pmd_needs_flush



#endif



#endif