Commit a923705c authored by Kefeng Wang's avatar Kefeng Wang Committed by Catalin Marinas
Browse files

arm64: optimize flush tlb kernel range 



Currently the kernel TLBs is flushed page by page if the target
VA range is less than MAX_DVM_OPS * PAGE_SIZE, otherwise we'll
brutally issue a TLBI ALL.

But we could optimize it when CPU supports TLB range operations,
convert to use __flush_tlb_range_op() like other tlb range flush
to improve performance.

Co-developed-by: default avatarYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: default avatarYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: default avatarKefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
Link: https://lore.kernel.org/r/20240923131351.713304-3-wangkefeng.wang@huawei.com


Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
parent 7ffc13e2

arch/arm64/include/asm/tlbflush.h

+9 −7
Original line number Diff line number Diff line
@@ -501,19 +501,21 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, 


static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)

{

	unsigned long addr;

	const unsigned long stride = PAGE_SIZE;

	unsigned long pages;



	if ((end - start) > (MAX_DVM_OPS * PAGE_SIZE)) {

	start = round_down(start, stride);

	end = round_up(end, stride);

	pages = (end - start) >> PAGE_SHIFT;



	if (__flush_tlb_range_limit_excess(start, end, pages, stride)) {

		flush_tlb_all();

		return;

	}



	start = __TLBI_VADDR(start, 0);

	end = __TLBI_VADDR(end, 0);



	dsb(ishst);

	for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))

		__tlbi(vaale1is, addr);

	__flush_tlb_range_op(vaale1is, start, pages, stride, 0,

			     TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());

	dsb(ish);

	isb();

}