Unverified Commit eb87e56d authored Apr 21, 2025 by Guo Ren Committed by Palmer Dabbelt Jun 05, 2025

riscv: xchg: Prefetch the destination word for sc.w



The cost of changing a cacheline from shared to exclusive state can be
significant, especially when this is triggered by an exclusive store,
since it may result in having to retry the transaction.

This patch makes use of prefetch.w to prefetch cachelines for write
prior to lr/sc loops when using the xchg_small atomic routine.

This patch is inspired by commit 0ea366f5 ("arm64: atomics:
prefetch the destination word for write prior to stxr").

Signed-off-by: Guo Ren <guoren@linux.alibaba.com>
Signed-off-by: Guo Ren <guoren@kernel.org>
Link: https://lore.kernel.org/r/20231231082955.16516-4-guoren@kernel.org


Tested-by: Andrea Parri <parri.andrea@gmail.com>
Link: https://lore.kernel.org/r/20250421142441.395849-5-alexghiti@rivosinc.com


Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>

parent a5f947c7

arch/riscv/include/asm/cmpxchg.h

+3 −1

Original line number	Diff line number	Diff line
		@@ -13,6 +13,7 @@
		#include <asm/hwcap.h>
		#include <asm/insn-def.h>
		#include <asm/cpufeature-macros.h>
		#include <asm/processor.h>

		#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \
		swap_append, r, p, n) \
		@@ -37,6 +38,7 @@
		\
		__asm__ __volatile__ ( \
		prepend \
		PREFETCHW_ASM(%5) \
		"0: lr.w %0, %2\n" \
		" and %1, %0, %z4\n" \
		" or %1, %1, %z3\n" \
		@@ -44,7 +46,7 @@
		" bnez %1, 0b\n" \
		sc_append \
		: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
		: "rJ" (__newx), "rJ" (~__mask) \
		: "rJ" (__newx), "rJ" (~__mask), "rJ" (__ptr32b) \
		: "memory"); \
		\
		r = (__typeof__(*(p)))((__retx & __mask) >> __s); \