Commit 4b038a96 authored by Hao Li's avatar Hao Li Committed by Vlastimil Babka
Browse files

slub: keep empty main sheaf as spare in __pcs_replace_empty_main() 



When __pcs_replace_empty_main() fails to obtain a full sheaf directly
from the barn, it may either:

  - Refill an empty sheaf obtained via barn_get_empty_sheaf(), or
  - Allocate a brand new full sheaf via alloc_full_sheaf().

After reacquiring the per-CPU lock, if pcs->main is still empty and
pcs->spare is NULL, the current code donates the empty main sheaf to
the barn via barn_put_empty_sheaf() and installs the full sheaf as
pcs->main, leaving pcs->spare unpopulated.

Instead, keep the existing empty main sheaf locally as the spare:

  pcs->spare = pcs->main;
  pcs->main = full;

This populates pcs->spare earlier, which can reduce future barn traffic.

Suggested-by: default avatarVlastimil Babka <vbabka@suse.cz>
Signed-off-by: default avatarHao Li <haolee.swjtu@gmail.com>
Reviewed-by: default avatarHarry Yoo <harry.yoo@oracle.com>
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Tested-by: default avatarZhao Liu <zhao1.liu@intel.com>
parent d907bf43

mm/slub.c

+4 −1
Original line number Diff line number Diff line
@@ -5049,6 +5049,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, 
	 */



	if (pcs->main->size == 0) {

		if (!pcs->spare)

			pcs->spare = pcs->main;

		else

			barn_put_empty_sheaf(barn, pcs->main);

		pcs->main = full;

		return pcs;