Commit b5ba474f authored by Nhat Pham's avatar Nhat Pham Committed by Andrew Morton
Browse files

zswap: shrink zswap pool based on memory pressure 

Currently, we only shrink the zswap pool when the user-defined limit is
hit.  This means that if we set the limit too high, cold data that are
unlikely to be used again will reside in the pool, wasting precious
memory.  It is hard to predict how much zswap space will be needed ahead
of time, as this depends on the workload (specifically, on factors such as
memory access patterns and compressibility of the memory pages).

This patch implements a memcg- and NUMA-aware shrinker for zswap, that is
initiated when there is memory pressure.  The shrinker does not have any
parameter that must be tuned by the user, and can be opted in or out on a
per-memcg basis.

Furthermore, to make it more robust for many workloads and prevent
overshrinking (i.e evicting warm pages that might be refaulted into
memory), we build in the following heuristics:

* Estimate the number of warm pages residing in zswap, and attempt to
  protect this region of the zswap LRU.
* Scale the number of freeable objects by an estimate of the memory
  saving factor. The better zswap compresses the data, the fewer pages
  we will evict to swap (as we will otherwise incur IO for relatively
  small memory saving).
* During reclaim, if the shrinker encounters a page that is also being
  brought into memory, the shrinker will cautiously terminate its
  shrinking action, as this is a sign that it is touching the warmer
  region of the zswap LRU.

As a proof of concept, we ran the following synthetic benchmark: build the
linux kernel in a memory-limited cgroup, and allocate some cold data in
tmpfs to see if the shrinker could write them out and improved the overall
performance.  Depending on the amount of cold data generated, we observe
from 14% to 35% reduction in kernel CPU time used in the kernel builds.

[nphamcs@gmail.com: check shrinker enablement early, use less costly stat flushing]
  Link: https://lkml.kernel.org/r/20231206194456.3234203-1-nphamcs@gmail.com
Link: https://lkml.kernel.org/r/20231130194023.4102148-7-nphamcs@gmail.com


Signed-off-by: default avatarNhat Pham <nphamcs@gmail.com>
Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
Tested-by: default avatarBagas Sanjaya <bagasdotme@gmail.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dan Streetman <ddstreet@ieee.org>
Cc: Domenico Cerasuolo <cerasuolodomenico@gmail.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Seth Jennings <sjenning@redhat.com>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Vitaly Wool <vitaly.wool@konsulko.com>
Cc: Yosry Ahmed <yosryahmed@google.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
parent a697dc2b

Documentation/admin-guide/mm/zswap.rst

+10 −0
Original line number Diff line number Diff line
@@ -153,6 +153,16 @@ attribute, e. g.:: 


Setting this parameter to 100 will disable the hysteresis.



When there is a sizable amount of cold memory residing in the zswap pool, it

can be advantageous to proactively write these cold pages to swap and reclaim

the memory for other use cases. By default, the zswap shrinker is disabled.

User can enable it as follows:



  echo Y > /sys/module/zswap/parameters/shrinker_enabled



This can be enabled at the boot time if ``CONFIG_ZSWAP_SHRINKER_DEFAULT_ON`` is

selected.



A debugfs interface is provided for various statistic about pool size, number

of pages stored, same-value filled pages and various counters for the reasons

pages are rejected.

include/linux/mmzone.h

+2 −0
Original line number Diff line number Diff line
@@ -22,6 +22,7 @@ 
#include <linux/mm_types.h>

#include <linux/page-flags.h>

#include <linux/local_lock.h>

#include <linux/zswap.h>

#include <asm/page.h>



/* Free memory management - zoned buddy allocator.  */
@@ -641,6 +642,7 @@ struct lruvec { 
#ifdef CONFIG_MEMCG

	struct pglist_data *pgdat;

#endif

	struct zswap_lruvec_state zswap_lruvec_state;

};



/* Isolate for asynchronous migration */

include/linux/zswap.h

+23 −2
Original line number Diff line number Diff line
@@ -5,20 +5,40 @@ 
#include <linux/types.h>

#include <linux/mm_types.h>



struct lruvec;



extern u64 zswap_pool_total_size;

extern atomic_t zswap_stored_pages;



#ifdef CONFIG_ZSWAP



struct zswap_lruvec_state {

	/*

	 * Number of pages in zswap that should be protected from the shrinker.

	 * This number is an estimate of the following counts:

	 *

	 * a) Recent page faults.

	 * b) Recent insertion to the zswap LRU. This includes new zswap stores,

	 *    as well as recent zswap LRU rotations.

	 *

	 * These pages are likely to be warm, and might incur IO if the are written

	 * to swap.

	 */

	atomic_long_t nr_zswap_protected;

};



bool zswap_store(struct folio *folio);

bool zswap_load(struct folio *folio);

void zswap_invalidate(int type, pgoff_t offset);

void zswap_swapon(int type);

void zswap_swapoff(int type);

void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);



void zswap_lruvec_state_init(struct lruvec *lruvec);

void zswap_page_swapin(struct page *page);

#else



struct zswap_lruvec_state {};



static inline bool zswap_store(struct folio *folio)

{

	return false;
@@ -33,7 +53,8 @@ static inline void zswap_invalidate(int type, pgoff_t offset) {} 
static inline void zswap_swapon(int type) {}

static inline void zswap_swapoff(int type) {}

static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {}



static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {}

static inline void zswap_page_swapin(struct page *page) {}

#endif



#endif /* _LINUX_ZSWAP_H */

mm/Kconfig

+14 −0
Original line number Diff line number Diff line
@@ -61,6 +61,20 @@ config ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON 
	  The cost is that if the page was never dirtied and needs to be

	  swapped out again, it will be re-compressed.



config ZSWAP_SHRINKER_DEFAULT_ON

	bool "Shrink the zswap pool on memory pressure"

	depends on ZSWAP

	default n

	help

	  If selected, the zswap shrinker will be enabled, and the pages

	  stored in the zswap pool will become available for reclaim (i.e

	  written back to the backing swap device) on memory pressure.



	  This means that zswap writeback could happen even if the pool is

	  not yet full, or the cgroup zswap limit has not been reached,

	  reducing the chance that cold pages will reside in the zswap pool

	  and consume memory indefinitely.



choice

	prompt "Default compressor"

	depends on ZSWAP

mm/mmzone.c

+1 −0
Original line number Diff line number Diff line
@@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec) 


	memset(lruvec, 0, sizeof(struct lruvec));

	spin_lock_init(&lruvec->lru_lock);

	zswap_lruvec_state_init(lruvec);



	for_each_lru(lru)

		INIT_LIST_HEAD(&lruvec->lists[lru]);