Commit f14cdb13 authored by Alexei Starovoitov's avatar Alexei Starovoitov
Browse files

Merge branch 'remove-kf_sleepable-from-arena-kfuncs' 

Puranjay Mohan says:

====================
Remove KF_SLEEPABLE from arena kfuncs

V7: https://lore.kernel.org/all/20251222190815.4112944-1-puranjay@kernel.org/
Changes in V7->v8:
- Use clear_lo32(arena->user_vm_start) in place of user_vm_start in patch 3

V6: https://lore.kernel.org/all/20251217184438.3557859-1-puranjay@kernel.org/
Changes in v6->v7:
- Fix a deadlock in patch 1, that was being fixed in patch 2. Move the fix to patch 1.
- Call flush_cache_vmap() after setting up the mappings as it is
  required by some architectures.

V5: https://lore.kernel.org/all/20251212044516.37513-1-puranjay@kernel.org/
Changes in v5->v6:
Patch 1:
	- Add a missing ; to make sure this patch builds individually. (AI)

V4: https://lore.kernel.org/all/20251212004350.6520-1-puranjay@kernel.org/
Changes in v4->v5:
Patch 1:
	- Fix a memory leak in arena_alloc_pages(), it was being fixed in
	  Patch 3 but, every patch should be complete in itself. (AI)
Patch 3:
	- Don't do useless addition in arena_alloc_pages() (Alexei)
	- Add a comment about kmalloc_nolock() failure and expectations.

v3: https://lore.kernel.org/all/20251117160150.62183-1-puranjay@kernel.org/
Changes in v3->v4:
	- Coding style changes related to comments in Patch 2/3 (Alexei)

v2: https://lore.kernel.org/all/20251114111700.43292-1-puranjay@kernel.org/
Changes in v2->v3:
Patch 1:
        - Call range_tree_destroy() in error path of
          populate_pgtable_except_pte() in arena_map_alloc() (AI)
Patch 2:
        - Fix double mutex_unlock() in the error path of
          arena_alloc_pages() (AI)
        - Fix coding style issues (Alexei)
Patch 3:
        - Unlock spinlock before returning from arena_vm_fault() in case
          BPF_F_SEGV_ON_FAULT is set by user. (AI)
        - Use __llist_del_all() in place of llist_del_all for on-stack
          llist (free_pages) (Alexei)
        - Fix build issues on 32-bit systems where arena.c is not compiled.
          (kernel test robot)
        - Make bpf_arena_alloc_pages() polymorphic so it knows if it has
          been called in sleepable or non-sleepable context. This
          information is passed to arena_free_pages() in the error path.
Patch 4:
        - Add a better comment for the big_alloc3() test that triggers
          kmalloc_nolock()'s limit and if bpf_arena_alloc_pages() works
          correctly above this limit.

v1: https://lore.kernel.org/all/20251111163424.16471-1-puranjay@kernel.org/
Changes in v1->v2:
Patch 1:
        - Import tlbflush.h to fix build issue in loongarch. (kernel
          test robot)
        - Fix unused variable error in apply_range_clear_cb() (kernel
          test robot)
        - Call bpf_map_area_free() on error path of
          populate_pgtable_except_pte() (AI)
        - Use PAGE_SIZE in apply_to_existing_page_range() (AI)
Patch 2:
        - Cap allocation made by kmalloc_nolock() for pages array to
          KMALLOC_MAX_CACHE_SIZE and reuse the array in an explicit loop
          to overcome this limit. (AI)
Patch 3:
        - Do page_ref_add(page, 1); under the spinlock to mitigate a
          race (AI)
Patch 4:
        - Add a new testcase big_alloc3() verifier_arena_large.c that
          tries to allocate a large number of pages at once, this is to
          trigger the kmalloc_nolock() limit in Patch 2 and see if the
          loop logic works correctly.

This set allows arena kfuncs to be called from non-sleepable contexts.
It is acheived by the following changes:

The range_tree is now protected with a rqspinlock and not a mutex,
this change is enough to make bpf_arena_reserve_pages() any context
safe.

bpf_arena_alloc_pages() had four points where it could sleep:

1. Mutex to protect range_tree: now replaced with rqspinlock

2. kvcalloc() for allocations: now replaced with kmalloc_nolock()

3. Allocating pages with bpf_map_alloc_pages(): this already calls
   alloc_pages_nolock() in non-sleepable contexts and therefore is safe.

4. Setting up kernel page tables with vm_area_map_pages():
   vm_area_map_pages() may allocate memory while inserting pages into
   bpf arena's vm_area. Now, at arena creation time populate all page
   table levels except the last level and when new pages need to be
   inserted call apply_to_page_range() again which will only do
   set_pte_at() for those pages and will not allocate memory.

The above four changes make bpf_arena_alloc_pages() any context safe.

bpf_arena_free_pages() has to do the following steps:

1. Update the range_tree
2. vm_area_unmap_pages(): to unmap pages from kernel vm_area
3. flush the tlb: done in step 2, already.
4. zap_pages(): to unmap pages from user page tables
5. free pages.

The third patch in this set makes bpf_arena_free_pages() polymorphic using
the specialize_kfunc() mechanism. When called from a sleepable context,
arena_free_pages() remains mostly unchanged except the following:
1. rqspinlock is taken now instead of the mutex for the range tree
2. Instead of using vm_area_unmap_pages() that can free intermediate page
   table levels, apply_to_existing_page_range() with a callback is used
   that only does pte_clear() on the last level and leaves the intermediate
   page table levels intact. This is needed to make sure that
   bpf_arena_alloc_pages() can safely do set_pte_at() without allocating
   intermediate page tables.

When arena_free_pages() is called from a non-sleepable context or it fails to
acquire the rqspinlock in the sleepable case, a lock-less list of struct
arena_free_span is used to queue the uaddr and page cnt. kmalloc_nolock()
is used to allocate this arena_free_span, this can fail but we need to make
this trade-off for frees done from non-sleepable contexts.

arena_free_pages() then raises an irq_work whose handler in turn schedules
work that iterate this list and clears ptes, flushes tlbs, zap pages, and
frees pages for the queued uaddr and page cnts.

apply_range_clear_cb() with apply_to_existing_page_range() is used to
clear PTEs and collect pages to be freed, struct llist_node pcp_llist;
in the struct page is used to do this.
====================

Link: https://patch.msgid.link/20251222195022.431211-1-puranjay@kernel.org


Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parents ac1c5bc7 efecc9e8

include/linux/bpf.h

+16 −0
Original line number Diff line number Diff line
@@ -673,6 +673,22 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj); 
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,

				   struct bpf_dynptr *ptr__uninit);



#if defined(CONFIG_MMU) && defined(CONFIG_64BIT)

void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,

					  u64 flags);

void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);

#else

static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,

							int node_id, u64 flags)

{

	return NULL;

}



static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)

{

}

#endif



extern const struct bpf_map_ops bpf_map_offload_ops;



/* bpf_type_flag contains a set of flags that are applicable to the values of

kernel/bpf/arena.c

+326 −54
Original line number Diff line number Diff line
@@ -2,11 +2,15 @@ 
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */

#include <linux/bpf.h>

#include <linux/btf.h>

#include <linux/cacheflush.h>

#include <linux/err.h>

#include <linux/irq_work.h>

#include "linux/filter.h"

#include <linux/llist.h>

#include <linux/btf_ids.h>

#include <linux/vmalloc.h>

#include <linux/pagemap.h>

#include <asm/tlbflush.h>

#include "range_tree.h"



/*
@@ -42,14 +46,31 @@ 
#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)

#define KERN_VM_SZ (SZ_4G + GUARD_SZ)



static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable);



struct bpf_arena {

	struct bpf_map map;

	u64 user_vm_start;

	u64 user_vm_end;

	struct vm_struct *kern_vm;

	struct range_tree rt;

	/* protects rt */

	rqspinlock_t spinlock;

	struct list_head vma_list;

	/* protects vma_list */

	struct mutex lock;

	struct irq_work     free_irq;

	struct work_struct  free_work;

	struct llist_head   free_spans;

};



static void arena_free_worker(struct work_struct *work);

static void arena_free_irq(struct irq_work *iw);



struct arena_free_span {

	struct llist_node node;

	unsigned long uaddr;

	u32 page_cnt;

};



u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)
@@ -92,6 +113,66 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) 
	return (u32)(uaddr - (u32)arena->user_vm_start) >> PAGE_SHIFT;

}



struct apply_range_data {

	struct page **pages;

	int i;

};



static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)

{

	struct apply_range_data *d = data;

	struct page *page;



	if (!data)

		return 0;

	/* sanity check */

	if (unlikely(!pte_none(ptep_get(pte))))

		return -EBUSY;



	page = d->pages[d->i];

	/* paranoia, similar to vmap_pages_pte_range() */

	if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))

		return -EINVAL;



	set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));

	d->i++;

	return 0;

}



static void flush_vmap_cache(unsigned long start, unsigned long size)

{

	flush_cache_vmap(start, start + size);

}



static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages)

{

	pte_t old_pte;

	struct page *page;



	/* sanity check */

	old_pte = ptep_get(pte);

	if (pte_none(old_pte) || !pte_present(old_pte))

		return 0; /* nothing to do */



	page = pte_page(old_pte);

	if (WARN_ON_ONCE(!page))

		return -EINVAL;



	pte_clear(&init_mm, addr, pte);



	/* Add page to the list so it is freed later */

	if (free_pages)

		__llist_add(&page->pcp_llist, free_pages);



	return 0;

}



static int populate_pgtable_except_pte(struct bpf_arena *arena)

{

	return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),

				   KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);

}



static struct bpf_map *arena_map_alloc(union bpf_attr *attr)

{

	struct vm_struct *kern_vm;
@@ -136,6 +217,9 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) 
		arena->user_vm_end = arena->user_vm_start + vm_range;



	INIT_LIST_HEAD(&arena->vma_list);

	init_llist_head(&arena->free_spans);

	init_irq_work(&arena->free_irq, arena_free_irq);

	INIT_WORK(&arena->free_work, arena_free_worker);

	bpf_map_init_from_attr(&arena->map, attr);

	range_tree_init(&arena->rt);

	err = range_tree_set(&arena->rt, 0, attr->max_entries);
@@ -144,6 +228,13 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) 
		goto err;

	}

	mutex_init(&arena->lock);

	raw_res_spin_lock_init(&arena->spinlock);

	err = populate_pgtable_except_pte(arena);

	if (err) {

		range_tree_destroy(&arena->rt);

		bpf_map_area_free(arena);

		goto err;

	}



	return &arena->map;

err:
@@ -184,6 +275,10 @@ static void arena_map_free(struct bpf_map *map) 
	if (WARN_ON_ONCE(!list_empty(&arena->vma_list)))

		return;



	/* Ensure no pending deferred frees */

	irq_work_sync(&arena->free_irq);

	flush_work(&arena->free_work);



	/*

	 * free_vm_area() calls remove_vm_area() that calls free_unmap_vmap_area().

	 * It unmaps everything from vmalloc area and clears pgtables.
@@ -267,12 +362,16 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) 
	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);

	struct page *page;

	long kbase, kaddr;

	unsigned long flags;

	int ret;



	kbase = bpf_arena_get_kern_vm_start(arena);

	kaddr = kbase + (u32)(vmf->address);



	guard(mutex)(&arena->lock);

	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))

		/* Make a reasonable effort to address impossible case */

		return VM_FAULT_RETRY;



	page = vmalloc_to_page((void *)kaddr);

	if (page)

		/* already have a page vmap-ed */
@@ -280,29 +379,35 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) 


	if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)

		/* User space requested to segfault when page is not allocated by bpf prog */

		return VM_FAULT_SIGSEGV;

		goto out_unlock_sigsegv;



	ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);

	if (ret)

		return VM_FAULT_SIGSEGV;

		goto out_unlock_sigsegv;



	struct apply_range_data data = { .pages = &page, .i = 0 };

	/* Account into memcg of the process that created bpf_arena */

	ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);

	if (ret) {

		range_tree_set(&arena->rt, vmf->pgoff, 1);

		return VM_FAULT_SIGSEGV;

		goto out_unlock_sigsegv;

	}



	ret = vm_area_map_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE, &page);

	ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);

	if (ret) {

		range_tree_set(&arena->rt, vmf->pgoff, 1);

		__free_page(page);

		return VM_FAULT_SIGSEGV;

		free_pages_nolock(page, 0);

		goto out_unlock_sigsegv;

	}

	flush_vmap_cache(kaddr, PAGE_SIZE);

out:

	page_ref_add(page, 1);

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

	vmf->page = page;

	return 0;

out_unlock_sigsegv:

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

	return VM_FAULT_SIGSEGV;

}



static const struct vm_operations_struct arena_vm_ops = {
@@ -423,12 +528,17 @@ static u64 clear_lo32(u64 val) 
 * Allocate pages and vmap them into kernel vmalloc area.

 * Later the pages will be mmaped into user space vma.

 */

static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id)

static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id,

			      bool sleepable)

{

	/* user_vm_end/start are fixed before bpf prog runs */

	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;

	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);

	struct page **pages;

	struct apply_range_data data;

	struct page **pages = NULL;

	long remaining, mapped = 0;

	long alloc_pages;

	unsigned long flags;

	long pgoff = 0;

	u32 uaddr32;

	int ret, i;
@@ -445,17 +555,20 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt 
			return 0;

	}



	/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */

	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL);

	/* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. */

	alloc_pages = min(page_cnt, KMALLOC_MAX_CACHE_SIZE / sizeof(struct page *));

	pages = kmalloc_nolock(alloc_pages * sizeof(struct page *), 0, NUMA_NO_NODE);

	if (!pages)

		return 0;

	data.pages = pages;



	guard(mutex)(&arena->lock);

	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))

		goto out_free_pages;



	if (uaddr) {

		ret = is_range_tree_set(&arena->rt, pgoff, page_cnt);

		if (ret)

			goto out_free_pages;

			goto out_unlock_free_pages;

		ret = range_tree_clear(&arena->rt, pgoff, page_cnt);

	} else {

		ret = pgoff = range_tree_find(&arena->rt, page_cnt);
@@ -463,33 +576,60 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt 
			ret = range_tree_clear(&arena->rt, pgoff, page_cnt);

	}

	if (ret)

		goto out_free_pages;

		goto out_unlock_free_pages;



	remaining = page_cnt;

	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);



	while (remaining) {

		long this_batch = min(remaining, alloc_pages);



	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);

		/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */

		memset(pages, 0, this_batch * sizeof(struct page *));



		ret = bpf_map_alloc_pages(&arena->map, node_id, this_batch, pages);

		if (ret)

			goto out;



	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);

	/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1

		/*

		 * Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1

		 * will not overflow 32-bit. Lower 32-bit need to represent

		 * contiguous user address range.

		 * Map these pages at kern_vm_start base.

		 * kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow

		 * lower 32-bit and it's ok.

		 */

	ret = vm_area_map_pages(arena->kern_vm, kern_vm_start + uaddr32,

				kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE, pages);

		data.i = 0;

		ret = apply_to_page_range(&init_mm,

					  kern_vm_start + uaddr32 + (mapped << PAGE_SHIFT),

					  this_batch << PAGE_SHIFT, apply_range_set_cb, &data);

		if (ret) {

		for (i = 0; i < page_cnt; i++)

			__free_page(pages[i]);

			/* data.i pages were mapped, account them and free the remaining */

			mapped += data.i;

			for (i = data.i; i < this_batch; i++)

				free_pages_nolock(pages[i], 0);

			goto out;

		}

	kvfree(pages);



		mapped += this_batch;

		remaining -= this_batch;

	}

	flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

	kfree_nolock(pages);

	return clear_lo32(arena->user_vm_start) + uaddr32;

out:

	range_tree_set(&arena->rt, pgoff, page_cnt);

	range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped);

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

	if (mapped) {

		flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);

		arena_free_pages(arena, uaddr32, mapped, sleepable);

	}

	goto out_free_pages;

out_unlock_free_pages:

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

out_free_pages:

	kvfree(pages);

	kfree_nolock(pages);

	return 0;

}
@@ -502,42 +642,64 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) 
{

	struct vma_list *vml;



	guard(mutex)(&arena->lock);

	/* iterate link list under lock */

	list_for_each_entry(vml, &arena->vma_list, head)

		zap_page_range_single(vml->vma, uaddr,

				      PAGE_SIZE * page_cnt, NULL);

}



static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt)

static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)

{

	u64 full_uaddr, uaddr_end;

	long kaddr, pgoff, i;

	long kaddr, pgoff;

	struct page *page;

	struct llist_head free_pages;

	struct llist_node *pos, *t;

	struct arena_free_span *s;

	unsigned long flags;

	int ret = 0;



	/* only aligned lower 32-bit are relevant */

	uaddr = (u32)uaddr;

	uaddr &= PAGE_MASK;

	kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;

	full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;

	uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));

	if (full_uaddr >= uaddr_end)

		return;



	page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT;

	pgoff = compute_pgoff(arena, uaddr);



	guard(mutex)(&arena->lock);

	if (!sleepable)

		goto defer;



	ret = raw_res_spin_lock_irqsave(&arena->spinlock, flags);



	/* Can't proceed without holding the spinlock so defer the free */

	if (ret)

		goto defer;



	pgoff = compute_pgoff(arena, uaddr);

	/* clear range */

	range_tree_set(&arena->rt, pgoff, page_cnt);



	init_llist_head(&free_pages);

	/* clear ptes and collect struct pages */

	apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,

				     apply_range_clear_cb, &free_pages);



	/* drop the lock to do the tlb flush and zap pages */

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);



	/* ensure no stale TLB entries */

	flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));



	if (page_cnt > 1)

		/* bulk zap if multiple pages being freed */

		zap_pages(arena, full_uaddr, page_cnt);



	kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;

	for (i = 0; i < page_cnt; i++, kaddr += PAGE_SIZE, full_uaddr += PAGE_SIZE) {

		page = vmalloc_to_page((void *)kaddr);

		if (!page)

			continue;

	llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {

		page = llist_entry(pos, struct page, pcp_llist);

		if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */

			/* Optimization for the common case of page_cnt==1:

			 * If page wasn't mapped into some user vma there
@@ -545,9 +707,25 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) 
			 * page_cnt is big it's faster to do the batched zap.

			 */

			zap_pages(arena, full_uaddr, 1);

		vm_area_unmap_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE);

		__free_page(page);

	}



	return;



defer:

	s = kmalloc_nolock(sizeof(struct arena_free_span), 0, -1);

	if (!s)

		/*

		 * If allocation fails in non-sleepable context, pages are intentionally left

		 * inaccessible (leaked) until the arena is destroyed. Cleanup or retries are not

		 * possible here, so we intentionally omit them for safety.

		 */

		return;



	s->page_cnt = page_cnt;

	s->uaddr = uaddr;

	llist_add(&s->node, &arena->free_spans);

	irq_work_queue(&arena->free_irq);

}



/*
@@ -557,6 +735,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) 
static int arena_reserve_pages(struct bpf_arena *arena, long uaddr, u32 page_cnt)

{

	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;

	unsigned long flags;

	long pgoff;

	int ret;
@@ -567,15 +746,87 @@ static int arena_reserve_pages(struct bpf_arena *arena, long uaddr, u32 page_cnt 
	if (pgoff + page_cnt > page_cnt_max)

		return -EINVAL;



	guard(mutex)(&arena->lock);

	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))

		return -EBUSY;



	/* Cannot guard already allocated pages. */

	ret = is_range_tree_set(&arena->rt, pgoff, page_cnt);

	if (ret)

		return -EBUSY;

	if (ret) {

		ret = -EBUSY;

		goto out;

	}



	/* "Allocate" the region to prevent it from being allocated. */

	return range_tree_clear(&arena->rt, pgoff, page_cnt);

	ret = range_tree_clear(&arena->rt, pgoff, page_cnt);

out:

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

	return ret;

}



static void arena_free_worker(struct work_struct *work)

{

	struct bpf_arena *arena = container_of(work, struct bpf_arena, free_work);

	struct llist_node *list, *pos, *t;

	struct arena_free_span *s;

	u64 arena_vm_start, user_vm_start;

	struct llist_head free_pages;

	struct page *page;

	unsigned long full_uaddr;

	long kaddr, page_cnt, pgoff;

	unsigned long flags;



	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) {

		schedule_work(work);

		return;

	}



	init_llist_head(&free_pages);

	arena_vm_start = bpf_arena_get_kern_vm_start(arena);

	user_vm_start = bpf_arena_get_user_vm_start(arena);



	list = llist_del_all(&arena->free_spans);

	llist_for_each(pos, list) {

		s = llist_entry(pos, struct arena_free_span, node);

		page_cnt = s->page_cnt;

		kaddr = arena_vm_start + s->uaddr;

		pgoff = compute_pgoff(arena, s->uaddr);



		/* clear ptes and collect pages in free_pages llist */

		apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,

					     apply_range_clear_cb, &free_pages);



		range_tree_set(&arena->rt, pgoff, page_cnt);

	}

	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);



	/* Iterate the list again without holding spinlock to do the tlb flush and zap_pages */

	llist_for_each_safe(pos, t, list) {

		s = llist_entry(pos, struct arena_free_span, node);

		page_cnt = s->page_cnt;

		full_uaddr = clear_lo32(user_vm_start) + s->uaddr;

		kaddr = arena_vm_start + s->uaddr;



		/* ensure no stale TLB entries */

		flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));



		/* remove pages from user vmas */

		zap_pages(arena, full_uaddr, page_cnt);



		kfree_nolock(s);

	}



	/* free all pages collected by apply_to_existing_page_range() in the first loop */

	llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {

		page = llist_entry(pos, struct page, pcp_llist);

		__free_page(page);

	}

}



static void arena_free_irq(struct irq_work *iw)

{

	struct bpf_arena *arena = container_of(iw, struct bpf_arena, free_irq);



	schedule_work(&arena->free_work);

}



__bpf_kfunc_start_defs();
@@ -589,9 +840,20 @@ __bpf_kfunc void *bpf_arena_alloc_pages(void *p__map, void *addr__ign, u32 page_ 
	if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)

		return NULL;



	return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id);

	return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true);

}



void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,

					  int node_id, u64 flags)

{

	struct bpf_map *map = p__map;

	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);



	if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)

		return NULL;



	return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);

}

__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)

{

	struct bpf_map *map = p__map;
@@ -599,7 +861,17 @@ __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt 


	if (map->map_type != BPF_MAP_TYPE_ARENA || !page_cnt || !ptr__ign)

		return;

	arena_free_pages(arena, (long)ptr__ign, page_cnt);

	arena_free_pages(arena, (long)ptr__ign, page_cnt, true);

}



void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)

{

	struct bpf_map *map = p__map;

	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);



	if (map->map_type != BPF_MAP_TYPE_ARENA || !page_cnt || !ptr__ign)

		return;

	arena_free_pages(arena, (long)ptr__ign, page_cnt, false);

}



__bpf_kfunc int bpf_arena_reserve_pages(void *p__map, void *ptr__ign, u32 page_cnt)
@@ -618,9 +890,9 @@ __bpf_kfunc int bpf_arena_reserve_pages(void *p__map, void *ptr__ign, u32 page_c 
__bpf_kfunc_end_defs();



BTF_KFUNCS_START(arena_kfuncs)

BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE | KF_ARENA_RET | KF_ARENA_ARG2)

BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE | KF_ARENA_ARG2)

BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE | KF_ARENA_ARG2)

BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_TRUSTED_ARGS | KF_ARENA_RET | KF_ARENA_ARG2)

BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_TRUSTED_ARGS | KF_ARENA_ARG2)

BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_TRUSTED_ARGS | KF_ARENA_ARG2)

BTF_KFUNCS_END(arena_kfuncs)



static const struct btf_kfunc_id_set common_kfunc_set = {

kernel/bpf/verifier.c

+10 −0
Original line number Diff line number Diff line
@@ -12380,6 +12380,8 @@ enum special_kfunc_type { 
	KF___bpf_trap,

	KF_bpf_task_work_schedule_signal_impl,

	KF_bpf_task_work_schedule_resume_impl,

	KF_bpf_arena_alloc_pages,

	KF_bpf_arena_free_pages,

};









BTF_ID_LIST(special_kfunc_list)








@@ -12454,6 +12456,8 @@ BTF_ID(func, bpf_dynptr_file_discard)







BTF_ID(func, __bpf_trap)









BTF_ID(func, bpf_task_work_schedule_signal_impl)









BTF_ID(func, bpf_task_work_schedule_resume_impl)









BTF_ID(func, bpf_arena_alloc_pages)









BTF_ID(func, bpf_arena_free_pages)

















static bool is_task_work_add_kfunc(u32 func_id)









{








@@ -22432,6 +22436,12 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc







	} else if (func_id == special_kfunc_list[KF_bpf_dynptr_from_file]) {









		if (!env->insn_aux_data[insn_idx].non_sleepable)









			addr = (unsigned long)bpf_dynptr_from_file_sleepable;









	} else if (func_id == special_kfunc_list[KF_bpf_arena_alloc_pages]) {









		if (env->insn_aux_data[insn_idx].non_sleepable)









			addr = (unsigned long)bpf_arena_alloc_pages_non_sleepable;









	} else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) {









		if (env->insn_aux_data[insn_idx].non_sleepable)









			addr = (unsigned long)bpf_arena_free_pages_non_sleepable;









	}









	desc->addr = addr;









	return 0;

























tools/testing/selftests/bpf/prog_tests/arena_list.c



+15
−5






























Original line number
Diff line number
Diff line







@@ -27,17 +27,23 @@ static int list_sum(struct arena_list_head *head)







	return sum;









}



















static void test_arena_list_add_del(int cnt)









static void test_arena_list_add_del(int cnt, bool nonsleepable)









{









	LIBBPF_OPTS(bpf_test_run_opts, opts);









	struct arena_list *skel;









	int expected_sum = (u64)cnt * (cnt - 1) / 2;









	int ret, sum;



















	skel = arena_list__open_and_load();









	if (!ASSERT_OK_PTR(skel, "arena_list__open_and_load"))









	skel = arena_list__open();









	if (!ASSERT_OK_PTR(skel, "arena_list__open"))









		return;



















	skel->rodata->nonsleepable = nonsleepable;



















	ret = arena_list__load(skel);









	if (!ASSERT_OK(ret, "arena_list__load"))









		goto out;



















	skel->bss->cnt = cnt;









	ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts);









	ASSERT_OK(ret, "ret_add");








@@ -65,7 +71,11 @@ static void test_arena_list_add_del(int cnt)







void test_arena_list(void)









{









	if (test__start_subtest("arena_list_1"))









		test_arena_list_add_del(1);









		test_arena_list_add_del(1, false);









	if (test__start_subtest("arena_list_1000"))









		test_arena_list_add_del(1000);









		test_arena_list_add_del(1000, false);









	if (test__start_subtest("arena_list_1_nonsleepable"))









		test_arena_list_add_del(1, true);









	if (test__start_subtest("arena_list_1000_nonsleepable"))









		test_arena_list_add_del(1000, true);









}
























tools/testing/selftests/bpf/progs/arena_list.c



+11
−0






























Original line number
Diff line number
Diff line







@@ -30,6 +30,7 @@ struct arena_list_head __arena *list_head;







int list_sum;









int cnt;









bool skip = false;









const volatile bool nonsleepable = false;



















#ifdef __BPF_FEATURE_ADDR_SPACE_CAST









long __arena arena_sum;








@@ -42,6 +43,9 @@ int test_val SEC(".addr_space.1");

















int zero;



















void bpf_rcu_read_lock(void) __ksym;









void bpf_rcu_read_unlock(void) __ksym;



















SEC("syscall")









int arena_list_add(void *ctx)









{








@@ -71,6 +75,10 @@ int arena_list_del(void *ctx)







	struct elem __arena *n;









	int sum = 0;



















	/* Take rcu_read_lock to test non-sleepable context */









	if (nonsleepable)









		bpf_rcu_read_lock();



















	arena_sum = 0;









	list_for_each_entry(n, list_head, node) {









		sum += n->value;








@@ -79,6 +87,9 @@ int arena_list_del(void *ctx)







		bpf_free(n);









	}









	list_sum = sum;



















	if (nonsleepable)









		bpf_rcu_read_unlock();









#else









	skip = true;









#endif