Commit 6e95ef02 authored by Linus Torvalds's avatar Linus Torvalds
Browse files
Pull bpf updates from Alexei Starovoitov:

 - Add BPF uprobe session support (Jiri Olsa)

 - Optimize uprobe performance (Andrii Nakryiko)

 - Add bpf_fastcall support to helpers and kfuncs (Eduard Zingerman)

 - Avoid calling free_htab_elem() under hash map bucket lock (Hou Tao)

 - Prevent tailcall infinite loop caused by freplace (Leon Hwang)

 - Mark raw_tracepoint arguments as nullable (Kumar Kartikeya Dwivedi)

 - Introduce uptr support in the task local storage map (Martin KaFai
   Lau)

 - Stringify errno log messages in libbpf (Mykyta Yatsenko)

 - Add kmem_cache BPF iterator for perf's lock profiling (Namhyung Kim)

 - Support BPF objects of either endianness in libbpf (Tony Ambardar)

 - Add ksym to struct_ops trampoline to fix stack trace (Xu Kuohai)

 - Introduce private stack for eligible BPF programs (Yonghong Song)

 - Migrate samples/bpf tests to selftests/bpf test_progs (Daniel T. Lee)

 - Migrate test_sock to selftests/bpf test_progs (Jordan Rife)

* tag 'bpf-next-6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (152 commits)
  libbpf: Change hash_combine parameters from long to unsigned long
  selftests/bpf: Fix build error with llvm 19
  libbpf: Fix memory leak in bpf_program__attach_uprobe_multi
  bpf: use common instruction history across all states
  bpf: Add necessary migrate_disable to range_tree.
  bpf: Do not alloc arena on unsupported arches
  selftests/bpf: Set test path for token/obj_priv_implicit_token_envvar
  selftests/bpf: Add a test for arena range tree algorithm
  bpf: Introduce range_tree data structure and use it in bpf arena
  samples/bpf: Remove unused variable in xdp2skb_meta_kern.c
  samples/bpf: Remove unused variables in tc_l2_redirect_kern.c
  bpftool: Cast variable `var` to long long
  bpf, x86: Propagate tailcall info only for subprogs
  bpf: Add kernel symbol for struct_ops trampoline
  bpf: Use function pointers count as struct_ops links count
  bpf: Remove unused member rcu from bpf_struct_ops_map
  selftests/bpf: Add struct_ops prog private stack tests
  bpf: Support private stack for struct_ops progs
  selftests/bpf: Add tracing prog private stack tests
  bpf, x86: Support private stack in jit
  ...
parents 43fb83c1 2c8b09ac
Loading
Loading
Loading
Loading
+76 −1
Original line number Diff line number Diff line
@@ -835,7 +835,7 @@ section named by ``btf_ext_info_sec->sec_name_off``.
See :ref:`Documentation/bpf/llvm_reloc.rst <btf-co-re-relocations>`
for more information on CO-RE relocations.

4.2 .BTF_ids section
4.3 .BTF_ids section
--------------------

The .BTF_ids section encodes BTF ID values that are used within the kernel.
@@ -896,6 +896,81 @@ and is used as a filter when resolving the BTF ID value.
All the BTF ID lists and sets are compiled in the .BTF_ids section and
resolved during the linking phase of kernel build by ``resolve_btfids`` tool.

4.4 .BTF.base section
---------------------
Split BTF - where the .BTF section only contains types not in the associated
base .BTF section - is an extremely efficient way to encode type information
for kernel modules, since they generally consist of a few module-specific
types along with a large set of shared kernel types. The former are encoded
in split BTF, while the latter are encoded in base BTF, resulting in more
compact representations. A type in split BTF that refers to a type in
base BTF refers to it using its base BTF ID, and split BTF IDs start
at last_base_BTF_ID + 1.

The downside of this approach however is that this makes the split BTF
somewhat brittle - when the base BTF changes, base BTF ID references are
no longer valid and the split BTF itself becomes useless. The role of the
.BTF.base section is to make split BTF more resilient for cases where
the base BTF may change, as is the case for kernel modules not built every
time the kernel is for example. .BTF.base contains named base types; INTs,
FLOATs, STRUCTs, UNIONs, ENUM[64]s and FWDs. INTs and FLOATs are fully
described in .BTF.base sections, while composite types like structs
and unions are not fully defined - the .BTF.base type simply serves as
a description of the type the split BTF referred to, so structs/unions
have 0 members in the .BTF.base section. ENUM[64]s are similarly recorded
with 0 members. Any other types are added to the split BTF. This
distillation process then leaves us with a .BTF.base section with
such minimal descriptions of base types and .BTF split section which refers
to those base types. Later, we can relocate the split BTF using both the
information stored in the .BTF.base section and the new .BTF base; the type
information in the .BTF.base section allows us to update the split BTF
references to point at the corresponding new base BTF IDs.

BTF relocation happens on kernel module load when a kernel module has a
.BTF.base section, and libbpf also provides a btf__relocate() API to
accomplish this.

As an example consider the following base BTF::

      [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
      [2] STRUCT 'foo' size=8 vlen=2
              'f1' type_id=1 bits_offset=0
              'f2' type_id=1 bits_offset=32

...and associated split BTF::

      [3] PTR '(anon)' type_id=2

i.e. split BTF describes a pointer to struct foo { int f1; int f2 };

.BTF.base will consist of::

      [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
      [2] STRUCT 'foo' size=8 vlen=0

If we relocate the split BTF later using the following new base BTF::

      [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
      [2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
      [3] STRUCT 'foo' size=8 vlen=2
              'f1' type_id=2 bits_offset=0
              'f2' type_id=2 bits_offset=32

...we can use our .BTF.base description to know that the split BTF reference
is to struct foo, and relocation results in new split BTF::

      [4] PTR '(anon)' type_id=3

Note that we had to update BTF ID and start BTF ID for the split BTF.

So we see how .BTF.base plays the role of facilitating later relocation,
leading to more resilient split BTF.

.BTF.base sections will be generated automatically for out-of-tree kernel module
builds - i.e. where KBUILD_EXTMOD is set (as it would be for "make M=path/2/mod"
cases). .BTF.base generation requires pahole support for the "distilled_base"
BTF feature; this is available in pahole v1.28 and later.

5. Using BTF
============

+31 −16
Original line number Diff line number Diff line
@@ -2094,6 +2094,12 @@ static void restore_args(struct jit_ctx *ctx, int args_off, int nregs)
	}
}

static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links)
{
	return fentry_links->nr_links == 1 &&
		fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS;
}

/* Based on the x86's implementation of arch_prepare_bpf_trampoline().
 *
 * bpf prog and function entry before bpf trampoline hooked:
@@ -2123,6 +2129,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
	struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
	bool save_ret;
	__le32 **branches = NULL;
	bool is_struct_ops = is_struct_ops_tramp(fentry);

	/* trampoline stack layout:
	 *                  [ parent ip         ]
@@ -2191,11 +2198,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
	 */
	emit_bti(A64_BTI_JC, ctx);

	/* x9 is not set for struct_ops */
	if (!is_struct_ops) {
		/* frame for parent function */
		emit(A64_PUSH(A64_FP, A64_R(9), A64_SP), ctx);
		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
	}

	/* frame for patched function */
	/* frame for patched function for tracing, or caller for struct_ops */
	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
	emit(A64_MOV(1, A64_FP, A64_SP), ctx);

@@ -2289,6 +2299,10 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
	/* reset SP  */
	emit(A64_MOV(1, A64_SP, A64_FP), ctx);

	if (is_struct_ops) {
		emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
		emit(A64_RET(A64_LR), ctx);
	} else {
		/* pop frames */
		emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
		emit(A64_POP(A64_FP, A64_R(9), A64_SP), ctx);
@@ -2303,6 +2317,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
			emit(A64_MOV(1, A64_LR, A64_R(9)), ctx);
			emit(A64_RET(A64_R(10)), ctx);
		}
	}

	kfree(branches);

+144 −5
Original line number Diff line number Diff line
@@ -325,6 +325,22 @@ struct jit_context {
/* Number of bytes that will be skipped on tailcall */
#define X86_TAIL_CALL_OFFSET	(12 + ENDBR_INSN_SIZE)

static void push_r9(u8 **pprog)
{
	u8 *prog = *pprog;

	EMIT2(0x41, 0x51);   /* push r9 */
	*pprog = prog;
}

static void pop_r9(u8 **pprog)
{
	u8 *prog = *pprog;

	EMIT2(0x41, 0x59);   /* pop r9 */
	*pprog = prog;
}

static void push_r12(u8 **pprog)
{
	u8 *prog = *pprog;
@@ -1404,6 +1420,24 @@ static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)
	*pprog = prog;
}

static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
{
	u8 *prog = *pprog;

	/* movabs r9, priv_frame_ptr */
	emit_mov_imm64(&prog, X86_REG_R9, (__force long) priv_frame_ptr >> 32,
		       (u32) (__force long) priv_frame_ptr);

#ifdef CONFIG_SMP
	/* add <r9>, gs:[<off>] */
	EMIT2(0x65, 0x4c);
	EMIT3(0x03, 0x0c, 0x25);
	EMIT((u32)(unsigned long)&this_cpu_off, 4);
#endif

	*pprog = prog;
}

#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))

#define __LOAD_TCC_PTR(off)			\
@@ -1412,6 +1446,10 @@ static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)
#define LOAD_TAIL_CALL_CNT_PTR(stack)				\
	__LOAD_TCC_PTR(BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack))

/* Memory size/value to protect private stack overflow/underflow */
#define PRIV_STACK_GUARD_SZ    8
#define PRIV_STACK_GUARD_VAL   0xEB9F12345678eb9fULL

static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image,
		  int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -1421,18 +1459,28 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
	int insn_cnt = bpf_prog->len;
	bool seen_exit = false;
	u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
	void __percpu *priv_frame_ptr = NULL;
	u64 arena_vm_start, user_vm_start;
	void __percpu *priv_stack_ptr;
	int i, excnt = 0;
	int ilen, proglen = 0;
	u8 *prog = temp;
	u32 stack_depth;
	int err;

	stack_depth = bpf_prog->aux->stack_depth;
	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
	if (priv_stack_ptr) {
		priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
		stack_depth = 0;
	}

	arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
	user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);

	detect_reg_usage(insn, insn_cnt, callee_regs_used);

	emit_prologue(&prog, bpf_prog->aux->stack_depth,
	emit_prologue(&prog, stack_depth,
		      bpf_prog_was_classic(bpf_prog), tail_call_reachable,
		      bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
	/* Exception callback will clobber callee regs for its own use, and
@@ -1454,6 +1502,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
		emit_mov_imm64(&prog, X86_REG_R12,
			       arena_vm_start >> 32, (u32) arena_vm_start);

	if (priv_frame_ptr)
		emit_priv_frame_ptr(&prog, priv_frame_ptr);

	ilen = prog - temp;
	if (rw_image)
		memcpy(rw_image + proglen, temp, ilen);
@@ -1473,6 +1524,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
		u8 *func;
		int nops;

		if (priv_frame_ptr) {
			if (src_reg == BPF_REG_FP)
				src_reg = X86_REG_R9;

			if (dst_reg == BPF_REG_FP)
				dst_reg = X86_REG_R9;
		}

		switch (insn->code) {
			/* ALU */
		case BPF_ALU | BPF_ADD | BPF_X:
@@ -2127,15 +2186,21 @@ st: if (is_imm8(insn->off))
			u8 *ip = image + addrs[i - 1];

			func = (u8 *) __bpf_call_base + imm32;
			if (tail_call_reachable) {
				LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
				ip += 7;
			}
			if (!imm32)
				return -EINVAL;
			if (priv_frame_ptr) {
				push_r9(&prog);
				ip += 2;
			}
			ip += x86_call_depth_emit_accounting(&prog, func, ip);
			if (emit_call(&prog, func, ip))
				return -EINVAL;
			if (priv_frame_ptr)
				pop_r9(&prog);
			break;
		}

@@ -2145,13 +2210,13 @@ st: if (is_imm8(insn->off))
							  &bpf_prog->aux->poke_tab[imm32 - 1],
							  &prog, image + addrs[i - 1],
							  callee_regs_used,
							  bpf_prog->aux->stack_depth,
							  stack_depth,
							  ctx);
			else
				emit_bpf_tail_call_indirect(bpf_prog,
							    &prog,
							    callee_regs_used,
							    bpf_prog->aux->stack_depth,
							    stack_depth,
							    image + addrs[i - 1],
							    ctx);
			break;
@@ -3303,6 +3368,42 @@ int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_func
	return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs, image, buf);
}

static const char *bpf_get_prog_name(struct bpf_prog *prog)
{
	if (prog->aux->ksym.prog)
		return prog->aux->ksym.name;
	return prog->aux->name;
}

static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)
{
	int cpu, underflow_idx = (alloc_size - PRIV_STACK_GUARD_SZ) >> 3;
	u64 *stack_ptr;

	for_each_possible_cpu(cpu) {
		stack_ptr = per_cpu_ptr(priv_stack_ptr, cpu);
		stack_ptr[0] = PRIV_STACK_GUARD_VAL;
		stack_ptr[underflow_idx] = PRIV_STACK_GUARD_VAL;
	}
}

static void priv_stack_check_guard(void __percpu *priv_stack_ptr, int alloc_size,
				   struct bpf_prog *prog)
{
	int cpu, underflow_idx = (alloc_size - PRIV_STACK_GUARD_SZ) >> 3;
	u64 *stack_ptr;

	for_each_possible_cpu(cpu) {
		stack_ptr = per_cpu_ptr(priv_stack_ptr, cpu);
		if (stack_ptr[0] != PRIV_STACK_GUARD_VAL ||
		    stack_ptr[underflow_idx] != PRIV_STACK_GUARD_VAL) {
			pr_err("BPF private stack overflow/underflow detected for prog %sx\n",
			       bpf_get_prog_name(prog));
			break;
		}
	}
}

struct x64_jit_data {
	struct bpf_binary_header *rw_header;
	struct bpf_binary_header *header;
@@ -3320,7 +3421,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
	struct bpf_binary_header *rw_header = NULL;
	struct bpf_binary_header *header = NULL;
	struct bpf_prog *tmp, *orig_prog = prog;
	void __percpu *priv_stack_ptr = NULL;
	struct x64_jit_data *jit_data;
	int priv_stack_alloc_sz;
	int proglen, oldproglen = 0;
	struct jit_context ctx = {};
	bool tmp_blinded = false;
@@ -3356,6 +3459,23 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
		}
		prog->aux->jit_data = jit_data;
	}
	priv_stack_ptr = prog->aux->priv_stack_ptr;
	if (!priv_stack_ptr && prog->aux->jits_use_priv_stack) {
		/* Allocate actual private stack size with verifier-calculated
		 * stack size plus two memory guards to protect overflow and
		 * underflow.
		 */
		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) +
				      2 * PRIV_STACK_GUARD_SZ;
		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL);
		if (!priv_stack_ptr) {
			prog = orig_prog;
			goto out_priv_stack;
		}

		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
		prog->aux->priv_stack_ptr = priv_stack_ptr;
	}
	addrs = jit_data->addrs;
	if (addrs) {
		ctx = jit_data->ctx;
@@ -3491,6 +3611,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
			bpf_prog_fill_jited_linfo(prog, addrs + 1);
out_addrs:
		kvfree(addrs);
		if (!image && priv_stack_ptr) {
			free_percpu(priv_stack_ptr);
			prog->aux->priv_stack_ptr = NULL;
		}
out_priv_stack:
		kfree(jit_data);
		prog->aux->jit_data = NULL;
	}
@@ -3529,6 +3654,8 @@ void bpf_jit_free(struct bpf_prog *prog)
	if (prog->jited) {
		struct x64_jit_data *jit_data = prog->aux->jit_data;
		struct bpf_binary_header *hdr;
		void __percpu *priv_stack_ptr;
		int priv_stack_alloc_sz;

		/*
		 * If we fail the final pass of JIT (from jit_subprogs),
@@ -3544,6 +3671,13 @@ void bpf_jit_free(struct bpf_prog *prog)
		prog->bpf_func = (void *)prog->bpf_func - cfi_get_offset();
		hdr = bpf_jit_binary_pack_hdr(prog);
		bpf_jit_binary_pack_free(hdr, NULL);
		priv_stack_ptr = prog->aux->priv_stack_ptr;
		if (priv_stack_ptr) {
			priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) +
					      2 * PRIV_STACK_GUARD_SZ;
			priv_stack_check_guard(priv_stack_ptr, priv_stack_alloc_sz, prog);
			free_percpu(prog->aux->priv_stack_ptr);
		}
		WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog));
	}

@@ -3559,6 +3693,11 @@ bool bpf_jit_supports_exceptions(void)
	return IS_ENABLED(CONFIG_UNWINDER_ORC);
}

bool bpf_jit_supports_private_stack(void)
{
	return true;
}

void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
{
#if defined(CONFIG_UNWINDER_ORC)
+54 −9
Original line number Diff line number Diff line
@@ -203,6 +203,7 @@ enum btf_field_type {
	BPF_GRAPH_ROOT = BPF_RB_ROOT | BPF_LIST_HEAD,
	BPF_REFCOUNT   = (1 << 9),
	BPF_WORKQUEUE  = (1 << 10),
	BPF_UPTR       = (1 << 11),
};

typedef void (*btf_dtor_kfunc_t)(void *);
@@ -322,6 +323,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)
		return "kptr";
	case BPF_KPTR_PERCPU:
		return "percpu_kptr";
	case BPF_UPTR:
		return "uptr";
	case BPF_LIST_HEAD:
		return "bpf_list_head";
	case BPF_LIST_NODE:
@@ -350,6 +353,7 @@ static inline u32 btf_field_type_size(enum btf_field_type type)
	case BPF_KPTR_UNREF:
	case BPF_KPTR_REF:
	case BPF_KPTR_PERCPU:
	case BPF_UPTR:
		return sizeof(u64);
	case BPF_LIST_HEAD:
		return sizeof(struct bpf_list_head);
@@ -379,6 +383,7 @@ static inline u32 btf_field_type_align(enum btf_field_type type)
	case BPF_KPTR_UNREF:
	case BPF_KPTR_REF:
	case BPF_KPTR_PERCPU:
	case BPF_UPTR:
		return __alignof__(u64);
	case BPF_LIST_HEAD:
		return __alignof__(struct bpf_list_head);
@@ -419,6 +424,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)
	case BPF_KPTR_UNREF:
	case BPF_KPTR_REF:
	case BPF_KPTR_PERCPU:
	case BPF_UPTR:
		break;
	default:
		WARN_ON_ONCE(1);
@@ -507,6 +513,25 @@ static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src
	bpf_obj_memcpy(map->record, dst, src, map->value_size, true);
}

static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src)
{
	unsigned long *src_uptr, *dst_uptr;
	const struct btf_field *field;
	int i;

	if (!btf_record_has_field(rec, BPF_UPTR))
		return;

	for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) {
		if (field->type != BPF_UPTR)
			continue;

		src_uptr = src + field->offset;
		dst_uptr = dst + field->offset;
		swap(*src_uptr, *dst_uptr);
	}
}

static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size)
{
	u32 curr_off = 0;
@@ -907,10 +932,6 @@ enum bpf_reg_type {
	 * additional context, assume the value is non-null.
	 */
	PTR_TO_BTF_ID,
	/* PTR_TO_BTF_ID_OR_NULL points to a kernel struct that has not
	 * been checked for null. Used primarily to inform the verifier
	 * an explicit null check is required for this struct.
	 */
	PTR_TO_MEM,		 /* reg points to valid memory region */
	PTR_TO_ARENA,
	PTR_TO_BUF,		 /* reg points to a read/write buffer */
@@ -923,6 +944,10 @@ enum bpf_reg_type {
	PTR_TO_SOCKET_OR_NULL		= PTR_MAYBE_NULL | PTR_TO_SOCKET,
	PTR_TO_SOCK_COMMON_OR_NULL	= PTR_MAYBE_NULL | PTR_TO_SOCK_COMMON,
	PTR_TO_TCP_SOCK_OR_NULL		= PTR_MAYBE_NULL | PTR_TO_TCP_SOCK,
	/* PTR_TO_BTF_ID_OR_NULL points to a kernel struct that has not
	 * been checked for null. Used primarily to inform the verifier
	 * an explicit null check is required for this struct.
	 */
	PTR_TO_BTF_ID_OR_NULL		= PTR_MAYBE_NULL | PTR_TO_BTF_ID,

	/* This must be the last entry. Its purpose is to ensure the enum is
@@ -1300,8 +1325,12 @@ void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u32 len);
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr);

#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr);
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
			     struct bpf_trampoline *tr,
			     struct bpf_prog *tgt_prog);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
			       struct bpf_trampoline *tr,
			       struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
					  struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
@@ -1373,7 +1402,8 @@ int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_func
void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
				struct bpf_prog *to);
/* Called only from JIT-enabled code, so there's no need for stubs. */
void bpf_image_ksym_add(void *data, unsigned int size, struct bpf_ksym *ksym);
void bpf_image_ksym_init(void *data, unsigned int size, struct bpf_ksym *ksym);
void bpf_image_ksym_add(struct bpf_ksym *ksym);
void bpf_image_ksym_del(struct bpf_ksym *ksym);
void bpf_ksym_add(struct bpf_ksym *ksym);
void bpf_ksym_del(struct bpf_ksym *ksym);
@@ -1382,12 +1412,14 @@ void bpf_jit_uncharge_modmem(u32 size);
bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
					   struct bpf_trampoline *tr)
					   struct bpf_trampoline *tr,
					   struct bpf_prog *tgt_prog)
{
	return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
					     struct bpf_trampoline *tr)
					     struct bpf_trampoline *tr,
					     struct bpf_prog *tgt_prog)
{
	return -ENOTSUPP;
}
@@ -1476,6 +1508,7 @@ struct bpf_prog_aux {
	u32 max_rdwr_access;
	struct btf *attach_btf;
	const struct bpf_ctx_arg_aux *ctx_arg_info;
	void __percpu *priv_stack_ptr;
	struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
	struct bpf_prog *dst_prog;
	struct bpf_trampoline *dst_trampoline;
@@ -1491,7 +1524,13 @@ struct bpf_prog_aux {
	bool xdp_has_frags;
	bool exception_cb;
	bool exception_boundary;
	bool is_extended; /* true if extended by freplace program */
	bool jits_use_priv_stack;
	bool priv_stack_requested;
	u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
	struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
	struct bpf_arena *arena;
	void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
	/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
	const struct btf_type *attach_func_proto;
	/* function name for valid attach_btf_id */
@@ -3461,4 +3500,10 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog)
	return prog->aux->func_idx != 0;
}

static inline bool bpf_prog_is_raw_tp(const struct bpf_prog *prog)
{
	return prog->type == BPF_PROG_TYPE_TRACING &&
	       prog->expected_attach_type == BPF_TRACE_RAW_TP;
}

#endif /* _LINUX_BPF_H */
+9 −3
Original line number Diff line number Diff line
@@ -77,7 +77,13 @@ struct bpf_local_storage_elem {
	struct hlist_node map_node;	/* Linked to bpf_local_storage_map */
	struct hlist_node snode;	/* Linked to bpf_local_storage */
	struct bpf_local_storage __rcu *local_storage;
	union {
		struct rcu_head rcu;
		struct hlist_node free_node;	/* used to postpone
						 * bpf_selem_free
						 * after raw_spin_unlock
						 */
	};
	/* 8 bytes hole */
	/* The data is stored in another cacheline to minimize
	 * the number of cachelines access during a cache hit.
@@ -181,7 +187,7 @@ void bpf_selem_link_map(struct bpf_local_storage_map *smap,

struct bpf_local_storage_elem *
bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value,
		bool charge_mem, gfp_t gfp_flags);
		bool charge_mem, bool swap_uptrs, gfp_t gfp_flags);

void bpf_selem_free(struct bpf_local_storage_elem *selem,
		    struct bpf_local_storage_map *smap,
@@ -195,7 +201,7 @@ bpf_local_storage_alloc(void *owner,

struct bpf_local_storage_data *
bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
			 void *value, u64 map_flags, gfp_t gfp_flags);
			 void *value, u64 map_flags, bool swap_uptrs, gfp_t gfp_flags);

u64 bpf_local_storage_map_mem_usage(const struct bpf_map *map);

Loading