Merge tag 'bpf-next-6.16' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

BPF Iterators

=============

--------

Overview

--------

BPF supports two separate entities collectively known as "BPF iterators": BPF

iterator *program type* and *open-coded* BPF iterators. The former is

a stand-alone BPF program type which, when attached and activated by user,

will be called once for each entity (task_struct, cgroup, etc) that is being

iterated. The latter is a set of BPF-side APIs implementing iterator

functionality and available across multiple BPF program types. Open-coded

iterators provide similar functionality to BPF iterator programs, but gives

more flexibility and control to all other BPF program types. BPF iterator

programs, on the other hand, can be used to implement anonymous or BPF

FS-mounted special files, whose contents are generated by attached BPF iterator

program, backed by seq_file functionality. Both are useful depending on

specific needs.

When adding a new BPF iterator program, it is expected that similar

functionality will be added as open-coded iterator for maximum flexibility.

It's also expected that iteration logic and code will be maximally shared and

reused between two iterator API surfaces.

----------

Motivation

----------

------------------------

Open-coded BPF Iterators

------------------------

Open-coded BPF iterators are implemented as tightly-coupled trios of kfuncs

(constructor, next element fetch, destructor) and iterator-specific type

describing on-the-stack iterator state, which is guaranteed by the BPF

verifier to not be tampered with outside of the corresponding

constructor/destructor/next APIs.

Each kind of open-coded BPF iterator has its own associated

struct bpf_iter_<type>, where <type> denotes a specific type of iterator.

bpf_iter_<type> state needs to live on BPF program stack, so make sure it's

small enough to fit on BPF stack. For performance reasons its best to avoid

dynamic memory allocation for iterator state and size the state struct big

enough to fit everything necessary. But if necessary, dynamic memory

allocation is a way to bypass BPF stack limitations. Note, state struct size

is part of iterator's user-visible API, so changing it will break backwards

compatibility, so be deliberate about designing it.

All kfuncs (constructor, next, destructor) have to be named consistently as

bpf_iter_<type>_{new,next,destroy}(), respectively. <type> represents iterator

type, and iterator state should be represented as a matching

`struct bpf_iter_<type>` state type. Also, all iter kfuncs should have

a pointer to this `struct bpf_iter_<type>` as the very first argument.

Additionally:

  - Constructor, i.e., `bpf_iter_<type>_new()`, can have arbitrary extra

    number of arguments. Return type is not enforced either.

  - Next method, i.e., `bpf_iter_<type>_next()`, has to return a pointer

    type and should have exactly one argument: `struct bpf_iter_<type> *`

    (const/volatile/restrict and typedefs are ignored).

  - Destructor, i.e., `bpf_iter_<type>_destroy()`, should return void and

    should have exactly one argument, similar to the next method.

  - `struct bpf_iter_<type>` size is enforced to be positive and

    a multiple of 8 bytes (to fit stack slots correctly).

Such strictness and consistency allows to build generic helpers abstracting

important, but boilerplate, details to be able to use open-coded iterators

effectively and ergonomically (see libbpf's bpf_for_each() macro). This is

enforced at kfunc registration point by the kernel.

Constructor/next/destructor implementation contract is as follows:

  - constructor, `bpf_iter_<type>_new()`, always initializes iterator state on

    the stack. If any of the input arguments are invalid, constructor should

    make sure to still initialize it such that subsequent next() calls will

    return NULL. I.e., on error, *return error and construct empty iterator*.

    Constructor kfunc is marked with KF_ITER_NEW flag.

  - next method, `bpf_iter_<type>_next()`, accepts pointer to iterator state

    and produces an element. Next method should always return a pointer. The

    contract between BPF verifier is that next method *guarantees* that it

    will eventually return NULL when elements are exhausted. Once NULL is

    returned, subsequent next calls *should keep returning NULL*. Next method

    is marked with KF_ITER_NEXT (and should also have KF_RET_NULL as

    NULL-returning kfunc, of course).

  - destructor, `bpf_iter_<type>_destroy()`, is always called once. Even if

    constructor failed or next returned nothing.  Destructor frees up any

    resources and marks stack space used by `struct bpf_iter_<type>` as usable

    for something else. Destructor is marked with KF_ITER_DESTROY flag.

Any open-coded BPF iterator implementation has to implement at least these

three methods. It is enforced that for any given type of iterator only

applicable constructor/destructor/next are callable. I.e., verifier ensures

you can't pass number iterator state into, say, cgroup iterator's next method.

From a 10,000-feet BPF verification point of view, next methods are the points

of forking a verification state, which are conceptually similar to what

verifier is doing when validating conditional jumps. Verifier is branching out

`call bpf_iter_<type>_next` instruction and simulates two outcomes: NULL

(iteration is done) and non-NULL (new element is returned). NULL is simulated

first and is supposed to reach exit without looping. After that non-NULL case

is validated and it either reaches exit (for trivial examples with no real

loop), or reaches another `call bpf_iter_<type>_next` instruction with the

state equivalent to already (partially) validated one. State equivalency at

that point means we technically are going to be looping forever without

"breaking out" out of established "state envelope" (i.e., subsequent

iterations don't add any new knowledge or constraints to the verifier state,

so running 1, 2, 10, or a million of them doesn't matter). But taking into

account the contract stating that iterator next method *has to* return NULL

eventually, we can conclude that loop body is safe and will eventually

terminate. Given we validated logic outside of the loop (NULL case), and

concluded that loop body is safe (though potentially looping many times),

verifier can claim safety of the overall program logic.

------------------------

BPF Iterators Motivation

------------------------

There are a few existing ways to dump kernel data into user space. The most

popular one is the ``/proc`` system. For example, ``cat /proc/net/tcp6`` dumps

::

  link = bpf_program__attach_iter(prog, &opts); iter_fd =

  bpf_iter_create(bpf_link__fd(link));

  link = bpf_program__attach_iter(prog, &opts);

  iter_fd = bpf_iter_create(bpf_link__fd(link));

If both *tid* and *pid* are zero, an iterator created from this struct

``bpf_iter_attach_opts`` will include every opened file of every task in the

                ...

        }

2.2.6 __prog Annotation

---------------------------

This annotation is used to indicate that the argument needs to be fixed up to

the bpf_prog_aux of the caller BPF program. Any value passed into this argument

is ignored, and rewritten by the verifier.

An example is given below::

        __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,

                                                 int (callback_fn)(void *map, int *key, void *value),

                                                 unsigned int flags,

                                                 void *aux__prog)

         {

                struct bpf_prog_aux *aux = aux__prog;

                ...

         }

.. _BPF_kfunc_nodef:

2.3 Using an existing kernel function

}

static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,

			    int args_off, int retval_off, int run_ctx_off,

			    int bargs_off, int retval_off, int run_ctx_off,

			    bool save_ret)

{

	__le32 *branch;

	branch = ctx->image + ctx->idx;

	emit(A64_NOP, ctx);

	emit(A64_ADD_I(1, A64_R(0), A64_SP, args_off), ctx);

	emit(A64_ADD_I(1, A64_R(0), A64_SP, bargs_off), ctx);

	if (!p->jited)

		emit_addr_mov_i64(A64_R(1), (const u64)p->insnsi, ctx);

}

static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,

			       int args_off, int retval_off, int run_ctx_off,

			       int bargs_off, int retval_off, int run_ctx_off,

			       __le32 **branches)

{

	int i;

	 */

	emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);

	for (i = 0; i < tl->nr_links; i++) {

		invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off,

		invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off,

				run_ctx_off, true);

		/* if (*(u64 *)(sp + retval_off) !=  0)

		 *	goto do_fexit;

	}

}

static void save_args(struct jit_ctx *ctx, int args_off, int nregs)

struct arg_aux {

	/* how many args are passed through registers, the rest of the args are

	 * passed through stack

	 */

	int args_in_regs;

	/* how many registers are used to pass arguments */

	int regs_for_args;

	/* how much stack is used for additional args passed to bpf program

	 * that did not fit in original function registers

	 */

	int bstack_for_args;

	/* home much stack is used for additional args passed to the

	 * original function when called from trampoline (this one needs

	 * arguments to be properly aligned)

	 */

	int ostack_for_args;

};

static int calc_arg_aux(const struct btf_func_model *m,

			 struct arg_aux *a)

{

	int i;

	int stack_slots, nregs, slots, i;

	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */

	for (i = 0, nregs = 0; i < m->nr_args; i++) {

		slots = (m->arg_size[i] + 7) / 8;

		if (nregs + slots <= 8) /* passed through register ? */

			nregs += slots;

		else

			break;

	}

	a->args_in_regs = i;

	a->regs_for_args = nregs;

	a->ostack_for_args = 0;

	a->bstack_for_args = 0;

	/* the rest arguments are passed through stack */

	for (; i < m->nr_args; i++) {

		/* We can not know for sure about exact alignment needs for

		 * struct passed on stack, so deny those

		 */

		if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)

			return -ENOTSUPP;

		stack_slots = (m->arg_size[i] + 7) / 8;

		a->bstack_for_args += stack_slots * 8;

		a->ostack_for_args = a->ostack_for_args + stack_slots * 8;

	}

	for (i = 0; i < nregs; i++) {

		emit(A64_STR64I(i, A64_SP, args_off), ctx);

		args_off += 8;

	return 0;

}

static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes)

{

	if (effective_bytes) {

		int garbage_bits = 64 - 8 * effective_bytes;

#ifdef CONFIG_CPU_BIG_ENDIAN

		/* garbage bits are at the right end */

		emit(A64_LSR(1, reg, reg, garbage_bits), ctx);

		emit(A64_LSL(1, reg, reg, garbage_bits), ctx);

#else

		/* garbage bits are at the left end */

		emit(A64_LSL(1, reg, reg, garbage_bits), ctx);

		emit(A64_LSR(1, reg, reg, garbage_bits), ctx);

#endif

	}

}

static void restore_args(struct jit_ctx *ctx, int args_off, int nregs)

static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,

		      const struct btf_func_model *m,

		      const struct arg_aux *a,

		      bool for_call_origin)

{

	int i;

	int reg;

	int doff;

	int soff;

	int slots;

	u8 tmp = bpf2a64[TMP_REG_1];

	/* store arguments to the stack for the bpf program, or restore

	 * arguments from stack for the original function

	 */

	for (reg = 0; reg < a->regs_for_args; reg++) {

		emit(for_call_origin ?

		     A64_LDR64I(reg, A64_SP, bargs_off) :

		     A64_STR64I(reg, A64_SP, bargs_off),

		     ctx);

		bargs_off += 8;

	}

	soff = 32; /* on stack arguments start from FP + 32 */

	doff = (for_call_origin ? oargs_off : bargs_off);

	/* save on stack arguments */

	for (i = a->args_in_regs; i < m->nr_args; i++) {

		slots = (m->arg_size[i] + 7) / 8;

		/* verifier ensures arg_size <= 16, so slots equals 1 or 2 */

		while (slots-- > 0) {

			emit(A64_LDR64I(tmp, A64_FP, soff), ctx);

			/* if there is unused space in the last slot, clear

			 * the garbage contained in the space.

			 */

			if (slots == 0 && !for_call_origin)

				clear_garbage(ctx, tmp, m->arg_size[i] % 8);

			emit(A64_STR64I(tmp, A64_SP, doff), ctx);

			soff += 8;

			doff += 8;

		}

	}

}

	for (i = 0; i < nregs; i++) {

		emit(A64_LDR64I(i, A64_SP, args_off), ctx);

		args_off += 8;

static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs)

{

	int reg;

	for (reg = 0; reg < nregs; reg++) {

		emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);

		bargs_off += 8;

	}

}

 */

static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,

			      struct bpf_tramp_links *tlinks, void *func_addr,

			      int nregs, u32 flags)

			      const struct btf_func_model *m,

			      const struct arg_aux *a,

			      u32 flags)

{

	int i;

	int stack_size;

	int retaddr_off;

	int regs_off;

	int retval_off;

	int args_off;

	int nregs_off;

	int bargs_off;

	int nfuncargs_off;

	int ip_off;

	int run_ctx_off;

	int oargs_off;

	int nfuncargs;

	struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];

	struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];

	struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];

	 *

	 * SP + retval_off    [ return value      ] BPF_TRAMP_F_CALL_ORIG or

	 *                                          BPF_TRAMP_F_RET_FENTRY_RET

	 *

	 *                    [ arg reg N         ]

	 *                    [ ...               ]

	 * SP + args_off    [ arg reg 1         ]

	 * SP + bargs_off     [ arg reg 1         ] for bpf

	 *

	 * SP + nregs_off   [ arg regs count    ]

	 * SP + nfuncargs_off [ arg regs count    ]

	 *

	 * SP + ip_off        [ traced function   ] BPF_TRAMP_F_IP_ARG flag

	 *

	 * SP + run_ctx_off   [ bpf_tramp_run_ctx ]

	 *

	 *                    [ stack arg N       ]

	 *                    [ ...               ]

	 * SP + oargs_off     [ stack arg 1       ] for original func

	 */

	stack_size = 0;

	oargs_off = stack_size;

	if (flags & BPF_TRAMP_F_CALL_ORIG)

		stack_size +=  a->ostack_for_args;

	run_ctx_off = stack_size;

	/* room for bpf_tramp_run_ctx */

	stack_size += round_up(sizeof(struct bpf_tramp_run_ctx), 8);

	if (flags & BPF_TRAMP_F_IP_ARG)

		stack_size += 8;

	nregs_off = stack_size;

	nfuncargs_off = stack_size;

	/* room for args count */

	stack_size += 8;

	args_off = stack_size;

	bargs_off = stack_size;

	/* room for args */

	stack_size += nregs * 8;

	nfuncargs = a->regs_for_args + a->bstack_for_args / 8;

	stack_size += 8 * nfuncargs;

	/* room for return value */

	retval_off = stack_size;

	}

	/* save arg regs count*/

	emit(A64_MOVZ(1, A64_R(10), nregs, 0), ctx);

	emit(A64_STR64I(A64_R(10), A64_SP, nregs_off), ctx);

	emit(A64_MOVZ(1, A64_R(10), nfuncargs, 0), ctx);

	emit(A64_STR64I(A64_R(10), A64_SP, nfuncargs_off), ctx);

	/* save arg regs */

	save_args(ctx, args_off, nregs);

	/* save args for bpf */

	save_args(ctx, bargs_off, oargs_off, m, a, false);

	/* save callee saved registers */

	emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx);

	}

	for (i = 0; i < fentry->nr_links; i++)

		invoke_bpf_prog(ctx, fentry->links[i], args_off,

		invoke_bpf_prog(ctx, fentry->links[i], bargs_off,

				retval_off, run_ctx_off,

				flags & BPF_TRAMP_F_RET_FENTRY_RET);

		if (!branches)

			return -ENOMEM;

		invoke_bpf_mod_ret(ctx, fmod_ret, args_off, retval_off,

		invoke_bpf_mod_ret(ctx, fmod_ret, bargs_off, retval_off,

				   run_ctx_off, branches);

	}

	if (flags & BPF_TRAMP_F_CALL_ORIG) {

		restore_args(ctx, args_off, nregs);

		/* save args for original func */

		save_args(ctx, bargs_off, oargs_off, m, a, true);

		/* call original func */

		emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx);

		emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx);

	}

	for (i = 0; i < fexit->nr_links; i++)

		invoke_bpf_prog(ctx, fexit->links[i], args_off, retval_off,

		invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off,

				run_ctx_off, false);

	if (flags & BPF_TRAMP_F_CALL_ORIG) {

	}

	if (flags & BPF_TRAMP_F_RESTORE_REGS)

		restore_args(ctx, args_off, nregs);

		restore_args(ctx, bargs_off, a->regs_for_args);

	/* restore callee saved register x19 and x20 */

	emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx);

	return ctx->idx;

}

static int btf_func_model_nregs(const struct btf_func_model *m)

{

	int nregs = m->nr_args;

	int i;

	/* extra registers needed for struct argument */

	for (i = 0; i < MAX_BPF_FUNC_ARGS; i++) {

		/* The arg_size is at most 16 bytes, enforced by the verifier. */

		if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)

			nregs += (m->arg_size[i] + 7) / 8 - 1;

	}

	return nregs;

}

int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,

			     struct bpf_tramp_links *tlinks, void *func_addr)

{

		.idx = 0,

	};

	struct bpf_tramp_image im;

	int nregs, ret;

	struct arg_aux aaux;

	int ret;

	nregs = btf_func_model_nregs(m);

	/* the first 8 registers are used for arguments */

	if (nregs > 8)

		return -ENOTSUPP;

	ret = calc_arg_aux(m, &aaux);

	if (ret < 0)

		return ret;

	ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, nregs, flags);

	ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags);

	if (ret < 0)

		return ret;

				u32 flags, struct bpf_tramp_links *tlinks,

				void *func_addr)

{

	int ret, nregs;

	void *image, *tmp;

	u32 size = ro_image_end - ro_image;

	struct arg_aux aaux;

	void *image, *tmp;

	int ret;

	/* image doesn't need to be in module memory range, so we can

	 * use kvmalloc.

		.write = true,

	};

	nregs = btf_func_model_nregs(m);

	/* the first 8 registers are used for arguments */

	if (nregs > 8)

		return -ENOTSUPP;

	jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));

	ret = prepare_trampoline(&ctx, im, tlinks, func_addr, nregs, flags);

	ret = calc_arg_aux(m, &aaux);

	if (ret)

		goto out;

	ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags);

	if (ret > 0 && validate_code(&ctx) < 0) {

		ret = -EINVAL;

	return rv_i_insn(imm11_0, 0, 0, 0, 0xf);

}

static inline void emit_fence_r_rw(struct rv_jit_context *ctx)

{

	emit(rv_fence(0x2, 0x3), ctx);

}

static inline void emit_fence_rw_w(struct rv_jit_context *ctx)

{

	emit(rv_fence(0x3, 0x1), ctx);

}

static inline void emit_fence_rw_rw(struct rv_jit_context *ctx)

{

	emit(rv_fence(0x3, 0x3), ctx);

}

static inline u32 rv_nop(void)

{

	return rv_i_insn(0, 0, 0, 0, 0x13);