Commit dedd6c89 authored by Alexei Starovoitov's avatar Alexei Starovoitov
Browse files

Merge branch 'exact-states-comparison-for-iterator-convergence-checks' 

Eduard Zingerman says:

====================
exact states comparison for iterator convergence checks

Iterator convergence logic in is_state_visited() uses state_equals()
for states with branches counter > 0 to check if iterator based loop
converges. This is not fully correct because state_equals() relies on
presence of read and precision marks on registers. These marks are not
guaranteed to be finalized while state has branches.
Commit message for patch #3 describes a program that exhibits such
behavior.

This patch-set aims to fix iterator convergence logic by adding notion
of exact states comparison. Exact comparison does not rely on presence
of read or precision marks and thus is more strict.
As explained in commit message for patch #3 exact comparisons require
addition of speculative register bounds widening. The end result for
BPF verifier users could be summarized as follows:

(!) After this update verifier would reject programs that conjure an
    imprecise value on the first loop iteration and use it as precise
    on the second (for iterator based loops).

I urge people to at least skim over the commit message for patch #3.

Patches are organized as follows:
- patches #1,2: moving/extracting utility functions;
- patch #3: introduces exact mode for states comparison and adds
  widening heuristic;
- patch #4: adds test-cases that demonstrate why the series is
  necessary;
- patch #5: extends patch #3 with a notion of state loop entries,
  these entries have to be tracked to correctly identify that
  different verifier states belong to the same states loop;
- patch #6: adds a test-case that demonstrates a program
  which requires loop entry tracking for correct verification;
- patch #7: just adds a few debug prints.

The following actions are planned as a followup for this patch-set:
- implementation has to be adapted for callbacks handling logic as a
  part of a fix for [1];
- it is necessary to explore ways to improve widening heuristic to
  handle iters_task_vma test w/o need to insert barrier_var() calls;
- explored states eviction logic on cache miss has to be extended
  to either:
  - allow eviction of checkpoint states -or-
  - be sped up in case if there are many active checkpoints associated
    with the same instruction.

The patch-set is a followup for mailing list discussion [1].

Changelog:
- V2 [3] -> V3:
  - correct check for stack spills in widen_imprecise_scalars(),
    added test case progs/iters.c:widen_spill to check the behavior
    (suggested by Andrii);
  - allow eviction of checkpoint states in is_state_visited() to avoid
    pathological verifier performance when iterator based loop does not
    converge (discussion with Alexei).
- V1 [2] -> V2, applied changes suggested by Alexei offlist:
  - __explored_state() function removed;
  - same_callsites() function is now used in clean_live_states();
  - patches #1,2 are added as preparatory code movement;
  - in process_iter_next_call() a safeguard is added to verify that
    cur_st->parent exists and has expected insn index / call sites.

[1] https://lore.kernel.org/bpf/97a90da09404c65c8e810cf83c94ac703705dc0e.camel@gmail.com/
[2] https://lore.kernel.org/bpf/20231021005939.1041-1-eddyz87@gmail.com/
[3] https://lore.kernel.org/bpf/20231022010812.9201-1-eddyz87@gmail.com/
====================

Link: https://lore.kernel.org/r/20231024000917.12153-1-eddyz87@gmail.com


Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parents b63dadd6 b4d82395

include/linux/bpf_verifier.h

+16 −0
Original line number Diff line number Diff line
@@ -373,10 +373,25 @@ struct bpf_verifier_state { 
	struct bpf_active_lock active_lock;

	bool speculative;

	bool active_rcu_lock;

	/* If this state was ever pointed-to by other state's loop_entry field

	 * this flag would be set to true. Used to avoid freeing such states

	 * while they are still in use.

	 */

	bool used_as_loop_entry;



	/* first and last insn idx of this verifier state */

	u32 first_insn_idx;

	u32 last_insn_idx;

	/* If this state is a part of states loop this field points to some

	 * parent of this state such that:

	 * - it is also a member of the same states loop;

	 * - DFS states traversal starting from initial state visits loop_entry

	 *   state before this state.

	 * Used to compute topmost loop entry for state loops.

	 * State loops might appear because of open coded iterators logic.

	 * See get_loop_entry() for more information.

	 */

	struct bpf_verifier_state *loop_entry;

	/* jmp history recorded from first to last.

	 * backtracking is using it to go from last to first.

	 * For most states jmp_history_cnt is [0-3].
@@ -384,6 +399,7 @@ struct bpf_verifier_state { 
	 */

	struct bpf_idx_pair *jmp_history;

	u32 jmp_history_cnt;

	u32 dfs_depth;

};



#define bpf_get_spilled_reg(slot, frame, mask)				\

kernel/bpf/verifier.c

+421 −54

File changed.

Preview size limit exceeded, changes collapsed.

tools/testing/selftests/bpf/progs/iters.c

+695 −0
Original line number Diff line number Diff line
@@ -14,6 +14,13 @@ int my_pid; 
int arr[256];

int small_arr[16] SEC(".data.small_arr");



struct {

	__uint(type, BPF_MAP_TYPE_HASH);

	__uint(max_entries, 10);

	__type(key, int);

	__type(value, int);

} amap SEC(".maps");



#ifdef REAL_TEST

#define MY_PID_GUARD() if (my_pid != (bpf_get_current_pid_tgid() >> 32)) return 0

#else
@@ -716,4 +723,692 @@ int iter_pass_iter_ptr_to_subprog(const void *ctx) 
	return 0;

}



SEC("?raw_tp")

__failure

__msg("R1 type=scalar expected=fp")

__naked int delayed_read_mark(void)

{

	/* This is equivalent to C program below.

	 * The call to bpf_iter_num_next() is reachable with r7 values &fp[-16] and 0xdead.

	 * State with r7=&fp[-16] is visited first and follows r6 != 42 ... continue branch.

	 * At this point iterator next() call is reached with r7 that has no read mark.

	 * Loop body with r7=0xdead would only be visited if verifier would decide to continue

	 * with second loop iteration. Absence of read mark on r7 might affect state

	 * equivalent logic used for iterator convergence tracking.

	 *

	 * r7 = &fp[-16]

	 * fp[-16] = 0

	 * r6 = bpf_get_prandom_u32()

	 * bpf_iter_num_new(&fp[-8], 0, 10)

	 * while (bpf_iter_num_next(&fp[-8])) {

	 *   r6++

	 *   if (r6 != 42) {

	 *     r7 = 0xdead

	 *     continue;

	 *   }

	 *   bpf_probe_read_user(r7, 8, 0xdeadbeef); // this is not safe

	 * }

	 * bpf_iter_num_destroy(&fp[-8])

	 * return 0

	 */

	asm volatile (

		"r7 = r10;"

		"r7 += -16;"

		"r0 = 0;"

		"*(u64 *)(r7 + 0) = r0;"

		"call %[bpf_get_prandom_u32];"

		"r6 = r0;"

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

	"1:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto 2f;"

		"r6 += 1;"

		"if r6 != 42 goto 3f;"

		"r7 = 0xdead;"

		"goto 1b;"

	"3:"

		"r1 = r7;"

		"r2 = 8;"

		"r3 = 0xdeadbeef;"

		"call %[bpf_probe_read_user];"

		"goto 1b;"

	"2:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_get_prandom_u32),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy),

		  __imm(bpf_probe_read_user)

		: __clobber_all

	);

}



SEC("?raw_tp")

__failure

__msg("math between fp pointer and register with unbounded")

__naked int delayed_precision_mark(void)

{

	/* This is equivalent to C program below.

	 * The test is similar to delayed_iter_mark but verifies that incomplete

	 * precision don't fool verifier.

	 * The call to bpf_iter_num_next() is reachable with r7 values -16 and -32.

	 * State with r7=-16 is visited first and follows r6 != 42 ... continue branch.

	 * At this point iterator next() call is reached with r7 that has no read

	 * and precision marks.

	 * Loop body with r7=-32 would only be visited if verifier would decide to continue

	 * with second loop iteration. Absence of precision mark on r7 might affect state

	 * equivalent logic used for iterator convergence tracking.

	 *

	 * r8 = 0

	 * fp[-16] = 0

	 * r7 = -16

	 * r6 = bpf_get_prandom_u32()

	 * bpf_iter_num_new(&fp[-8], 0, 10)

	 * while (bpf_iter_num_next(&fp[-8])) {

	 *   if (r6 != 42) {

	 *     r7 = -32

	 *     r6 = bpf_get_prandom_u32()

	 *     continue;

	 *   }

	 *   r0 = r10

	 *   r0 += r7

	 *   r8 = *(u64 *)(r0 + 0)           // this is not safe

	 *   r6 = bpf_get_prandom_u32()

	 * }

	 * bpf_iter_num_destroy(&fp[-8])

	 * return r8

	 */

	asm volatile (

		"r8 = 0;"

		"*(u64 *)(r10 - 16) = r8;"

		"r7 = -16;"

		"call %[bpf_get_prandom_u32];"

		"r6 = r0;"

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

	"1:"

		"r1 = r10;"

		"r1 += -8;\n"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto 2f;"

		"if r6 != 42 goto 3f;"

		"r7 = -32;"

		"call %[bpf_get_prandom_u32];"

		"r6 = r0;"

		"goto 1b;\n"

	"3:"

		"r0 = r10;"

		"r0 += r7;"

		"r8 = *(u64 *)(r0 + 0);"

		"call %[bpf_get_prandom_u32];"

		"r6 = r0;"

		"goto 1b;\n"

	"2:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r0 = r8;"

		"exit;"

		:

		: __imm(bpf_get_prandom_u32),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy),

		  __imm(bpf_probe_read_user)

		: __clobber_all

	);

}



SEC("?raw_tp")

__failure

__msg("math between fp pointer and register with unbounded")

__flag(BPF_F_TEST_STATE_FREQ)

__naked int loop_state_deps1(void)

{

	/* This is equivalent to C program below.

	 *

	 * The case turns out to be tricky in a sense that:

	 * - states with c=-25 are explored only on a second iteration

	 *   of the outer loop;

	 * - states with read+precise mark on c are explored only on

	 *   second iteration of the inner loop and in a state which

	 *   is pushed to states stack first.

	 *

	 * Depending on the details of iterator convergence logic

	 * verifier might stop states traversal too early and miss

	 * unsafe c=-25 memory access.

	 *

	 *   j = iter_new();		 // fp[-16]

	 *   a = 0;			 // r6

	 *   b = 0;			 // r7

	 *   c = -24;			 // r8

	 *   while (iter_next(j)) {

	 *     i = iter_new();		 // fp[-8]

	 *     a = 0;			 // r6

	 *     b = 0;			 // r7

	 *     while (iter_next(i)) {

	 *	 if (a == 1) {

	 *	   a = 0;

	 *	   b = 1;

	 *	 } else if (a == 0) {

	 *	   a = 1;

	 *	   if (random() == 42)

	 *	     continue;

	 *	   if (b == 1) {

	 *	     *(r10 + c) = 7;  // this is not safe

	 *	     iter_destroy(i);

	 *	     iter_destroy(j);

	 *	     return;

	 *	   }

	 *	 }

	 *     }

	 *     iter_destroy(i);

	 *     a = 0;

	 *     b = 0;

	 *     c = -25;

	 *   }

	 *   iter_destroy(j);

	 *   return;

	 */

	asm volatile (

		"r1 = r10;"

		"r1 += -16;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

		"r6 = 0;"

		"r7 = 0;"

		"r8 = -24;"

	"j_loop_%=:"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto j_loop_end_%=;"

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

		"r6 = 0;"

		"r7 = 0;"

	"i_loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto i_loop_end_%=;"

	"check_one_r6_%=:"

		"if r6 != 1 goto check_zero_r6_%=;"

		"r6 = 0;"

		"r7 = 1;"

		"goto i_loop_%=;"

	"check_zero_r6_%=:"

		"if r6 != 0 goto i_loop_%=;"

		"r6 = 1;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto check_one_r7_%=;"

		"goto i_loop_%=;"

	"check_one_r7_%=:"

		"if r7 != 1 goto i_loop_%=;"

		"r0 = r10;"

		"r0 += r8;"

		"r1 = 7;"

		"*(u64 *)(r0 + 0) = r1;"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

	"i_loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r6 = 0;"

		"r7 = 0;"

		"r8 = -25;"

		"goto j_loop_%=;"

	"j_loop_end_%=:"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_get_prandom_u32),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy)

		: __clobber_all

	);

}



SEC("?raw_tp")

__failure

__msg("math between fp pointer and register with unbounded")

__flag(BPF_F_TEST_STATE_FREQ)

__naked int loop_state_deps2(void)

{

	/* This is equivalent to C program below.

	 *

	 * The case turns out to be tricky in a sense that:

	 * - states with read+precise mark on c are explored only on a second

	 *   iteration of the first inner loop and in a state which is pushed to

	 *   states stack first.

	 * - states with c=-25 are explored only on a second iteration of the

	 *   second inner loop and in a state which is pushed to states stack

	 *   first.

	 *

	 * Depending on the details of iterator convergence logic

	 * verifier might stop states traversal too early and miss

	 * unsafe c=-25 memory access.

	 *

	 *   j = iter_new();             // fp[-16]

	 *   a = 0;                      // r6

	 *   b = 0;                      // r7

	 *   c = -24;                    // r8

	 *   while (iter_next(j)) {

	 *     i = iter_new();           // fp[-8]

	 *     a = 0;                    // r6

	 *     b = 0;                    // r7

	 *     while (iter_next(i)) {

	 *       if (a == 1) {

	 *         a = 0;

	 *         b = 1;

	 *       } else if (a == 0) {

	 *         a = 1;

	 *         if (random() == 42)

	 *           continue;

	 *         if (b == 1) {

	 *           *(r10 + c) = 7;     // this is not safe

	 *           iter_destroy(i);

	 *           iter_destroy(j);

	 *           return;

	 *         }

	 *       }

	 *     }

	 *     iter_destroy(i);

	 *     i = iter_new();           // fp[-8]

	 *     a = 0;                    // r6

	 *     b = 0;                    // r7

	 *     while (iter_next(i)) {

	 *       if (a == 1) {

	 *         a = 0;

	 *         b = 1;

	 *       } else if (a == 0) {

	 *         a = 1;

	 *         if (random() == 42)

	 *           continue;

	 *         if (b == 1) {

	 *           a = 0;

	 *           c = -25;

	 *         }

	 *       }

	 *     }

	 *     iter_destroy(i);

	 *   }

	 *   iter_destroy(j);

	 *   return;

	 */

	asm volatile (

		"r1 = r10;"

		"r1 += -16;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

		"r6 = 0;"

		"r7 = 0;"

		"r8 = -24;"

	"j_loop_%=:"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto j_loop_end_%=;"



		/* first inner loop */

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

		"r6 = 0;"

		"r7 = 0;"

	"i_loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto i_loop_end_%=;"

	"check_one_r6_%=:"

		"if r6 != 1 goto check_zero_r6_%=;"

		"r6 = 0;"

		"r7 = 1;"

		"goto i_loop_%=;"

	"check_zero_r6_%=:"

		"if r6 != 0 goto i_loop_%=;"

		"r6 = 1;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto check_one_r7_%=;"

		"goto i_loop_%=;"

	"check_one_r7_%=:"

		"if r7 != 1 goto i_loop_%=;"

		"r0 = r10;"

		"r0 += r8;"

		"r1 = 7;"

		"*(u64 *)(r0 + 0) = r1;"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

	"i_loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"



		/* second inner loop */

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

		"r6 = 0;"

		"r7 = 0;"

	"i2_loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto i2_loop_end_%=;"

	"check2_one_r6_%=:"

		"if r6 != 1 goto check2_zero_r6_%=;"

		"r6 = 0;"

		"r7 = 1;"

		"goto i2_loop_%=;"

	"check2_zero_r6_%=:"

		"if r6 != 0 goto i2_loop_%=;"

		"r6 = 1;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto check2_one_r7_%=;"

		"goto i2_loop_%=;"

	"check2_one_r7_%=:"

		"if r7 != 1 goto i2_loop_%=;"

		"r6 = 0;"

		"r8 = -25;"

		"goto i2_loop_%=;"

	"i2_loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"



		"r6 = 0;"

		"r7 = 0;"

		"goto j_loop_%=;"

	"j_loop_end_%=:"

		"r1 = r10;"

		"r1 += -16;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_get_prandom_u32),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy)

		: __clobber_all

	);

}



SEC("?raw_tp")

__success

__naked int triple_continue(void)

{

	/* This is equivalent to C program below.

	 * High branching factor of the loop body turned out to be

	 * problematic for one of the iterator convergence tracking

	 * algorithms explored.

	 *

	 * r6 = bpf_get_prandom_u32()

	 * bpf_iter_num_new(&fp[-8], 0, 10)

	 * while (bpf_iter_num_next(&fp[-8])) {

	 *   if (bpf_get_prandom_u32() != 42)

	 *     continue;

	 *   if (bpf_get_prandom_u32() != 42)

	 *     continue;

	 *   if (bpf_get_prandom_u32() != 42)

	 *     continue;

	 *   r0 += 0;

	 * }

	 * bpf_iter_num_destroy(&fp[-8])

	 * return 0

	 */

	asm volatile (

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

	"loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto loop_end_%=;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto loop_%=;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto loop_%=;"

		"call %[bpf_get_prandom_u32];"

		"if r0 != 42 goto loop_%=;"

		"r0 += 0;"

		"goto loop_%=;"

	"loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_get_prandom_u32),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy)

		: __clobber_all

	);

}



SEC("?raw_tp")

__success

__naked int widen_spill(void)

{

	/* This is equivalent to C program below.

	 * The counter is stored in fp[-16], if this counter is not widened

	 * verifier states representing loop iterations would never converge.

	 *

	 * fp[-16] = 0

	 * bpf_iter_num_new(&fp[-8], 0, 10)

	 * while (bpf_iter_num_next(&fp[-8])) {

	 *   r0 = fp[-16];

	 *   r0 += 1;

	 *   fp[-16] = r0;

	 * }

	 * bpf_iter_num_destroy(&fp[-8])

	 * return 0

	 */

	asm volatile (

		"r0 = 0;"

		"*(u64 *)(r10 - 16) = r0;"

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

	"loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto loop_end_%=;"

		"r0 = *(u64 *)(r10 - 16);"

		"r0 += 1;"

		"*(u64 *)(r10 - 16) = r0;"

		"goto loop_%=;"

	"loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy)

		: __clobber_all

	);

}



SEC("raw_tp")

__success

__naked int checkpoint_states_deletion(void)

{

	/* This is equivalent to C program below.

	 *

	 *   int *a, *b, *c, *d, *e, *f;

	 *   int i, sum = 0;

	 *   bpf_for(i, 0, 10) {

	 *     a = bpf_map_lookup_elem(&amap, &i);

	 *     b = bpf_map_lookup_elem(&amap, &i);

	 *     c = bpf_map_lookup_elem(&amap, &i);

	 *     d = bpf_map_lookup_elem(&amap, &i);

	 *     e = bpf_map_lookup_elem(&amap, &i);

	 *     f = bpf_map_lookup_elem(&amap, &i);

	 *     if (a) sum += 1;

	 *     if (b) sum += 1;

	 *     if (c) sum += 1;

	 *     if (d) sum += 1;

	 *     if (e) sum += 1;

	 *     if (f) sum += 1;

	 *   }

	 *   return 0;

	 *

	 * The body of the loop spawns multiple simulation paths

	 * with different combination of NULL/non-NULL information for a/b/c/d/e/f.

	 * Each combination is unique from states_equal() point of view.

	 * Explored states checkpoint is created after each iterator next call.

	 * Iterator convergence logic expects that eventually current state

	 * would get equal to one of the explored states and thus loop

	 * exploration would be finished (at-least for a specific path).

	 * Verifier evicts explored states with high miss to hit ratio

	 * to to avoid comparing current state with too many explored

	 * states per instruction.

	 * This test is designed to "stress test" eviction policy defined using formula:

	 *

	 *    sl->miss_cnt > sl->hit_cnt * N + N // if true sl->state is evicted

	 *

	 * Currently N is set to 64, which allows for 6 variables in this test.

	 */

	asm volatile (

		"r6 = 0;"                  /* a */

		"r7 = 0;"                  /* b */

		"r8 = 0;"                  /* c */

		"*(u64 *)(r10 - 24) = r6;" /* d */

		"*(u64 *)(r10 - 32) = r6;" /* e */

		"*(u64 *)(r10 - 40) = r6;" /* f */

		"r9 = 0;"                  /* sum */

		"r1 = r10;"

		"r1 += -8;"

		"r2 = 0;"

		"r3 = 10;"

		"call %[bpf_iter_num_new];"

	"loop_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_next];"

		"if r0 == 0 goto loop_end_%=;"



		"*(u64 *)(r10 - 16) = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"r6 = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"r7 = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"r8 = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"*(u64 *)(r10 - 24) = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"*(u64 *)(r10 - 32) = r0;"



		"r1 = %[amap] ll;"

		"r2 = r10;"

		"r2 += -16;"

		"call %[bpf_map_lookup_elem];"

		"*(u64 *)(r10 - 40) = r0;"



		"if r6 == 0 goto +1;"

		"r9 += 1;"

		"if r7 == 0 goto +1;"

		"r9 += 1;"

		"if r8 == 0 goto +1;"

		"r9 += 1;"

		"r0 = *(u64 *)(r10 - 24);"

		"if r0 == 0 goto +1;"

		"r9 += 1;"

		"r0 = *(u64 *)(r10 - 32);"

		"if r0 == 0 goto +1;"

		"r9 += 1;"

		"r0 = *(u64 *)(r10 - 40);"

		"if r0 == 0 goto +1;"

		"r9 += 1;"



		"goto loop_%=;"

	"loop_end_%=:"

		"r1 = r10;"

		"r1 += -8;"

		"call %[bpf_iter_num_destroy];"

		"r0 = 0;"

		"exit;"

		:

		: __imm(bpf_map_lookup_elem),

		  __imm(bpf_iter_num_new),

		  __imm(bpf_iter_num_next),

		  __imm(bpf_iter_num_destroy),

		  __imm_addr(amap)

		: __clobber_all

	);

}



char _license[] SEC("license") = "GPL";

tools/testing/selftests/bpf/progs/iters_task_vma.c

+1 −0
Original line number Diff line number Diff line
@@ -30,6 +30,7 @@ int iter_task_vma_for_each(const void *ctx) 
	bpf_for_each(task_vma, vma, task, 0) {

		if (seen >= 1000)

			break;

		barrier_var(seen);



		vm_ranges[seen].vm_start = vma->vm_start;

		vm_ranges[seen].vm_end = vma->vm_end;