Commit b9da1739 authored by Andrii Nakryiko's avatar Andrii Nakryiko
Browse files

Merge branch 'improve-the-performance-of-btf-type-lookups-with-binary-search' 

Donglin Peng says:

====================
Improve the performance of BTF type lookups with binary search

From: Donglin Peng <pengdonglin@xiaomi.com>

The series addresses the performance limitations of linear search in large
BTFs by:
1. Adding BTF permutation support
2. Using resolve_btfids to sort BTF during the build phase
3. Checking BTF sorting
4. Using binary search when looking up types

Patch #1 introduces an interface for btf__permute in libbpf to relay out BTF.
Patch #2 adds test cases to validate the functionality of btf__permute in base
and split BTF scenarios.
Patch #3 introduces a new phase in the resolve_btfids tool to sort BTF by name
in ascending order.
Patches #4-#7 implement the sorting check and binary search.
Patches #8-#10 optimize type lookup performance of some functions by skipping
anonymous types or invoking btf_find_by_name_kind.
Patch #11 refactors the code by calling str_is_empty.

Here is a simple performance test result [1] for lookups to find 87,584 named
types in vmlinux BTF:

./vmtest.sh -- ./test_progs -t btf_permute/perf -v

Results:
| Condition          | Lookup Time | Improvement  |
|--------------------|-------------|--------------|
| Unsorted (Linear)  |  36,534 ms  | Baseline     |
| Sorted (Binary)    |      15 ms  | 2437x faster |

The binary search implementation reduces lookup time from 36.5 seconds to 15
milliseconds, achieving a **2437x** speedup for large-scale type queries.

Changelog:
v12:
- Set the start_id to 1 instead of btf->start_id in the btf__find_by_name (AI)

v11:
- Link: https://lore.kernel.org/bpf/20260108031645.1350069-1-dolinux.peng@gmail.com/
- PATCH #1: Modify implementation of btf__permute: id_map[0] must be 0 for base BTF (Andrii)
- PATCH #3: Refactor the code (Andrii)
- PATCH #4~8:
  - Revert to using the binary search in v7 to simplify the code (Andrii)
  - Refactor the code of btf_check_sorted (Andrii, Eduard)
  - Rename sorted_start_id to named_start_id
  - Rename btf_sorted_start_id to btf_named_start_id, and add comments (Andrii, Eduard)

v10:
- Link: https://lore.kernel.org/all/20251218113051.455293-1-dolinux.peng@gmail.com/
- Improve btf__permute() documentation (Eduard)
- Fall back to linear search when locating anonymous types (Eduard)
- Remove redundant NULL name check in libbpf's linear search path (Eduard)
- Simplify btf_check_sorted() implementation (Eduard)
- Treat kernel modules as unsorted by default
- Introduce btf_is_sorted and btf_sorted_start_id for clarity (Eduard)
- Fix optimizations in btf_find_decl_tag_value() and btf_prepare_func_args()
  to support split BTF
- Remove linear search branch in determine_ptr_size()
- Rebase onto Ihor's v4 patch series [4]

v9:
- Link: https://lore.kernel.org/bpf/20251208062353.1702672-1-dolinux.peng@gmail.com/
- Optimize the performance of the function determine_ptr_size by invoking
  btf__find_by_name_kind
- Optimize the performance of btf_find_decl_tag_value/btf_prepare_func_args/
  bpf_core_add_cands by skipping anonymous types
- Rebase the patch series onto Ihor's v3 patch series [3]

v8
- Link: https://lore.kernel.org/bpf/20251126085025.784288-1-dolinux.peng@gmail.com/
- Remove the type dropping feature of btf__permute (Andrii)
- Refactor the code of btf__permute (Andrii, Eduard)
- Make the self-test code cleaner (Eduard)
- Reconstruct the BTF sorting patch based on Ihor's patch series [2]
- Simplify the sorting logic and place anonymous types before named types
  (Andrii, Eduard)
- Optimize type lookup performance of two kernel functions
- Refactoring the binary search and type lookup logic achieves a 4.2%
  performance gain, reducing the average lookup time (via the perf test
  code in [1] for 60,995 named types in vmlinux BTF) from 10,217 us (v7) to
  9,783 us (v8).

v7:
- Link: https://lore.kernel.org/all/20251119031531.1817099-1-dolinux.peng@gmail.com/
- btf__permute API refinement: Adjusted id_map and id_map_cnt parameter
  usage so that for base BTF, id_map[0] now contains the new id of original
  type id 1 (instead of VOID type id 0), improving logical consistency
- Selftest updates: Modified test cases to align with the API usage changes
- Refactor the code of resolve_btfids

v6:
- Link: https://lore.kernel.org/all/20251117132623.3807094-1-dolinux.peng@gmail.com/
- ID Map-based reimplementation of btf__permute (Andrii)
- Build-time BTF sorting using resolve_btfids (Alexei, Eduard)
- Binary search method refactoring (Andrii)
- Enhanced selftest coverage

v5:
- Link: https://lore.kernel.org/all/20251106131956.1222864-1-dolinux.peng@gmail.com/
- Refactor binary search implementation for improved efficiency
  (Thanks to Andrii and Eduard)
- Extend btf__permute interface with 'ids_sz' parameter to support
  type dropping feature (suggested by Andrii). Plan subsequent reimplementation of
  id_map version for comparative analysis with current sequence interface
- Add comprehensive test coverage for type dropping functionality
- Enhance function comment clarity and accuracy

v4:
- Link: https://lore.kernel.org/all/20251104134033.344807-1-dolinux.peng@gmail.com/
- Abstracted btf_dedup_remap_types logic into a helper function (suggested by Eduard).
- Removed btf_sort.c and implemented sorting separately for libbpf and kernel (suggested by Andrii).
- Added test cases for both base BTF and split BTF scenarios (suggested by Eduard).
- Added validation for name-only sorting of types (suggested by Andrii)
- Refactored btf__permute implementation to reduce complexity (suggested by Andrii)
- Add doc comments for btf__permute (suggested by Andrii)

v3:
- Link: https://lore.kernel.org/all/20251027135423.3098490-1-dolinux.peng@gmail.com/
- Remove sorting logic from libbpf and provide a generic btf__permute() interface (suggested
  by Andrii)
- Omitted the search direction patch to avoid conflicts with base BTF (suggested by Eduard).
- Include btf_sort.c directly in btf.c to reduce function call overhead

v2:
- Link: https://lore.kernel.org/all/20251020093941.548058-1-dolinux.peng@gmail.com/
- Moved sorting to the build phase to reduce overhead (suggested by Alexei).
- Integrated sorting into btf_dedup_compact_and_sort_types (suggested by Eduard).
- Added sorting checks during BTF parsing.
- Consolidated common logic into btf_sort.c for sharing (suggested by Alan).

v1:
- Link: https://lore.kernel.org/all/20251013131537.1927035-1-dolinux.peng@gmail.com/

[1] https://github.com/pengdonglin137/btf_sort_test
[2] https://lore.kernel.org/bpf/20251126012656.3546071-1-ihor.solodrai@linux.dev/
[3] https://lore.kernel.org/bpf/20251205223046.4155870-1-ihor.solodrai@linux.dev/
[4] https://lore.kernel.org/bpf/20251218003314.260269-1-ihor.solodrai@linux.dev/
====================

Link: https://patch.msgid.link/20260109130003.3313716-1-dolinux.peng@gmail.com


Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
parents c9c9f6bf 9282a42a

include/linux/btf.h

+1 −0
Original line number Diff line number Diff line
@@ -219,6 +219,7 @@ bool btf_is_module(const struct btf *btf); 
bool btf_is_vmlinux(const struct btf *btf);

struct module *btf_try_get_module(const struct btf *btf);

u32 btf_nr_types(const struct btf *btf);

u32 btf_named_start_id(const struct btf *btf, bool own);

struct btf *btf_base_btf(const struct btf *btf);

bool btf_type_is_i32(const struct btf_type *t);

bool btf_type_is_i64(const struct btf_type *t);

kernel/bpf/btf.c

+130 −13
Original line number Diff line number Diff line
@@ -259,6 +259,7 @@ struct btf { 
	void *nohdr_data;

	struct btf_header hdr;

	u32 nr_types; /* includes VOID for base BTF */

	u32 named_start_id;

	u32 types_size;

	u32 data_size;

	refcount_t refcnt;
@@ -494,6 +495,11 @@ static bool btf_type_is_modifier(const struct btf_type *t) 
	return false;

}



static int btf_start_id(const struct btf *btf)

{

	return btf->start_id + (btf->base_btf ? 0 : 1);

}



bool btf_type_is_void(const struct btf_type *t)

{

	return t == &btf_void;
@@ -544,21 +550,125 @@ u32 btf_nr_types(const struct btf *btf) 
	return total;

}



/*

 * Note that vmlinux and kernel module BTFs are always sorted

 * during the building phase.

 */

static void btf_check_sorted(struct btf *btf)

{

	u32 i, n, named_start_id = 0;



	n = btf_nr_types(btf);

	if (btf_is_vmlinux(btf)) {

		for (i = btf_start_id(btf); i < n; i++) {

			const struct btf_type *t = btf_type_by_id(btf, i);

			const char *n = btf_name_by_offset(btf, t->name_off);



			if (n[0] != '\0') {

				btf->named_start_id = i;

				return;

			}

		}

		return;

	}



	for (i = btf_start_id(btf) + 1; i < n; i++) {

		const struct btf_type *ta = btf_type_by_id(btf, i - 1);

		const struct btf_type *tb = btf_type_by_id(btf, i);

		const char *na = btf_name_by_offset(btf, ta->name_off);

		const char *nb = btf_name_by_offset(btf, tb->name_off);



		if (strcmp(na, nb) > 0)

			return;



		if (named_start_id == 0 && na[0] != '\0')

			named_start_id = i - 1;

		if (named_start_id == 0 && nb[0] != '\0')

			named_start_id = i;

	}



	if (named_start_id)

		btf->named_start_id = named_start_id;

}



/*

 * btf_named_start_id - Get the named starting ID for the BTF

 * @btf: Pointer to the target BTF object

 * @own: Flag indicating whether to query only the current BTF (true = current BTF only,

 *       false = recursively traverse the base BTF chain)

 *

 * Return value rules:

 * 1. For a sorted btf, return its named_start_id

 * 2. Else for a split BTF, return its start_id

 * 3. Else for a base BTF, return 1

 */

u32 btf_named_start_id(const struct btf *btf, bool own)

{

	const struct btf *base_btf = btf;



	while (!own && base_btf->base_btf)

		base_btf = base_btf->base_btf;



	return base_btf->named_start_id ?: (base_btf->start_id ?: 1);

}



static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)

{

	const struct btf_type *t;

	const char *tname;

	s32 l, r, m;



	l = btf_named_start_id(btf, true);

	r = btf_nr_types(btf) - 1;

	while (l <= r) {

		m = l + (r - l) / 2;

		t = btf_type_by_id(btf, m);

		tname = btf_name_by_offset(btf, t->name_off);

		if (strcmp(tname, name) >= 0) {

			if (l == r)

				return r;

			r = m;

		} else {

			l = m + 1;

		}

	}



	return btf_nr_types(btf);

}



s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)

{

	const struct btf *base_btf = btf_base_btf(btf);

	const struct btf_type *t;

	const char *tname;

	u32 i, total;

	s32 id, total;



	if (base_btf) {

		id = btf_find_by_name_kind(base_btf, name, kind);

		if (id > 0)

			return id;

	}



	total = btf_nr_types(btf);

	for (i = 1; i < total; i++) {

		t = btf_type_by_id(btf, i);

	if (btf->named_start_id > 0 && name[0]) {

		id = btf_find_by_name_kind_bsearch(btf, name);

		for (; id < total; id++) {

			t = btf_type_by_id(btf, id);

			tname = btf_name_by_offset(btf, t->name_off);

			if (strcmp(tname, name) != 0)

				return -ENOENT;

			if (BTF_INFO_KIND(t->info) == kind)

				return id;

		}

	} else {

		for (id = btf_start_id(btf); id < total; id++) {

			t = btf_type_by_id(btf, id);

			if (BTF_INFO_KIND(t->info) != kind)

				continue;



			tname = btf_name_by_offset(btf, t->name_off);

		if (!strcmp(tname, name))

			return i;

			if (strcmp(tname, name) == 0)

				return id;

		}

	}



	return -ENOENT;
@@ -3424,7 +3534,8 @@ const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type 
	const struct btf_type *t;

	int len, id;



	id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, 0);

	id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,

				    btf_named_start_id(btf, false) - 1);

	if (id < 0)

		return ERR_PTR(id);
@@ -5791,6 +5902,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat 
		goto errout;

	}

	env->btf = btf;

	btf->named_start_id = 0;



	data = kvmalloc(attr->btf_size, GFP_KERNEL | __GFP_NOWARN);

	if (!data) {
@@ -6210,6 +6322,7 @@ static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name 
	btf->data = data;

	btf->data_size = data_size;

	btf->kernel_btf = true;

	btf->named_start_id = 0;

	snprintf(btf->name, sizeof(btf->name), "%s", name);



	err = btf_parse_hdr(env);
@@ -6230,6 +6343,7 @@ static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name 
	if (err)

		goto errout;



	btf_check_sorted(btf);

	refcount_set(&btf->refcnt, 1);



	return btf;
@@ -6327,6 +6441,7 @@ static struct btf *btf_parse_module(const char *module_name, const void *data, 
	btf->start_id = base_btf->nr_types;

	btf->start_str_off = base_btf->hdr.str_len;

	btf->kernel_btf = true;

	btf->named_start_id = 0;

	snprintf(btf->name, sizeof(btf->name), "%s", module_name);



	btf->data = kvmemdup(data, data_size, GFP_KERNEL | __GFP_NOWARN);
@@ -6363,6 +6478,7 @@ static struct btf *btf_parse_module(const char *module_name, const void *data, 
	}



	btf_verifier_env_free(env);

	btf_check_sorted(btf);

	refcount_set(&btf->refcnt, 1);

	return btf;
@@ -7729,12 +7845,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) 
			tname);

		return -EINVAL;

	}



	/* Convert BTF function arguments into verifier types.

	 * Only PTR_TO_CTX and SCALAR are supported atm.

	 */

	for (i = 0; i < nargs; i++) {

		u32 tags = 0;

		int id = 0;

		int id = btf_named_start_id(btf, false) - 1;



		/* 'arg:<tag>' decl_tag takes precedence over derivation of

		 * register type from BTF type itself
@@ -9223,7 +9340,7 @@ bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id) 
	}



	/* Attempt to find target candidates in vmlinux BTF first */

	cands = bpf_core_add_cands(cands, main_btf, 1);

	cands = bpf_core_add_cands(cands, main_btf, btf_named_start_id(main_btf, true));

	if (IS_ERR(cands))

		return ERR_CAST(cands);
@@ -9255,7 +9372,7 @@ bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id) 
		 */

		btf_get(mod_btf);

		spin_unlock_bh(&btf_idr_lock);

		cands = bpf_core_add_cands(cands, mod_btf, btf_nr_types(main_btf));

		cands = bpf_core_add_cands(cands, mod_btf, btf_named_start_id(mod_btf, true));

		btf_put(mod_btf);

		if (IS_ERR(cands))

			return ERR_CAST(cands);

kernel/bpf/inode.c

+17 −25
Original line number Diff line number Diff line
@@ -600,10 +600,17 @@ struct bpffs_btf_enums { 


static int find_bpffs_btf_enums(struct bpffs_btf_enums *info)

{

	const struct btf *btf;

	const struct btf_type *t;

	struct {

		const struct btf_type **type;

		const char *name;

	int i, n;

	} btf_enums[] = {

		{&info->cmd_t,		"bpf_cmd"},

		{&info->map_t,		"bpf_map_type"},

		{&info->prog_t,		"bpf_prog_type"},

		{&info->attach_t,	"bpf_attach_type"},

	};

	const struct btf *btf;

	int i, id;



	memset(info, 0, sizeof(*info));
@@ -615,31 +622,16 @@ static int find_bpffs_btf_enums(struct bpffs_btf_enums *info) 


	info->btf = btf;



	for (i = 1, n = btf_nr_types(btf); i < n; i++) {

		t = btf_type_by_id(btf, i);

		if (!btf_type_is_enum(t))

			continue;



		name = btf_name_by_offset(btf, t->name_off);

		if (!name)

			continue;



		if (strcmp(name, "bpf_cmd") == 0)

			info->cmd_t = t;

		else if (strcmp(name, "bpf_map_type") == 0)

			info->map_t = t;

		else if (strcmp(name, "bpf_prog_type") == 0)

			info->prog_t = t;

		else if (strcmp(name, "bpf_attach_type") == 0)

			info->attach_t = t;

		else

			continue;

	for (i = 0; i < ARRAY_SIZE(btf_enums); i++) {

		id = btf_find_by_name_kind(btf, btf_enums[i].name,

					   BTF_KIND_ENUM);

		if (id < 0)

			return -ESRCH;



		if (info->cmd_t && info->map_t && info->prog_t && info->attach_t)

			return 0;

		*btf_enums[i].type = btf_type_by_id(btf, id);

	}



	return -ESRCH;

	return 0;

}



static bool find_btf_enum_const(const struct btf *btf, const struct btf_type *enum_t,

kernel/bpf/verifier.c

+1 −6
Original line number Diff line number Diff line
@@ -20687,12 +20687,7 @@ static int find_btf_percpu_datasec(struct btf *btf) 
	 * types to look at only module's own BTF types.

	 */

	n = btf_nr_types(btf);

	if (btf_is_module(btf))

		i = btf_nr_types(btf_vmlinux);

	else

		i = 1;









	for(; i < n; i++) {









	for (i = btf_named_start_id(btf, true); i < n; i++) {









		t = btf_type_by_id(btf, i);









		if (BTF_INFO_KIND(t->info) != BTF_KIND_DATASEC)









			continue;

























tools/bpf/resolve_btfids/main.c



+64
−0






























Original line number
Diff line number
Diff line







@@ -850,6 +850,67 @@ static int dump_raw_btf(struct btf *btf, const char *out_path)







	return 0;









}



















/*









 * Sort types by name in ascending order resulting in all









 * anonymous types being placed before named types.









 */









static int cmp_type_names(const void *a, const void *b, void *priv)









{









	struct btf *btf = (struct btf *)priv;









	const struct btf_type *ta = btf__type_by_id(btf, *(__u32 *)a);









	const struct btf_type *tb = btf__type_by_id(btf, *(__u32 *)b);









	const char *na, *nb;



















	na = btf__str_by_offset(btf, ta->name_off);









	nb = btf__str_by_offset(btf, tb->name_off);









	return strcmp(na, nb);









}



















static int sort_btf_by_name(struct btf *btf)









{









	__u32 *permute_ids = NULL, *id_map = NULL;









	int nr_types, i, err = 0;









	__u32 start_id = 0, start_offs = 1, id;



















	if (btf__base_btf(btf)) {









		start_id = btf__type_cnt(btf__base_btf(btf));









		start_offs = 0;









	}









	nr_types = btf__type_cnt(btf) - start_id;



















	permute_ids = calloc(nr_types, sizeof(*permute_ids));









	if (!permute_ids) {









		err = -ENOMEM;









		goto out;









	}



















	id_map = calloc(nr_types, sizeof(*id_map));









	if (!id_map) {









		err = -ENOMEM;









		goto out;









	}



















	for (i = 0, id = start_id; i < nr_types; i++, id++)









		permute_ids[i] = id;



















	qsort_r(permute_ids + start_offs, nr_types - start_offs,









		sizeof(*permute_ids), cmp_type_names, btf);



















	for (i = 0; i < nr_types; i++) {









		id = permute_ids[i] - start_id;









		id_map[id] = i + start_id;









	}



















	err = btf__permute(btf, id_map, nr_types, NULL);









	if (err)









		pr_err("FAILED: btf permute: %s\n", strerror(-err));



















out:









	free(permute_ids);









	free(id_map);









	return err;









}



















static inline int make_out_path(char *buf, u32 buf_sz, const char *in_path, const char *suffix)









{









	int len = snprintf(buf, buf_sz, "%s%s", in_path, suffix);








@@ -1025,6 +1086,9 @@ int main(int argc, const char **argv)







	if (load_btf(&obj))









		goto out;



















	if (sort_btf_by_name(obj.btf))









		goto out;



















	if (elf_collect(&obj))









		goto out;