+15
−26
+8
−35
+29
−58
+18
−43
+2
−20
Loading
Merge branch 'emit-endbr-bti-instructions-for-indirect'
Xu Kuohai says: ==================== emit ENDBR/BTI instructions for indirect On architectures with CFI protection enabled that require landing pad instructions at indirect jump targets, such as x86 with CET/IBT enabled and arm64 with BTI enabled, kernel panics when an indirect jump lands on a target without landing pad. Therefore, the JIT must emit landing pad instructions for indirect jump targets. The verifier already recognizes which instructions are indirect jump targets during the verification phase. So we can store this information in env->insn_aux_data and pass it to the JIT as new parameter, allowing the JIT to consult env->insn_aux_data to determine which instructions are indirect jump targets. During JIT, constants blinding is performed. It rewrites the private copy of instructions for the JITed program, but it does not adjust the global env->insn_aux_data array. As a result, after constants blinding, the instruction indexes used by JIT may no longer match the indexes in env->insn_aux_data, so the JIT can not use env->insn_aux_data directly. To avoid this mismatch, and given that all existing arch-specific JITs already implement constants blinding with largely duplicated code, move constants blinding from JIT to generic code. v15: - Rebase and target bpf tree - Resotre subprog_start of the fake 'exit' subprog on failure - Fix wrong function name used in comment v14: https://lore.kernel.org/all/cover.1776062885.git.xukuohai@hotmail.com/ - Rebase - Fix comment style - Fix incorrect variable and function name used in commit message v13: https://lore.kernel.org/bpf/20260411133847.1042658-1-xukuohai@huaweicloud.com - Use vmalloc to allocate memory for insn_aux_data copies to match with vfree - Do not free the copied memory of insn_aux_data when restoring from failure - Code cleanup v12: https://lore.kernel.org/bpf/20260403132811.753894-1-xukuohai@huaweicloud.com - Restore env->insn_aux_data on JIT failure - Fix incorrect error code sign (-EFAULT vs EFAULT) - Fix incorrect prog used in the restore path v11: https://lore.kernel.org/bpf/20260403090915.473493-1-xukuohai@huaweicloud.com - Restore env->subprog_info after jit_subprogs() fails - Clear prog->jit_requested and prog->blinding_requested on failure - Use the actual env->insn_aux_data size in clear_insn_aux_data() on failure v10: https://lore.kernel.org/bpf/20260324122052.342751-1-xukuohai@huaweicloud.com - Fix the incorrect call_imm restore in jit_subprogs - Define a dummy void version of bpf_jit_prog_release_other and bpf_patch_insn_data when the corresponding config is not set - Remove the unnecessary #ifdef in x86_64 JIT (Leon Hwang) v9: https://lore.kernel.org/bpf/20260312170255.3427799-1-xukuohai@huaweicloud.com - Make constant blinding available for classic bpf (Eduard) - Clear prog->bpf_func, prog->jited ... on the error path of extra pass (Eduard) - Fix spelling errors and remove unused parameter (Anton Protopopov) v8: https://lore.kernel.org/bpf/20260309140044.2652538-1-xukuohai@huaweicloud.com - Define void bpf_jit_blind_constants() function when CONFIG_BPF_JIT is not set - Move indirect_target fixup for insn patching from bpf_jit_blind_constants() to adjust_insn_aux_data() v7: https://lore.kernel.org/bpf/20260307103949.2340104-1-xukuohai@huaweicloud.com - Move constants blinding logic back to bpf/core.c - Compute ip address before switch statement in x86 JIT - Clear JIT state from error path on arm64 and loongarch v6: https://lore.kernel.org/bpf/20260306102329.2056216-1-xukuohai@huaweicloud.com - Move constants blinding from JIT to verifier - Move call to bpf_prog_select_runtime from bpf_prog_load to verifier v5: https://lore.kernel.org/bpf/20260302102726.1126019-1-xukuohai@huaweicloud.com - Switch to pass env to JIT directly to get rid of copying private insn_aux_data for each prog v4: https://lore.kernel.org/all/20260114093914.2403982-1-xukuohai@huaweicloud.com - Switch to the approach proposed by Eduard, using insn_aux_data to identify indirect jump targets, and emit ENDBR on x86 v3: https://lore.kernel.org/bpf/20251227081033.240336-1-xukuohai@huaweicloud.com - Get rid of unnecessary enum definition (Yonghong Song, Anton Protopopov) v2: https://lore.kernel.org/bpf/20251223085447.139301-1-xukuohai@huaweicloud.com - Exclude instruction arrays not used for indirect jumps (Anton Protopopov) v1: https://lore.kernel.org/bpf/20251127140318.3944249-1-xukuohai@huaweicloud.com ==================== Link: https://patch.msgid.link/20260416064341.151802-1-xukuohai@huaweicloud.com Signed-off-by:Alexei Starovoitov <ast@kernel.org>