Commit f7e6bd33 authored by Alexei Starovoitov's avatar Alexei Starovoitov
Browse files

Merge branch 'bpf-support-new-insns-from-cpu-v4' 

Yonghong Song says:

====================
bpf: Support new insns from cpu v4

In previous discussion ([1]), it is agreed that we should introduce
cpu version 4 (llvm flag -mcpu=v4) which contains some instructions
which can simplify code, make code easier to understand, fix the
existing problem, or simply for feature completeness. More specifically,
the following new insns are proposed:
  . sign extended load
  . sign extended mov
  . bswap
  . signed div/mod
  . ja with 32-bit offset

This patch set added kernel support for insns proposed in [1] except
BPF_ST which already has full kernel support. Beside the above proposed
insns, LLVM will generate BPF_ST insn as well under -mcpu=v4.
The llvm patch ([2]) has been merged into llvm-project 'main' branch.

The patchset implements interpreter, jit and verifier support for these new
insns.

For this patch set, I tested cpu v2/v3/v4 and the selftests are all passed.
I also tested selftests introduced in this patch set with additional changes
beside normal jit testing (bpf_jit_enable = 1 and bpf_jit_harden = 0)
  - bpf_jit_enable = 0
  - bpf_jit_enable = 1 and bpf_jit_harden = 1
and both testing passed.

  [1] https://lore.kernel.org/bpf/4bfe98be-5333-1c7e-2f6d-42486c8ec039@meta.com/
  [2] https://reviews.llvm.org/D144829

Changelogs:
  v4 -> v5:
   . for v4, patch 8/17 missed in mailing list and patchwork, so resend.
   . rebase on top of master
  v3 -> v4:
   . some minor asm syntax adjustment based on llvm change.
   . add clang version and target arch guard for new tests
     so they can still compile with old llvm compilers.
   . some changes to the bpf doc.
  v2 -> v3:
   . add missed disasm change from v2.
   . handle signed load of ctx fields properly.
   . fix some interpreter sdiv/smod error when bpf_jit_enable = 0.
   . fix some verifier range bounding errors.
   . add more C tests.
  RFCv1 -> v2:
   . add more verifier supports for signed extend load and mov insns.
   . rename some insn names to be more consistent with intel practice.
   . add cpuv4 test runner for test progs.
   . add more unit and C tests.
   . add documentation.
====================

Link: https://lore.kernel.org/r/20230728011143.3710005-1-yonghong.song@linux.dev


Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parents 10d78a66 245d4c40

Documentation/bpf/bpf_design_QA.rst

+0 −5
Original line number Diff line number Diff line
@@ -140,11 +140,6 @@ A: Because if we picked one-to-one relationship to x64 it would have made 
it more complicated to support on arm64 and other archs. Also it

needs div-by-zero runtime check.



Q: Why there is no BPF_SDIV for signed divide operation?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A: Because it would be rarely used. llvm errors in such case and

prints a suggestion to use unsigned divide instead.



Q: Why BPF has implicit prologue and epilogue?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A: Because architectures like sparc have register windows and in general

Documentation/bpf/standardization/instruction-set.rst

+79 −36
Original line number Diff line number Diff line
@@ -154,24 +154,27 @@ otherwise identical operations. 
The 'code' field encodes the operation as below, where 'src' and 'dst' refer

to the values of the source and destination registers, respectively.



========  =====  ==========================================================

code      value  description

========  =====  ==========================================================

BPF_ADD   0x00   dst += src

BPF_SUB   0x10   dst -= src

BPF_MUL   0x20   dst \*= src

BPF_DIV   0x30   dst = (src != 0) ? (dst / src) : 0

BPF_OR    0x40   dst \|= src

BPF_AND   0x50   dst &= src

BPF_LSH   0x60   dst <<= (src & mask)

BPF_RSH   0x70   dst >>= (src & mask)

BPF_NEG   0x80   dst = -dst

BPF_MOD   0x90   dst = (src != 0) ? (dst % src) : dst

BPF_XOR   0xa0   dst ^= src

BPF_MOV   0xb0   dst = src

BPF_ARSH  0xc0   sign extending dst >>= (src & mask)

BPF_END   0xd0   byte swap operations (see `Byte swap instructions`_ below)

========  =====  ==========================================================

========  =====  =======  ==========================================================

code      value  offset   description

========  =====  =======  ==========================================================

BPF_ADD   0x00   0        dst += src

BPF_SUB   0x10   0        dst -= src

BPF_MUL   0x20   0        dst \*= src

BPF_DIV   0x30   0        dst = (src != 0) ? (dst / src) : 0

BPF_SDIV  0x30   1        dst = (src != 0) ? (dst s/ src) : 0

BPF_OR    0x40   0        dst \|= src

BPF_AND   0x50   0        dst &= src

BPF_LSH   0x60   0        dst <<= (src & mask)

BPF_RSH   0x70   0        dst >>= (src & mask)

BPF_NEG   0x80   0        dst = -dst

BPF_MOD   0x90   0        dst = (src != 0) ? (dst % src) : dst

BPF_SMOD  0x90   1        dst = (src != 0) ? (dst s% src) : dst

BPF_XOR   0xa0   0        dst ^= src

BPF_MOV   0xb0   0        dst = src

BPF_MOVSX 0xb0   8/16/32  dst = (s8,s16,s32)src

BPF_ARSH  0xc0   0        sign extending dst >>= (src & mask)

BPF_END   0xd0   0        byte swap operations (see `Byte swap instructions`_ below)

========  =====  ============  ==========================================================



Underflow and overflow are allowed during arithmetic operations, meaning

the 64-bit or 32-bit value will wrap. If eBPF program execution would
@@ -198,11 +201,20 @@ where '(u32)' indicates that the upper 32 bits are zeroed. 


  dst = dst ^ imm32



Also note that the division and modulo operations are unsigned. Thus, for

``BPF_ALU``, 'imm' is first interpreted as an unsigned 32-bit value, whereas

for ``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result

interpreted as an unsigned 64-bit value. There are no instructions for

signed division or modulo.

Note that most instructions have instruction offset of 0. But three instructions

(BPF_SDIV, BPF_SMOD, BPF_MOVSX) have non-zero offset.



The devision and modulo operations support both unsigned and signed flavors.

For unsigned operation (BPF_DIV and BPF_MOD), for ``BPF_ALU``, 'imm' is first

interpreted as an unsigned 32-bit value, whereas for ``BPF_ALU64``, 'imm' is

first sign extended to 64 bits and the result interpreted as an unsigned 64-bit

value.  For signed operation (BPF_SDIV and BPF_SMOD), for ``BPF_ALU``, 'imm' is

interpreted as a signed value. For ``BPF_ALU64``, the 'imm' is sign extended

from 32 to 64 and interpreted as a signed 64-bit value.



Instruction BPF_MOVSX does move operation with sign extension.

``BPF_ALU | MOVSX`` sign extendes 8-bit and 16-bit into 32-bit and upper 32-bit are zeroed.

``BPF_ALU64 | MOVSX`` sign extends 8-bit, 16-bit and 32-bit into 64-bit.



Shift operations use a mask of 0x3F (63) for 64-bit operations and 0x1F (31)

for 32-bit operations.
@@ -210,21 +222,23 @@ for 32-bit operations. 
Byte swap instructions

~~~~~~~~~~~~~~~~~~~~~~



The byte swap instructions use an instruction class of ``BPF_ALU`` and a 4-bit

'code' field of ``BPF_END``.

The byte swap instructions use instruction classes of ``BPF_ALU`` and ``BPF_ALU64``

and a 4-bit 'code' field of ``BPF_END``.



The byte swap instructions operate on the destination register

only and do not use a separate source register or immediate value.



The 1-bit source operand field in the opcode is used to select what byte

order the operation convert from or to:

For ``BPF_ALU``, the 1-bit source operand field in the opcode is used to select what byte

order the operation convert from or to. For ``BPF_ALU64``, the 1-bit source operand

field in the opcode is not used and must be 0.



=========  =====  =================================================

source     value  description

=========  =====  =================================================

BPF_TO_LE  0x00   convert between host byte order and little endian

BPF_TO_BE  0x08   convert between host byte order and big endian

=========  =====  =================================================

=========  =========  =====  =================================================

class      source     value  description

=========  =========  =====  =================================================

BPF_ALU    BPF_TO_LE  0x00   convert between host byte order and little endian

BPF_ALU    BPF_TO_BE  0x08   convert between host byte order and big endian

BPF_ALU64  BPF_TO_LE  0x00   do byte swap unconditionally

=========  =========  =====  =================================================



The 'imm' field encodes the width of the swap operations.  The following widths

are supported: 16, 32 and 64.
@@ -239,6 +253,12 @@ Examples: 


  dst = htobe64(dst)



``BPF_ALU64 | BPF_TO_LE | BPF_END`` with imm = 16/32/64 means::



  dst = bswap16 dst

  dst = bswap32 dst

  dst = bswap64 dst



Jump instructions

-----------------
@@ -249,7 +269,8 @@ The 'code' field encodes the operation as below: 
========  =====  ===  ===========================================  =========================================

code      value  src  description                                  notes

========  =====  ===  ===========================================  =========================================

BPF_JA    0x0    0x0  PC += offset                                 BPF_JMP only

BPF_JA    0x0    0x0  PC += offset                                 BPF_JMP class

BPF_JA    0x0    0x0  PC += imm                                    BPF_JMP32 class

BPF_JEQ   0x1    any  PC += offset if dst == src

BPF_JGT   0x2    any  PC += offset if dst > src                    unsigned

BPF_JGE   0x3    any  PC += offset if dst >= src                   unsigned
@@ -278,6 +299,16 @@ Example: 


where 's>=' indicates a signed '>=' comparison.



``BPF_JA | BPF_K | BPF_JMP32`` (0x06) means::



  gotol +imm



where 'imm' means the branch offset comes from insn 'imm' field.



Note there are two flavors of BPF_JA instrions. BPF_JMP class permits 16-bit jump offset while

BPF_JMP32 permits 32-bit jump offset. A >16bit conditional jmp can be converted to a <16bit

conditional jmp plus a 32-bit unconditional jump.



Helper functions

~~~~~~~~~~~~~~~~
@@ -320,6 +351,7 @@ The mode modifier is one of: 
  BPF_ABS        0x20   legacy BPF packet access (absolute)   `Legacy BPF Packet access instructions`_

  BPF_IND        0x40   legacy BPF packet access (indirect)   `Legacy BPF Packet access instructions`_

  BPF_MEM        0x60   regular load and store operations     `Regular load and store operations`_

  BPF_MEMSX      0x80   sign-extension load operations        `Sign-extension load operations`_

  BPF_ATOMIC     0xc0   atomic operations                     `Atomic operations`_

  =============  =====  ====================================  =============
@@ -350,9 +382,20 @@ instructions that transfer data between a register and memory. 


``BPF_MEM | <size> | BPF_LDX`` means::



  dst = *(size *) (src + offset)

  dst = *(unsigned size *) (src + offset)



Where size is one of: ``BPF_B``, ``BPF_H``, ``BPF_W``, or ``BPF_DW`` and

'unsigned size' is one of u8, u16, u32 and u64.



The ``BPF_MEMSX`` mode modifier is used to encode sign-extension load

instructions that transfer data between a register and memory.



``BPF_MEMSX | <size> | BPF_LDX`` means::



  dst = *(signed size *) (src + offset)



Where size is one of: ``BPF_B``, ``BPF_H``, ``BPF_W``, or ``BPF_DW``.

Where size is one of: ``BPF_B``, ``BPF_H`` or ``BPF_W``, and

'signed size' is one of s8, s16 and s32.



Atomic operations

-----------------

arch/x86/net/bpf_jit_comp.c

+117 −24
Original line number Diff line number Diff line
@@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) 
	*pprog = prog;

}



static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,

			   u32 src_reg)

{

	u8 *prog = *pprog;



	if (is64) {

		/* movs[b,w,l]q dst, src */

		if (num_bits == 8)

			EMIT4(add_2mod(0x48, src_reg, dst_reg), 0x0f, 0xbe,

			      add_2reg(0xC0, src_reg, dst_reg));

		else if (num_bits == 16)

			EMIT4(add_2mod(0x48, src_reg, dst_reg), 0x0f, 0xbf,

			      add_2reg(0xC0, src_reg, dst_reg));

		else if (num_bits == 32)

			EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x63,

			      add_2reg(0xC0, src_reg, dst_reg));

	} else {

		/* movs[b,w]l dst, src */

		if (num_bits == 8) {

			EMIT4(add_2mod(0x40, src_reg, dst_reg), 0x0f, 0xbe,

			      add_2reg(0xC0, src_reg, dst_reg));

		} else if (num_bits == 16) {

			if (is_ereg(dst_reg) || is_ereg(src_reg))

				EMIT1(add_2mod(0x40, src_reg, dst_reg));

			EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,

			      add_2reg(0xC0, src_reg, dst_reg));

		}

	}



	*pprog = prog;

}



/* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */

static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)

{
@@ -779,6 +811,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) 
	*pprog = prog;

}



/* LDSX: dst_reg = *(s8*)(src_reg + off) */

static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)

{

	u8 *prog = *pprog;



	switch (size) {

	case BPF_B:

		/* Emit 'movsx rax, byte ptr [rax + off]' */

		EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x0F, 0xBE);

		break;

	case BPF_H:

		/* Emit 'movsx rax, word ptr [rax + off]' */

		EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x0F, 0xBF);

		break;

	case BPF_W:

		/* Emit 'movsx rax, dword ptr [rax+0x14]' */

		EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63);

		break;

	}

	emit_insn_suffix(&prog, src_reg, dst_reg, off);

	*pprog = prog;

}



/* STX: *(u8*)(dst_reg + off) = src_reg */

static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)

{
@@ -1028,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image 


		case BPF_ALU64 | BPF_MOV | BPF_X:

		case BPF_ALU | BPF_MOV | BPF_X:

			if (insn->off == 0)

				emit_mov_reg(&prog,

					     BPF_CLASS(insn->code) == BPF_ALU64,

					     dst_reg, src_reg);

			else

				emit_movsx_reg(&prog, insn->off,

					       BPF_CLASS(insn->code) == BPF_ALU64,

					       dst_reg, src_reg);

			break;



			/* neg dst */
@@ -1134,6 +1194,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image 
				/* mov rax, dst_reg */

				emit_mov_reg(&prog, is64, BPF_REG_0, dst_reg);



			if (insn->off == 0) {

				/*

				 * xor edx, edx

				 * equivalent to 'xor rdx, rdx', but one byte less
@@ -1143,6 +1204,16 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image 
				/* div src_reg */

				maybe_emit_1mod(&prog, src_reg, is64);

				EMIT2(0xF7, add_1reg(0xF0, src_reg));

			} else {

				if (BPF_CLASS(insn->code) == BPF_ALU)

					EMIT1(0x99); /* cdq */

				else

					EMIT2(0x48, 0x99); /* cqo */



				/* idiv src_reg */

				maybe_emit_1mod(&prog, src_reg, is64);

				EMIT2(0xF7, add_1reg(0xF8, src_reg));

			}



			if (BPF_OP(insn->code) == BPF_MOD &&

			    dst_reg != BPF_REG_3)
@@ -1262,6 +1333,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image 
			break;



		case BPF_ALU | BPF_END | BPF_FROM_BE:

		case BPF_ALU64 | BPF_END | BPF_FROM_LE:

			switch (imm32) {

			case 16:

				/* Emit 'ror %ax, 8' to swap lower 2 bytes */
@@ -1370,9 +1442,17 @@ st: if (is_imm8(insn->off)) 
		case BPF_LDX | BPF_PROBE_MEM | BPF_W:

		case BPF_LDX | BPF_MEM | BPF_DW:

		case BPF_LDX | BPF_PROBE_MEM | BPF_DW:

			/* LDXS: dst_reg = *(s8*)(src_reg + off) */

		case BPF_LDX | BPF_MEMSX | BPF_B:

		case BPF_LDX | BPF_MEMSX | BPF_H:

		case BPF_LDX | BPF_MEMSX | BPF_W:

		case BPF_LDX | BPF_PROBE_MEMSX | BPF_B:

		case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:

		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:

			insn_off = insn->off;



			if (BPF_MODE(insn->code) == BPF_PROBE_MEM) {

			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||

			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {

				/* Conservatively check that src_reg + insn->off is a kernel address:

				 *   src_reg + insn->off >= TASK_SIZE_MAX + PAGE_SIZE

				 * src_reg is used as scratch for src_reg += insn->off and restored
@@ -1415,8 +1495,13 @@ st: if (is_imm8(insn->off)) 
				start_of_ldx = prog;

				end_of_jmp[-1] = start_of_ldx - end_of_jmp;

			}

			if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||

			    BPF_MODE(insn->code) == BPF_MEMSX)

				emit_ldsx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);

			else

				emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);

			if (BPF_MODE(insn->code) == BPF_PROBE_MEM) {

			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||

			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {

				struct exception_table_entry *ex;

				u8 *_insn = image + proglen + (start_of_ldx - temp);

				s64 delta;
@@ -1730,6 +1815,8 @@ st: if (is_imm8(insn->off)) 
			break;



		case BPF_JMP | BPF_JA:

		case BPF_JMP32 | BPF_JA:

			if (BPF_CLASS(insn->code) == BPF_JMP) {

				if (insn->off == -1)

					/* -1 jmp instructions will always jump

					 * backwards two bytes. Explicitly handling
@@ -1740,6 +1827,12 @@ st: if (is_imm8(insn->off)) 
					jmp_offset = -2;

				else

					jmp_offset = addrs[i + insn->off] - addrs[i];

			} else {

				if (insn->imm == -1)

					jmp_offset = -2;

				else

					jmp_offset = addrs[i + insn->imm] - addrs[i];

			}



			if (!jmp_offset) {

				/*

include/linux/filter.h

+13 −4
Original line number Diff line number Diff line
@@ -69,6 +69,9 @@ struct ctl_table_header; 
/* unused opcode to mark special load instruction. Same as BPF_ABS */

#define BPF_PROBE_MEM	0x20



/* unused opcode to mark special ldsx instruction. Same as BPF_IND */

#define BPF_PROBE_MEMSX	0x40



/* unused opcode to mark call to interpreter with arguments */

#define BPF_CALL_ARGS	0xe0
@@ -90,22 +93,28 @@ struct ctl_table_header; 


/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */



#define BPF_ALU64_REG(OP, DST, SRC)				\

#define BPF_ALU64_REG_OFF(OP, DST, SRC, OFF)			\

	((struct bpf_insn) {					\

		.code  = BPF_ALU64 | BPF_OP(OP) | BPF_X,	\

		.dst_reg = DST,					\

		.src_reg = SRC,					\

		.off   = 0,					\

		.off   = OFF,					\

		.imm   = 0 })



#define BPF_ALU32_REG(OP, DST, SRC)				\

#define BPF_ALU64_REG(OP, DST, SRC)				\

	BPF_ALU64_REG_OFF(OP, DST, SRC, 0)



#define BPF_ALU32_REG_OFF(OP, DST, SRC, OFF)			\

	((struct bpf_insn) {					\

		.code  = BPF_ALU | BPF_OP(OP) | BPF_X,		\

		.dst_reg = DST,					\

		.src_reg = SRC,					\

		.off   = 0,					\

		.off   = OFF,					\

		.imm   = 0 })



#define BPF_ALU32_REG(OP, DST, SRC)				\

	BPF_ALU32_REG_OFF(OP, DST, SRC, 0)



/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */



#define BPF_ALU64_IMM(OP, DST, IMM)				\

include/uapi/linux/bpf.h

+1 −0
Original line number Diff line number Diff line
@@ -19,6 +19,7 @@ 


/* ld/ldx fields */

#define BPF_DW		0x18	/* double word (64-bit) */

#define BPF_MEMSX	0x80	/* load with sign extension */

#define BPF_ATOMIC	0xc0	/* atomic memory ops - op type in immediate */

#define BPF_XADD	0xc0	/* exclusive add - legacy name */