Commit b5145153 authored Nov 29, 2023 by Alexei Starovoitov

Merge branch 'xsk-tx-metadata'

Stanislav Fomichev says:

====================
xsk: TX metadata

This series implements initial TX metadata (offloads) for AF_XDP.
See patch #2 for the main implementation and mlx5/stmmac ones for the
example on how to consume the metadata on the device side.

Starting with two types of offloads:
- request TX timestamp (and write it back into the metadata area)
- request TX checksum offload

Changes since v5:
- preserve xsk_tx_metadata flags across tx and completion by moving
  them out of 'request' union (Jesper)
- fix xdp_metadata checksum failure in big endian (Alexei)
- add SPDX to xdp-rx-metadata.rst (Simon)

v5: https://lore.kernel.org/bpf/20231102225837.1141915-1-sdf@google.com/

Performance (mlx5):

I've implemented a small xskgen tool to try to saturate single tx queue:
https://github.com/fomichev/xskgen/tree/master



Here are the performance numbers with some analysis.

1. Baseline. Running with commit eb62e6ae ("Merge branch 'bpf:
Support bpf_get_func_ip helper in uprobes'"), nothing from this series:

- with 1400 bytes of payload: 98 gbps, 8 mpps
./xskgen -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.189130 sec, 98.357623 gbps 8.409509 mpps

- with 200 bytes of payload: 49 gbps, 23 mpps
./xskgen -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000064 packets 20960134144 bits, took 0.422235 sec, 49.640921 gbps 23.683645 mpps

2. Adding single commit that supports reserving tx_metadata_len
   changes nothing numbers-wise.

- baseline for 1400
./xskgen -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.189247 sec, 98.347946 gbps 8.408682 mpps

- baseline for 200
./xskgen -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 20960000000 bits, took 0.421248 sec, 49.756913 gbps 23.738985 mpps

3. Adding -M flag causes xskgen to reserve the metadata and fill it, but
   doesn't set XDP_TX_METADATA descriptor option.

- new baseline for 1400 (with only filling the metadata)
./xskgen -M -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.188767 sec, 98.387657 gbps 8.412077 mpps

- new baseline for 200 (with only filling the metadata)
./xskgen -M -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 20960000000 bits, took 0.410213 sec, 51.095407 gbps 24.377579 mpps
(the numbers go sligtly up here, not really sure why, maybe some cache-related
side-effects?

4. Next, I'm running the same test but with the commit that adds actual
   general infra to parse XDP_TX_METADATA (but no driver support).
   Essentially applying "xsk: add TX timestamp and TX checksum offload support"
   from this series. Numbers are the same.

- fill metadata for 1400
./xskgen -M -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.188430 sec, 98.415557 gbps 8.414463 mpps

- fill metadata for 200
./xskgen -M -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 20960000000 bits, took 0.411559 sec, 50.928299 gbps 24.297853 mpps

- request metadata for 1400
./xskgen -m -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.188723 sec, 98.391299 gbps 8.412389 mpps

- request metadata for 200
./xskgen -m -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000064 packets 20960134144 bits, took 0.411240 sec, 50.968131 gbps 24.316856 mpps

5. Now, for the most interesting part, I'm adding mlx5 driver support.
   The mpps for 200 bytes case goes down from 23 mpps to 19 mpps, but
   _only_ when I enable the metadata. This looks like a side effect
   of me pushing extra metadata pointer via mlx5e_xdpi_fifo_push.
   Hence, this part is wrapped into 'if (xp_tx_metadata_enabled)'
   to not affect the existing non-metadata use-cases. Since this is not
   regressing existing workloads, I'm not spending any time trying to
   optimize it more (and leaving it up to mlx owners to purse if
   they see any good way to do it).

- same baseline
./xskgen -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.189434 sec, 98.332484 gbps 8.407360 mpps

./xskgen -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000128 packets 20960268288 bits, took 0.425254 sec, 49.288821 gbps 23.515659 mpps

- fill metadata for 1400
./xskgen -M -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.189528 sec, 98.324714 gbps 8.406696 mpps

- fill metadata for 200
./xskgen -M -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000128 packets 20960268288 bits, took 0.519085 sec, 40.379260 gbps 19.264914 mpps

- request metadata for 1400
./xskgen -m -s 1400 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000000 packets 116960000000 bits, took 1.189329 sec, 98.341165 gbps 8.408102 mpps

- request metadata for 200
./xskgen -m -s 200 -b eth3 10:70:fd:48:10:77 10:70:fd:48:10:87 fe80::1270:fdff:fe48:1077 fe80::1270:fdff:fe48:1087 1 1
sent 10000128 packets 20960268288 bits, took 0.519929 sec, 40.313713 gbps 19.233642 mpps

Acked-by: Magnus Karlsson <magnus.karlsson@intel.com>
Acked-by: Jakub Kicinski <kuba@kernel.org>
====================

Link: https://lore.kernel.org/r/20231127190319.1190813-1-sdf@google.com


Signed-off-by: Alexei Starovoitov <ast@kernel.org>

parents 40d0eb02 60523115

Documentation/netlink/specs/netdev.yaml

+18 −1

Original line number	Diff line number	Diff line
		@@ -45,7 +45,6 @@ definitions:
		-
		type: flags
		name: xdp-rx-metadata
		render-max: true
		entries:
		-
		name: timestamp
		@@ -55,6 +54,18 @@ definitions:
		name: hash
		doc:
		Device is capable of exposing receive packet hash via bpf_xdp_metadata_rx_hash().
		-
		type: flags
		name: xsk-flags
		entries:
		-
		name: tx-timestamp
		doc:
		HW timestamping egress packets is supported by the driver.
		-
		name: tx-checksum
		doc:
		L3 checksum HW offload is supported by the driver.

		attribute-sets:
		-
		@@ -86,6 +97,11 @@ attribute-sets:
		See Documentation/networking/xdp-rx-metadata.rst for more details.
		type: u64
		enum: xdp-rx-metadata
		-
		name: xsk-features
		doc: Bitmask of enabled AF_XDP features.
		type: u64
		enum: xsk-flags

		operations:
		list:
		@@ -103,6 +119,7 @@ operations:
		- xdp-features
		- xdp-zc-max-segs
		- xdp-rx-metadata-features
		- xsk-features
		dump:
		reply: *dev-all
		-

Documentation/networking/index.rst

+1 −0

Original line number	Diff line number	Diff line
		@@ -123,6 +123,7 @@ Contents:
		xfrm_sync
		xfrm_sysctl
		xdp-rx-metadata
		xsk-tx-metadata

		.. only:: subproject and html

Documentation/networking/xdp-rx-metadata.rst

+2 −0

Original line number	Diff line number	Diff line
		.. SPDX-License-Identifier: GPL-2.0

		===============
		XDP RX Metadata
		===============

Documentation/networking/xsk-tx-metadata.rst

0 → 100644

+79 −0

Original line number	Diff line number	Diff line
		==================
		AF_XDP TX Metadata
		==================

		This document describes how to enable offloads when transmitting packets
		via :doc:`af_xdp`. Refer to :doc:`xdp-rx-metadata` on how to access similar
		metadata on the receive side.

		General Design
		==============

		The headroom for the metadata is reserved via ``tx_metadata_len`` in
		``struct xdp_umem_reg``. The metadata length is therefore the same for
		every socket that shares the same umem. The metadata layout is a fixed UAPI,
		refer to ``union xsk_tx_metadata`` in ``include/uapi/linux/if_xdp.h``.
		Thus, generally, the ``tx_metadata_len`` field above should contain
		``sizeof(union xsk_tx_metadata)``.

		The headroom and the metadata itself should be located right before
		``xdp_desc->addr`` in the umem frame. Within a frame, the metadata
		layout is as follows::

		tx_metadata_len
		/ \
		+-----------------+---------+----------------------------+
		\| xsk_tx_metadata \| padding \| payload \|
		+-----------------+---------+----------------------------+
		^
		\|
		xdp_desc->addr

		An AF_XDP application can request headrooms larger than ``sizeof(struct
		xsk_tx_metadata)``. The kernel will ignore the padding (and will still
		use ``xdp_desc->addr - tx_metadata_len`` to locate
		the ``xsk_tx_metadata``). For the frames that shouldn't carry
		any metadata (i.e., the ones that don't have ``XDP_TX_METADATA`` option),
		the metadata area is ignored by the kernel as well.

		The flags field enables the particular offload:

		- ``XDP_TXMD_FLAGS_TIMESTAMP``: requests the device to put transmission
		timestamp into ``tx_timestamp`` field of ``union xsk_tx_metadata``.
		- ``XDP_TXMD_FLAGS_CHECKSUM``: requests the device to calculate L4
		checksum. ``csum_start`` specifies byte offset of where the checksumming
		should start and ``csum_offset`` specifies byte offset where the
		device should store the computed checksum.

		Besides the flags above, in order to trigger the offloads, the first
		packet's ``struct xdp_desc`` descriptor should set ``XDP_TX_METADATA``
		bit in the ``options`` field. Also note that in a multi-buffer packet
		only the first chunk should carry the metadata.

		Software TX Checksum
		====================

		For development and testing purposes its possible to pass
		``XDP_UMEM_TX_SW_CSUM`` flag to ``XDP_UMEM_REG`` UMEM registration call.
		In this case, when running in ``XDK_COPY`` mode, the TX checksum
		is calculated on the CPU. Do not enable this option in production because
		it will negatively affect performance.

		Querying Device Capabilities
		============================

		Every devices exports its offloads capabilities via netlink netdev family.
		Refer to ``xsk-flags`` features bitmask in
		``Documentation/netlink/specs/netdev.yaml``.

		- ``tx-timestamp``: device supports ``XDP_TXMD_FLAGS_TIMESTAMP``
		- ``tx-checksum``: device supports ``XDP_TXMD_FLAGS_CHECKSUM``

		See ``tools/net/ynl/samples/netdev.c`` on how to query this information.

		Example
		=======

		See ``tools/testing/selftests/bpf/xdp_hw_metadata.c`` for an example
		program that handles TX metadata. Also see https://github.com/fomichev/xskgen
		for a more bare-bones example.

drivers/net/ethernet/mellanox/mlx5/core/en.h

+3 −1

Original line number	Diff line number	Diff line
		@@ -484,10 +484,12 @@ struct mlx5e_xdp_info_fifo {

		struct mlx5e_xdpsq;
		struct mlx5e_xmit_data;
		struct xsk_tx_metadata;
		typedef int (mlx5e_fp_xmit_xdp_frame_check)(struct mlx5e_xdpsq );
		typedef bool (mlx5e_fp_xmit_xdp_frame)(struct mlx5e_xdpsq ,
		struct mlx5e_xmit_data *,
		int);
		int,
		struct xsk_tx_metadata *);

		struct mlx5e_xdpsq {
		/* data path */