Commit 02c163e9 authored by Linus Torvalds's avatar Linus Torvalds
Browse files
Pull CXL updates from Dan Williams:
 "CXL has mechanisms to enumerate the performance characteristics of
  memory devices. Those mechanisms allow Linux to build the equivalent
  of ACPI SRAT, SLIT, and HMAT tables dynamically at runtime. That
  capability is necessary because static ACPI can not represent dynamic
  CXL configurations (and reconfigurations).

  So, building on the v6.8 work to add "Quality of Service" enumeration,
  this update plumbs CXL "access coordinates" (read/write access latency
  and bandwidth) in all the same places that ACPI HMAT feeds similar
  data. Follow-on patches from the -mm side can then use that data to
  feed mechanisms like mm/memory-tiers.c. Greg has acked the touch to
  drivers/base/.

  The other feature update this cycle is support for CXL error injection
  via the ACPI EINJ module. That facility enables injection of bus
  protocol errors provided the user knows the magic address values to
  insert in the interface. To hide that magic, and make this easier to
  use, new error injection attributes were added to CXL debugfs. That
  interface injects the errors relative to a CXL object rather than
  require user tooling to know how to lookup and inject RCRB (Root
  Complex Register Block) addresses into the raw EINJ debugfs interface.
  It received some helpful review comments from Tony, but no explicit
  acks from the ACPI side. The primary user visible change for existing
  EINJ users is that they may find that einj.ko was already loaded by
  cxl_core.ko. Previously, einj.ko was only loaded on demand.

  The usual collection of miscellaneous cleanups are also present this
  cycle.

  Summary:

   - Supplement ACPI HMAT reported memory performance with native CXL
     memory performance enumeration

   - Add support for CXL error injection via the ACPI EINJ mechanism

   - Cleanup CXL DOE and CDAT integration

   - Miscellaneous cleanups and fixes"

* tag 'cxl-for-6.9' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl: (21 commits)
  Documentation/ABI/testing/debugfs-cxl: Fix "Unexpected indentation"
  lib/firmware_table: Provide buffer length argument to cdat_table_parse()
  cxl/pci: Get rid of pointer arithmetic reading CDAT table
  cxl/pci: Rename DOE mailbox handle to doe_mb
  cxl: Fix the incorrect assignment of SSLBIS entry pointer initial location
  cxl/core: Add CXL EINJ debugfs files
  EINJ, Documentation: Update EINJ kernel doc
  EINJ: Add CXL error type support
  EINJ: Migrate to a platform driver
  cxl/region: Deal with numa nodes not enumerated by SRAT
  cxl/region: Add memory hotplug notifier for cxl region
  cxl/region: Add sysfs attribute for locality attributes of CXL regions
  cxl/region: Calculate performance data for a region
  cxl: Set cxlmd->endpoint before adding port device
  cxl: Move QoS class to be calculated from the nearest CPU
  cxl: Split out host bridge access coordinates
  cxl: Split out combine_coordinates() for common shared usage
  ACPI: HMAT / cxl: Add retrieval of generic port coordinates for both access classes
  ACPI: HMAT: Introduce 2 levels of generic port access class
  base/node / ACPI: Enumerate node access class for 'struct access_coordinate'
  ...
parents 5c84b051 ed1ff2fb
Loading
Loading
Loading
Loading
+34 −0
Original line number Diff line number Diff line
@@ -33,3 +33,37 @@ Description:
		device cannot clear poison from the address, -ENXIO is returned.
		The clear_poison attribute is only visible for devices
		supporting the capability.

What:		/sys/kernel/debug/cxl/einj_types
Date:		January, 2024
KernelVersion:	v6.9
Contact:	linux-cxl@vger.kernel.org
Description:
		(RO) Prints the CXL protocol error types made available by
		the platform in the format:

			0x<error number> <error type>

		The possible error types are (as of ACPI v6.5):

			0x1000	CXL.cache Protocol Correctable
			0x2000	CXL.cache Protocol Uncorrectable non-fatal
			0x4000	CXL.cache Protocol Uncorrectable fatal
			0x8000	CXL.mem Protocol Correctable
			0x10000	CXL.mem Protocol Uncorrectable non-fatal
			0x20000	CXL.mem Protocol Uncorrectable fatal

		The <error number> can be written to einj_inject to inject
		<error type> into a chosen dport.

What:		/sys/kernel/debug/cxl/$dport_dev/einj_inject
Date:		January, 2024
KernelVersion:	v6.9
Contact:	linux-cxl@vger.kernel.org
Description:
		(WO) Writing an integer to this file injects the corresponding
		CXL protocol error into $dport_dev ($dport_dev will be a device
		name from /sys/bus/pci/devices). The integer to type mapping for
		injection can be found by reading from einj_types. If the dport
		was enumerated in RCH mode, a CXL 1.1 error is injected, otherwise
		a CXL 2.0 error is injected.
+34 −0
Original line number Diff line number Diff line
@@ -552,3 +552,37 @@ Description:
		attribute is only visible for devices supporting the
		capability. The retrieved errors are logged as kernel
		events when cxl_poison event tracing is enabled.


What:		/sys/bus/cxl/devices/regionZ/accessY/read_bandwidth
		/sys/bus/cxl/devices/regionZ/accessY/write_banwidth
Date:		Jan, 2024
KernelVersion:	v6.9
Contact:	linux-cxl@vger.kernel.org
Description:
		(RO) The aggregated read or write bandwidth of the region. The
		number is the accumulated read or write bandwidth of all CXL memory
		devices that contributes to the region in MB/s. It is
		identical data that should appear in
		/sys/devices/system/node/nodeX/accessY/initiators/read_bandwidth or
		/sys/devices/system/node/nodeX/accessY/initiators/write_bandwidth.
		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
		the number to the closest initiator and access1 provides the
		number to the closest CPU.


What:		/sys/bus/cxl/devices/regionZ/accessY/read_latency
		/sys/bus/cxl/devices/regionZ/accessY/write_latency
Date:		Jan, 2024
KernelVersion:	v6.9
Contact:	linux-cxl@vger.kernel.org
Description:
		(RO) The read or write latency of the region. The number is
		the worst read or write latency of all CXL memory devices that
		contributes to the region in nanoseconds. It is identical data
		that should appear in
		/sys/devices/system/node/nodeX/accessY/initiators/read_latency or
		/sys/devices/system/node/nodeX/accessY/initiators/write_latency.
		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
		the number to the closest initiator and access1 provides the
		number to the closest CPU.
+34 −0
Original line number Diff line number Diff line
@@ -32,6 +32,10 @@ configuration::
  CONFIG_ACPI_APEI
  CONFIG_ACPI_APEI_EINJ

...and to (optionally) enable CXL protocol error injection set::

  CONFIG_ACPI_APEI_EINJ_CXL

The EINJ user interface is in <debugfs mount point>/apei/einj.

The following files belong to it:
@@ -118,6 +122,24 @@ The following files belong to it:
  this actually works depends on what operations the BIOS actually
  includes in the trigger phase.

CXL error types are supported from ACPI 6.5 onwards (given a CXL port
is present). The EINJ user interface for CXL error types is at
<debugfs mount point>/cxl. The following files belong to it:

- einj_types:

  Provides the same functionality as available_error_types above, but
  for CXL error types

- $dport_dev/einj_inject:

  Injects a CXL error type into the CXL port represented by $dport_dev,
  where $dport_dev is the name of the CXL port (usually a PCIe device name).
  Error injections targeting a CXL 2.0+ port can use the legacy interface
  under <debugfs mount point>/apei/einj, while CXL 1.1/1.0 port injections
  must use this file.


BIOS versions based on the ACPI 4.0 specification have limited options
in controlling where the errors are injected. Your BIOS may support an
extension (enabled with the param_extension=1 module parameter, or boot
@@ -181,6 +203,18 @@ You should see something like this in dmesg::
  [22715.834759] EDAC sbridge MC3: PROCESSOR 0:306e7 TIME 1422553404 SOCKET 0 APIC 0
  [22716.616173] EDAC MC3: 1 CE memory read error on CPU_SrcID#0_Channel#0_DIMM#0 (channel:0 slot:0 page:0x12345 offset:0x0 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0090 socket:0 channel_mask:1 rank:0)

A CXL error injection example with $dport_dev=0000:e0:01.1::

    # cd /sys/kernel/debug/cxl/
    # ls
    0000:e0:01.1 0000:0c:00.0
    # cat einj_types                # See which errors can be injected
	0x00008000  CXL.mem Protocol Correctable
	0x00010000  CXL.mem Protocol Uncorrectable non-fatal
	0x00020000  CXL.mem Protocol Uncorrectable fatal
    # cd 0000:e0:01.1               # Navigate to dport to inject into
    # echo 0x8000 > einj_inject     # Inject error

Special notes for injection into SGX enclaves:

There may be a separate BIOS setup option to enable SGX injection.
+1 −0
Original line number Diff line number Diff line
@@ -5321,6 +5321,7 @@ M: Dan Williams <dan.j.williams@intel.com>
L:	linux-cxl@vger.kernel.org
S:	Maintained
F:	drivers/cxl/
F:	include/linux/cxl-einj.h
F:	include/linux/cxl-event.h
F:	include/uapi/linux/cxl_mem.h
F:	tools/testing/cxl/
+13 −0
Original line number Diff line number Diff line
@@ -60,6 +60,19 @@ config ACPI_APEI_EINJ
	  mainly used for debugging and testing the other parts of
	  APEI and some other RAS features.

config ACPI_APEI_EINJ_CXL
	bool "CXL Error INJection Support"
	default ACPI_APEI_EINJ
	depends on ACPI_APEI_EINJ
	depends on CXL_BUS && CXL_BUS <= ACPI_APEI_EINJ
	help
	  Support for CXL protocol Error INJection through debugfs/cxl.
	  Availability and which errors are supported is dependent on
	  the host platform. Look to ACPI v6.5 section 18.6.4 and kernel
	  EINJ documentation for more information.

	  If unsure say 'n'

config ACPI_APEI_ERST_DEBUG
	tristate "APEI Error Record Serialization Table (ERST) Debug Support"
	depends on ACPI_APEI
Loading