cxl: docs/linux/cxl-driver - add example configurations (ef3a43a6) · Commits · git / linux-nf

Documentation/driver-api/cxl/linux/cxl-driver.rst

+10 −0

Original line number	Diff line number	Diff line
		@@ -520,3 +520,13 @@ from HPA to DPA. This is why they must be aware of the entire interleave set.

		Linux does not support unbalanced interleave configurations. As a result, all
		endpoints in an interleave set must have the same ways and granularity.

		Example Configurations
		======================
		.. toctree::
		:maxdepth: 1

		example-configurations/single-device.rst
		example-configurations/hb-interleave.rst
		example-configurations/intra-hb-interleave.rst
		example-configurations/multi-interleave.rst

Documentation/driver-api/cxl/linux/example-configurations/hb-interleave.rst

0 → 100644

+314 −0

Original line number	Diff line number	Diff line
		.. SPDX-License-Identifier: GPL-2.0

		============================
		Inter-Host-Bridge Interleave
		============================
		This cxl-cli configuration dump shows the following host configuration:

		* A single socket system with one CXL root
		* CXL Root has Four (4) CXL Host Bridges
		* Two CXL Host Bridges have a single CXL Memory Expander Attached
		* The CXL root is configured to interleave across the two host bridges.

		This output is generated by :code:`cxl list -v` and describes the relationships
		between objects exposed in :code:`/sys/bus/cxl/devices/`.

		::

		[
		{
		"bus":"root0",
		"provider":"ACPI.CXL",
		"nr_dports":4,
		"dports":[
		{
		"dport":"pci0000:00",
		"alias":"ACPI0016:01",
		"id":0
		},
		{
		"dport":"pci0000:a8",
		"alias":"ACPI0016:02",
		"id":4
		},
		{
		"dport":"pci0000:2a",
		"alias":"ACPI0016:03",
		"id":1
		},
		{
		"dport":"pci0000:d2",
		"alias":"ACPI0016:00",
		"id":5
		}
		],

		This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to CXL
		Host Bridges. The `Root` can be considered the singular upstream port attached
		to the platform's memory controller - which routes memory requests to it.

		The `ports:root0` section lays out how each of these downstream ports are
		configured. If a port is not configured (id's 0 and 1), they are omitted.

		::

		"ports:root0":[
		{
		"port":"port1",
		"host":"pci0000:d2",
		"depth":1,
		"nr_dports":3,
		"dports":[
		{
		"dport":"0000:d2:01.1",
		"alias":"device:02",
		"id":0
		},
		{
		"dport":"0000:d2:01.3",
		"alias":"device:05",
		"id":2
		},
		{
		"dport":"0000:d2:07.1",
		"alias":"device:0d",
		"id":113
		}
		],

		This chunk shows the available downstream ports associated with the CXL Host
		Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstream
		ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`..

		::

		"endpoints:port1":[
		{
		"endpoint":"endpoint5",
		"host":"mem0",
		"parent_dport":"0000:d2:01.1",
		"depth":2,
		"memdev":{
		"memdev":"mem0",
		"ram_size":137438953472,
		"serial":0,
		"numa_node":0,
		"host":"0000:d3:00.0"
		},
		"decoders:endpoint5":[
		{
		"decoder":"decoder5.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"region":"region0",
		"dpa_resource":0,
		"dpa_size":137438953472,
		"mode":"ram"
		}
		]
		}
		],

		This chunk shows the endpoints attached to the host bridge :code:`port1`.

		:code:`endpoint5` contains a single configured decoder :code:`decoder5.0`
		which has the same interleave configuration as :code:`region0` (shown later).

		Next we have the decodesr belonging to the host bridge:

		::

		"decoders:port1":[
		{
		"decoder":"decoder1.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":1,
		"region":"region0",
		"nr_targets":1,
		"targets":[
		{
		"target":"0000:d2:01.1",
		"alias":"device:02",
		"position":0,
		"id":0
		}
		]
		}
		]
		},

		Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose only
		target is :code:`dport1` - which is attached to :code:`endpoint5`.

		The following chunk shows a similar configuration for Host Bridge :code:`port3`,
		the second host bridge with a memory device attached.

		::

		{
		"port":"port3",
		"host":"pci0000:a8",
		"depth":1,
		"nr_dports":1,
		"dports":[
		{
		"dport":"0000:a8:01.1",
		"alias":"device:c3",
		"id":0
		}
		],
		"endpoints:port3":[
		{
		"endpoint":"endpoint6",
		"host":"mem1",
		"parent_dport":"0000:a8:01.1",
		"depth":2,
		"memdev":{
		"memdev":"mem1",
		"ram_size":137438953472,
		"serial":0,
		"numa_node":0,
		"host":"0000:a9:00.0"
		},
		"decoders:endpoint6":[
		{
		"decoder":"decoder6.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"region":"region0",
		"dpa_resource":0,
		"dpa_size":137438953472,
		"mode":"ram"
		}
		]
		}
		],
		"decoders:port3":[
		{
		"decoder":"decoder3.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":1,
		"region":"region0",
		"nr_targets":1,
		"targets":[
		{
		"target":"0000:a8:01.1",
		"alias":"device:c3",
		"position":0,
		"id":0
		}
		]
		}
		]
		},


		The next chunk shows the two CXL host bridges without attached endpoints.

		::

		{
		"port":"port2",
		"host":"pci0000:00",
		"depth":1,
		"nr_dports":2,
		"dports":[
		{
		"dport":"0000:00:01.3",
		"alias":"device:55",
		"id":2
		},
		{
		"dport":"0000:00:07.1",
		"alias":"device:5d",
		"id":113
		}
		]
		},
		{
		"port":"port4",
		"host":"pci0000:2a",
		"depth":1,
		"nr_dports":1,
		"dports":[
		{
		"dport":"0000:2a:01.1",
		"alias":"device:d0",
		"id":0
		}
		]
		}
		],

		Next we have the `Root Decoders` belonging to :code:`root0`. This root decoder
		applies the interleave across the downstream ports :code:`port1` and
		:code:`port3` - with a granularity of 256 bytes.

		This information is generated by the CXL driver reading the ACPI CEDT CMFWS.

		::

		"decoders:root0":[
		{
		"decoder":"decoder0.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"max_available_extent":0,
		"volatile_capable":true,
		"nr_targets":2,
		"targets":[
		{
		"target":"pci0000:a8",
		"alias":"ACPI0016:02",
		"position":1,
		"id":4
		},
		{
		"target":"pci0000:d2",
		"alias":"ACPI0016:00",
		"position":0,
		"id":5
		}
		],

		Finally we have the `Memory Region` associated with the `Root Decoder`
		:code:`decoder0.0`. This region describes the overall interleave configuration
		of the interleave set.

		::

		"regions:decoder0.0":[
		{
		"region":"region0",
		"resource":825975898112,
		"size":274877906944,
		"type":"ram",
		"interleave_ways":2,
		"interleave_granularity":256,
		"decode_state":"commit",
		"mappings":[
		{
		"position":1,
		"memdev":"mem1",
		"decoder":"decoder6.0"
		},
		{
		"position":0,
		"memdev":"mem0",
		"decoder":"decoder5.0"
		}
		]
		}
		]
		}
		]
		}
		]

Documentation/driver-api/cxl/linux/example-configurations/intra-hb-interleave.rst

0 → 100644

+291 −0

Original line number	Diff line number	Diff line
		.. SPDX-License-Identifier: GPL-2.0

		============================
		Intra-Host-Bridge Interleave
		============================
		This cxl-cli configuration dump shows the following host configuration:

		* A single socket system with one CXL root
		* CXL Root has Four (4) CXL Host Bridges
		* One (1) CXL Host Bridges has two CXL Memory Expanders Attached
		* The Host bridge decoder is programmed to interleave across the expanders.

		This output is generated by :code:`cxl list -v` and describes the relationships
		between objects exposed in :code:`/sys/bus/cxl/devices/`.

		::

		[
		{
		"bus":"root0",
		"provider":"ACPI.CXL",
		"nr_dports":4,
		"dports":[
		{
		"dport":"pci0000:00",
		"alias":"ACPI0016:01",
		"id":0
		},
		{
		"dport":"pci0000:a8",
		"alias":"ACPI0016:02",
		"id":4
		},
		{
		"dport":"pci0000:2a",
		"alias":"ACPI0016:03",
		"id":1
		},
		{
		"dport":"pci0000:d2",
		"alias":"ACPI0016:00",
		"id":5
		}
		],

		This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to CXL
		Host Bridges. The `Root` can be considered the singular upstream port attached
		to the platform's memory controller - which routes memory requests to it.

		The `ports:root0` section lays out how each of these downstream ports are
		configured. If a port is not configured (id's 0 and 1), they are omitted.

		::

		"ports:root0":[
		{
		"port":"port1",
		"host":"pci0000:d2",
		"depth":1,
		"nr_dports":3,
		"dports":[
		{
		"dport":"0000:d2:01.1",
		"alias":"device:02",
		"id":0
		},
		{
		"dport":"0000:d2:01.3",
		"alias":"device:05",
		"id":2
		},
		{
		"dport":"0000:d2:07.1",
		"alias":"device:0d",
		"id":113
		}
		],

		This chunk shows the available downstream ports associated with the CXL Host
		Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstream
		ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`..

		::

		"endpoints:port1":[
		{
		"endpoint":"endpoint5",
		"host":"mem0",
		"parent_dport":"0000:d2:01.1",
		"depth":2,
		"memdev":{
		"memdev":"mem0",
		"ram_size":137438953472,
		"serial":0,
		"numa_node":0,
		"host":"0000:d3:00.0"
		},
		"decoders:endpoint5":[
		{
		"decoder":"decoder5.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"region":"region0",
		"dpa_resource":0,
		"dpa_size":137438953472,
		"mode":"ram"
		}
		]
		},
		{
		"endpoint":"endpoint6",
		"host":"mem1",
		"parent_dport":"0000:d2:01.3,
		"depth":2,
		"memdev":{
		"memdev":"mem1",
		"ram_size":137438953472,
		"serial":0,
		"numa_node":0,
		"host":"0000:a9:00.0"
		},
		"decoders:endpoint6":[
		{
		"decoder":"decoder6.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"region":"region0",
		"dpa_resource":0,
		"dpa_size":137438953472,
		"mode":"ram"
		}
		]
		}
		],

		This chunk shows the endpoints attached to the host bridge :code:`port1`.

		:code:`endpoint5` contains a single configured decoder :code:`decoder5.0`
		which has the same interleave configuration memory region they belong to
		(show later).

		Next we have the decoders belonging to the host bridge:

		::

		"decoders:port1":[
		{
		"decoder":"decoder1.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":2,
		"interleave_granularity":256,
		"region":"region0",
		"nr_targets":2,
		"targets":[
		{
		"target":"0000:d2:01.1",
		"alias":"device:02",
		"position":0,
		"id":0
		},
		{
		"target":"0000:d2:01.3",
		"alias":"device:05",
		"position":1,
		"id":0
		}
		]
		}
		]
		},

		Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`) with two
		targets: :code:`dport1` and :code:`dport3` - which are attached to
		:code:`endpoint5` and :code:`endpoint6` respectively.

		The host bridge decoder interleaves these devices at a 256 byte granularity.

		The next chunk shows the three CXL host bridges without attached endpoints.

		::

		{
		"port":"port2",
		"host":"pci0000:00",
		"depth":1,
		"nr_dports":2,
		"dports":[
		{
		"dport":"0000:00:01.3",
		"alias":"device:55",
		"id":2
		},
		{
		"dport":"0000:00:07.1",
		"alias":"device:5d",
		"id":113
		}
		]
		},
		{
		"port":"port3",
		"host":"pci0000:a8",
		"depth":1,
		"nr_dports":1,
		"dports":[
		{
		"dport":"0000:a8:01.1",
		"alias":"device:c3",
		"id":0
		}
		],
		},
		{
		"port":"port4",
		"host":"pci0000:2a",
		"depth":1,
		"nr_dports":1,
		"dports":[
		{
		"dport":"0000:2a:01.1",
		"alias":"device:d0",
		"id":0
		}
		]
		}
		],

		Next we have the `Root Decoders` belonging to :code:`root0`. This root decoder
		applies the interleave across the downstream ports :code:`port1` and
		:code:`port3` - with a granularity of 256 bytes.

		This information is generated by the CXL driver reading the ACPI CEDT CMFWS.

		::

		"decoders:root0":[
		{
		"decoder":"decoder0.0",
		"resource":825975898112,
		"size":274877906944,
		"interleave_ways":1,
		"max_available_extent":0,
		"volatile_capable":true,
		"nr_targets":2,
		"targets":[
		{
		"target":"pci0000:a8",
		"alias":"ACPI0016:02",
		"position":1,
		"id":4
		},
		],

		Finally we have the `Memory Region` associated with the `Root Decoder`
		:code:`decoder0.0`. This region describes the overall interleave configuration
		of the interleave set.

		::

		"regions:decoder0.0":[
		{
		"region":"region0",
		"resource":825975898112,
		"size":274877906944,
		"type":"ram",
		"interleave_ways":2,
		"interleave_granularity":256,
		"decode_state":"commit",
		"mappings":[
		{
		"position":1,
		"memdev":"mem1",
		"decoder":"decoder6.0"
		},
		{
		"position":0,
		"memdev":"mem0",
		"decoder":"decoder5.0"
		}
		]
		}
		]
		}
		]
		}
		]

Documentation/driver-api/cxl/linux/example-configurations/multi-interleave.rst

0 → 100644

+401 −0

File added.

Preview size limit exceeded, changes collapsed.

Documentation/driver-api/cxl/linux/example-configurations/single-device.rst

0 → 100644

+246 −0

File added.

Preview size limit exceeded, changes collapsed.