Commit 12724ce3 authored by Lu Baolu's avatar Lu Baolu Committed by Will Deacon
Browse files

iommu/vt-d: Optimize iotlb_sync_map for non-caching/non-RWBF modes 



The iotlb_sync_map iommu ops allows drivers to perform necessary cache
flushes when new mappings are established. For the Intel iommu driver,
this callback specifically serves two purposes:

- To flush caches when a second-stage page table is attached to a device
  whose iommu is operating in caching mode (CAP_REG.CM==1).
- To explicitly flush internal write buffers to ensure updates to memory-
  resident remapping structures are visible to hardware (CAP_REG.RWBF==1).

However, in scenarios where neither caching mode nor the RWBF flag is
active, the cache_tag_flush_range_np() helper, which is called in the
iotlb_sync_map path, effectively becomes a no-op.

Despite being a no-op, cache_tag_flush_range_np() involves iterating
through all cache tags of the iommu's attached to the domain, protected
by a spinlock. This unnecessary execution path introduces overhead,
leading to a measurable I/O performance regression. On systems with NVMes
under the same bridge, performance was observed to drop from approximately
~6150 MiB/s down to ~4985 MiB/s.

Introduce a flag in the dmar_domain structure. This flag will only be set
when iotlb_sync_map is required (i.e., when CM or RWBF is set). The
cache_tag_flush_range_np() is called only for domains where this flag is
set. This flag, once set, is immutable, given that there won't be mixed
configurations in real-world scenarios where some IOMMUs in a system
operate in caching mode while others do not. Theoretically, the
immutability of this flag does not impact functionality.

Reported-by: default avatarIoanna Alifieraki <ioanna-maria.alifieraki@canonical.com>
Closes: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2115738
Link: https://lore.kernel.org/r/20250701171154.52435-1-ioanna-maria.alifieraki@canonical.com


Fixes: 129dab6e ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map")
Cc: stable@vger.kernel.org
Signed-off-by: default avatarLu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: default avatarKevin Tian <kevin.tian@intel.com>
Link: https://lore.kernel.org/r/20250703031545.3378602-1-baolu.lu@linux.intel.com
Link: https://lore.kernel.org/r/20250714045028.958850-3-baolu.lu@linux.intel.com


Signed-off-by: default avatarWill Deacon <will@kernel.org>
parent bd26cd9d

drivers/iommu/intel/iommu.c

+18 −1
Original line number Diff line number Diff line
@@ -1796,6 +1796,18 @@ static int domain_setup_first_level(struct intel_iommu *iommu, 
					  (pgd_t *)pgd, flags, old);

}



static bool domain_need_iotlb_sync_map(struct dmar_domain *domain,

				       struct intel_iommu *iommu)

{

	if (cap_caching_mode(iommu->cap) && !domain->use_first_level)

		return true;



	if (rwbf_quirk || cap_rwbf(iommu->cap))

		return true;



	return false;

}



static int dmar_domain_attach_device(struct dmar_domain *domain,

				     struct device *dev)

{
@@ -1833,6 +1845,8 @@ static int dmar_domain_attach_device(struct dmar_domain *domain, 
	if (ret)

		goto out_block_translation;



	domain->iotlb_sync_map |= domain_need_iotlb_sync_map(domain, iommu);



	return 0;



out_block_translation:
@@ -3945,7 +3959,10 @@ static bool risky_device(struct pci_dev *pdev) 
static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,

				      unsigned long iova, size_t size)

{

	cache_tag_flush_range_np(to_dmar_domain(domain), iova, iova + size - 1);

	struct dmar_domain *dmar_domain = to_dmar_domain(domain);



	if (dmar_domain->iotlb_sync_map)

		cache_tag_flush_range_np(dmar_domain, iova, iova + size - 1);



	return 0;

}

drivers/iommu/intel/iommu.h

+3 −0
Original line number Diff line number Diff line
@@ -614,6 +614,9 @@ struct dmar_domain { 
	u8 has_mappings:1;		/* Has mappings configured through

					 * iommu_map() interface.

					 */

	u8 iotlb_sync_map:1;		/* Need to flush IOTLB cache or write

					 * buffer when creating mappings.

					 */



	spinlock_t lock;		/* Protect device tracking lists */

	struct list_head devices;	/* all devices' list */