Merge branch 'kvm-tdx-initial' into HEAD

mmap() that affects the region will be made visible immediately.  Another

example is madvise(MADV_DROP).

For TDX guest, deleting/moving memory region loses guest memory contents.

Read only region isn't supported.  Only as-id 0 is supported.

Note: On arm64, a write generated by the page-table walker (to update

the Access and Dirty flags, for example) never results in a

KVM_EXIT_MMIO exit when the slot has the KVM_MEM_READONLY flag. This

:Capability: basic

:Architectures: x86

:Type: vm

:Type: vm ioctl, vcpu ioctl

:Parameters: an opaque platform specific structure (in/out)

:Returns: 0 on success; -1 on error

for issuing platform-specific memory encryption commands to manage those

encrypted VMs.

Currently, this ioctl is used for issuing Secure Encrypted Virtualization

(SEV) commands on AMD Processors. The SEV commands are defined in

Documentation/virt/kvm/x86/amd-memory-encryption.rst.

Currently, this ioctl is used for issuing both Secure Encrypted Virtualization

(SEV) commands on AMD Processors and Trusted Domain Extensions (TDX) commands

on Intel Processors.  The detailed commands are defined in

Documentation/virt/kvm/x86/amd-memory-encryption.rst and

Documentation/virt/kvm/x86/intel-tdx.rst.

4.111 KVM_MEMORY_ENCRYPT_REG_REGION

-----------------------------------

  #define KVM_SYSTEM_EVENT_WAKEUP         4

  #define KVM_SYSTEM_EVENT_SUSPEND        5

  #define KVM_SYSTEM_EVENT_SEV_TERM       6

  #define KVM_SYSTEM_EVENT_TDX_FATAL      7

			__u32 type;

                        __u32 ndata;

                        __u64 data[16];

   reset/shutdown of the VM.

 - KVM_SYSTEM_EVENT_SEV_TERM -- an AMD SEV guest requested termination.

   The guest physical address of the guest's GHCB is stored in `data[0]`.

 - KVM_SYSTEM_EVENT_TDX_FATAL -- a TDX guest reported a fatal error state.

   KVM doesn't do any parsing or conversion, it just dumps 16 general-purpose

   registers to userspace, in ascending order of the 4-bit indices for x86-64

   general-purpose registers in instruction encoding, as defined in the Intel

   SDM.

 - KVM_SYSTEM_EVENT_WAKEUP -- the exiting vCPU is in a suspended state and

   KVM has recognized a wakeup event. Userspace may honor this event by

   marking the exiting vCPU as runnable, or deny it and call KVM_RUN again.

                                    and 0x489), as KVM does now allow them to

                                    be set by userspace (KVM sets them based on

                                    guest CPUID, for safety purposes).

KVM_X86_QUIRK_IGNORE_GUEST_PAT      By default, on Intel platforms, KVM ignores

                                    guest PAT and forces the effective memory

                                    type to WB in EPT.  The quirk is not available

                                    on Intel platforms which are incapable of

                                    safely honoring guest PAT (i.e., without CPU

                                    self-snoop, KVM always ignores guest PAT and

                                    forces effective memory type to WB).  It is

                                    also ignored on AMD platforms or, on Intel,

                                    when a VM has non-coherent DMA devices

                                    assigned; KVM always honors guest PAT in

                                    such case. The quirk is needed to avoid

                                    slowdowns on certain Intel Xeon platforms

                                    (e.g. ICX, SPR) where self-snoop feature is

                                    supported but UC is slow enough to cause

                                    issues with some older guests that use

                                    UC instead of WC to map the video RAM.

                                    Userspace can disable the quirk to honor

                                    guest PAT if it knows that there is no such

                                    guest software, for example if it does not

                                    expose a bochs graphics device (which is

                                    known to have had a buggy driver).

=================================== ============================================

7.32 KVM_CAP_MAX_VCPU_ID

   cpuid

   errata

   hypercalls

   intel-tdx

   mmu

   msr

   nested-vmx

.. SPDX-License-Identifier: GPL-2.0

===================================

Intel Trust Domain Extensions (TDX)

===================================

Overview

========

Intel's Trust Domain Extensions (TDX) protect confidential guest VMs from the

host and physical attacks.  A CPU-attested software module called 'the TDX

module' runs inside a new CPU isolated range to provide the functionalities to

manage and run protected VMs, a.k.a, TDX guests or TDs.

Please refer to [1] for the whitepaper, specifications and other resources.

This documentation describes TDX-specific KVM ABIs.  The TDX module needs to be

initialized before it can be used by KVM to run any TDX guests.  The host

core-kernel provides the support of initializing the TDX module, which is

described in the Documentation/arch/x86/tdx.rst.

API description

===============

KVM_MEMORY_ENCRYPT_OP

---------------------

:Type: vm ioctl, vcpu ioctl

For TDX operations, KVM_MEMORY_ENCRYPT_OP is re-purposed to be generic

ioctl with TDX specific sub-ioctl() commands.

::

  /* Trust Domain Extensions sub-ioctl() commands. */

  enum kvm_tdx_cmd_id {

          KVM_TDX_CAPABILITIES = 0,

          KVM_TDX_INIT_VM,

          KVM_TDX_INIT_VCPU,

          KVM_TDX_INIT_MEM_REGION,

          KVM_TDX_FINALIZE_VM,

          KVM_TDX_GET_CPUID,

          KVM_TDX_CMD_NR_MAX,

  };

  struct kvm_tdx_cmd {

        /* enum kvm_tdx_cmd_id */

        __u32 id;

        /* flags for sub-command. If sub-command doesn't use this, set zero. */

        __u32 flags;

        /*

         * data for each sub-command. An immediate or a pointer to the actual

         * data in process virtual address.  If sub-command doesn't use it,

         * set zero.

         */

        __u64 data;

        /*

         * Auxiliary error code.  The sub-command may return TDX SEAMCALL

         * status code in addition to -Exxx.

         */

        __u64 hw_error;

  };

KVM_TDX_CAPABILITIES

--------------------

:Type: vm ioctl

:Returns: 0 on success, <0 on error

Return the TDX capabilities that current KVM supports with the specific TDX

module loaded in the system.  It reports what features/capabilities are allowed

to be configured to the TDX guest.

- id: KVM_TDX_CAPABILITIES

- flags: must be 0

- data: pointer to struct kvm_tdx_capabilities

- hw_error: must be 0

::

  struct kvm_tdx_capabilities {

        __u64 supported_attrs;

        __u64 supported_xfam;

        __u64 reserved[254];

        /* Configurable CPUID bits for userspace */

        struct kvm_cpuid2 cpuid;

  };

KVM_TDX_INIT_VM

---------------

:Type: vm ioctl

:Returns: 0 on success, <0 on error

Perform TDX specific VM initialization.  This needs to be called after

KVM_CREATE_VM and before creating any VCPUs.

- id: KVM_TDX_INIT_VM

- flags: must be 0

- data: pointer to struct kvm_tdx_init_vm

- hw_error: must be 0

::

  struct kvm_tdx_init_vm {

          __u64 attributes;

          __u64 xfam;

          __u64 mrconfigid[6];          /* sha384 digest */

          __u64 mrowner[6];             /* sha384 digest */

          __u64 mrownerconfig[6];       /* sha384 digest */

          /* The total space for TD_PARAMS before the CPUIDs is 256 bytes */

          __u64 reserved[12];

        /*

         * Call KVM_TDX_INIT_VM before vcpu creation, thus before

         * KVM_SET_CPUID2.

         * This configuration supersedes KVM_SET_CPUID2s for VCPUs because the

         * TDX module directly virtualizes those CPUIDs without VMM.  The user

         * space VMM, e.g. qemu, should make KVM_SET_CPUID2 consistent with

         * those values.  If it doesn't, KVM may have wrong idea of vCPUIDs of

         * the guest, and KVM may wrongly emulate CPUIDs or MSRs that the TDX

         * module doesn't virtualize.

         */

          struct kvm_cpuid2 cpuid;

  };

KVM_TDX_INIT_VCPU

-----------------

:Type: vcpu ioctl

:Returns: 0 on success, <0 on error

Perform TDX specific VCPU initialization.

- id: KVM_TDX_INIT_VCPU

- flags: must be 0

- data: initial value of the guest TD VCPU RCX

- hw_error: must be 0

KVM_TDX_INIT_MEM_REGION

-----------------------

:Type: vcpu ioctl

:Returns: 0 on success, <0 on error

Initialize @nr_pages TDX guest private memory starting from @gpa with userspace

provided data from @source_addr.

Note, before calling this sub command, memory attribute of the range

[gpa, gpa + nr_pages] needs to be private.  Userspace can use

KVM_SET_MEMORY_ATTRIBUTES to set the attribute.

If KVM_TDX_MEASURE_MEMORY_REGION flag is specified, it also extends measurement.

- id: KVM_TDX_INIT_MEM_REGION

- flags: currently only KVM_TDX_MEASURE_MEMORY_REGION is defined

- data: pointer to struct kvm_tdx_init_mem_region

- hw_error: must be 0

::

  #define KVM_TDX_MEASURE_MEMORY_REGION   (1UL << 0)

  struct kvm_tdx_init_mem_region {

          __u64 source_addr;

          __u64 gpa;

          __u64 nr_pages;

  };

KVM_TDX_FINALIZE_VM

-------------------

:Type: vm ioctl

:Returns: 0 on success, <0 on error

Complete measurement of the initial TD contents and mark it ready to run.

- id: KVM_TDX_FINALIZE_VM

- flags: must be 0

- data: must be 0

- hw_error: must be 0

KVM_TDX_GET_CPUID

-----------------

:Type: vcpu ioctl

:Returns: 0 on success, <0 on error

Get the CPUID values that the TDX module virtualizes for the TD guest.

When it returns -E2BIG, the user space should allocate a larger buffer and

retry. The minimum buffer size is updated in the nent field of the

struct kvm_cpuid2.

- id: KVM_TDX_GET_CPUID

- flags: must be 0

- data: pointer to struct kvm_cpuid2 (in/out)

- hw_error: must be 0 (out)

::

  struct kvm_cpuid2 {

	  __u32 nent;

	  __u32 padding;

	  struct kvm_cpuid_entry2 entries[0];

  };

  struct kvm_cpuid_entry2 {

	  __u32 function;

	  __u32 index;

	  __u32 flags;

	  __u32 eax;

	  __u32 ebx;

	  __u32 ecx;

	  __u32 edx;

	  __u32 padding[3];

  };

KVM TDX creation flow

=====================

In addition to the standard KVM flow, new TDX ioctls need to be called.  The

control flow is as follows:

#. Check system wide capability

   * KVM_CAP_VM_TYPES: Check if VM type is supported and if KVM_X86_TDX_VM

     is supported.

#. Create VM

   * KVM_CREATE_VM

   * KVM_TDX_CAPABILITIES: Query TDX capabilities for creating TDX guests.

   * KVM_CHECK_EXTENSION(KVM_CAP_MAX_VCPUS): Query maximum VCPUs the TD can

     support at VM level (TDX has its own limitation on this).

   * KVM_SET_TSC_KHZ: Configure TD's TSC frequency if a different TSC frequency

     than host is desired.  This is Optional.

   * KVM_TDX_INIT_VM: Pass TDX specific VM parameters.

#. Create VCPU

   * KVM_CREATE_VCPU

   * KVM_TDX_INIT_VCPU: Pass TDX specific VCPU parameters.

   * KVM_SET_CPUID2: Configure TD's CPUIDs.

   * KVM_SET_MSRS: Configure TD's MSRs.

#. Initialize initial guest memory

   * Prepare content of initial guest memory.

   * KVM_TDX_INIT_MEM_REGION: Add initial guest memory.

   * KVM_TDX_FINALIZE_VM: Finalize the measurement of the TDX guest.

#. Run VCPU

References

==========

.. [1] https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/documentation.html

KVM_X86_OP(vcpu_after_set_cpuid)

KVM_X86_OP(vm_init)

KVM_X86_OP_OPTIONAL(vm_destroy)

KVM_X86_OP_OPTIONAL(vm_pre_destroy)

KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)

KVM_X86_OP(vcpu_create)

KVM_X86_OP(vcpu_free)

KVM_X86_OP_OPTIONAL(apicv_pre_state_restore)

KVM_X86_OP_OPTIONAL(apicv_post_state_restore)

KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt)

KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt)

KVM_X86_OP_OPTIONAL(set_hv_timer)

KVM_X86_OP_OPTIONAL(cancel_hv_timer)

KVM_X86_OP(setup_mce)

KVM_X86_OP(enable_smi_window)

#endif

KVM_X86_OP_OPTIONAL(dev_get_attr)

KVM_X86_OP_OPTIONAL(mem_enc_ioctl)

KVM_X86_OP(mem_enc_ioctl)

KVM_X86_OP_OPTIONAL(vcpu_mem_enc_ioctl)

KVM_X86_OP_OPTIONAL(mem_enc_register_region)

KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)

KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)

struct kvm_pmu_ops;

enum {

	KVM_DEBUGREG_BP_ENABLED = 1,

	KVM_DEBUGREG_WONT_EXIT = 2,

	KVM_DEBUGREG_BP_ENABLED		= BIT(0),

	KVM_DEBUGREG_WONT_EXIT		= BIT(1),

	/*

	 * Guest debug registers (DR0-3, DR6 and DR7) are saved/restored by

	 * hardware on exit from or enter to guest. KVM needn't switch them.

	 * DR0-3, DR6 and DR7 are set to their architectural INIT value on VM

	 * exit, host values need to be restored.

	 */

	KVM_DEBUGREG_AUTO_SWITCH	= BIT(2),

};

struct kvm_mtrr {

	struct kvm_mmu_memory_cache split_desc_cache;

	gfn_t gfn_direct_bits;

	/*

	 * Size of the CPU's dirty log buffer, i.e. VMX's PML buffer. A Zero

	 * value indicates CPU dirty logging is unsupported or disabled in

	 * current VM.

	 */

	int cpu_dirty_log_size;

};

struct kvm_vm_stat {

	unsigned int vm_size;

	int (*vm_init)(struct kvm *kvm);

	void (*vm_destroy)(struct kvm *kvm);

	void (*vm_pre_destroy)(struct kvm *kvm);

	/* Create, but do not attach this VCPU */

	int (*vcpu_precreate)(struct kvm *kvm);

			       struct x86_exception *exception);

	void (*handle_exit_irqoff)(struct kvm_vcpu *vcpu);

	/*

	 * Size of the CPU's dirty log buffer, i.e. VMX's PML buffer.  A zero

	 * value indicates CPU dirty logging is unsupported or disabled.

	 */

	int cpu_dirty_log_size;

	void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu);

	const struct kvm_x86_nested_ops *nested_ops;

	void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);

	void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);

	bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);

	bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu);

	int (*set_hv_timer)(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc,

			    bool *expired);

	int (*dev_get_attr)(u32 group, u64 attr, u64 *val);

	int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);

	int (*vcpu_mem_enc_ioctl)(struct kvm_vcpu *vcpu, void __user *argp);

	int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);

	int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);

	int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);

int kvm_add_user_return_msr(u32 msr);

int kvm_find_user_return_msr(u32 msr);

int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask);

void kvm_user_return_msr_update_cache(unsigned int index, u64 val);

static inline bool kvm_is_supported_user_return_msr(u32 msr)

{

	 KVM_X86_QUIRK_FIX_HYPERCALL_INSN |	\

	 KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS |	\

	 KVM_X86_QUIRK_SLOT_ZAP_ALL |		\

	 KVM_X86_QUIRK_STUFF_FEATURE_MSRS)

	 KVM_X86_QUIRK_STUFF_FEATURE_MSRS |	\

	 KVM_X86_QUIRK_IGNORE_GUEST_PAT)

#define KVM_X86_CONDITIONAL_QUIRKS		\

	(KVM_X86_QUIRK_CD_NW_CLEARED |		\

	 KVM_X86_QUIRK_IGNORE_GUEST_PAT)

/*

 * KVM previously used a u32 field in kvm_run to indicate the hypercall was