Commit 360a1f3e authored by Tomasz Lis's avatar Tomasz Lis Committed by Michal Wajdeczko
Browse files

drm/xe/vf: Document SRIOV VF restore flow 



This adds a documentation chapter, containing high level flow
of VF restore procedure.

v2: Better describe initial conditions, include GuC states on
  sequence diagram (Michal)
v3: moved DOC to .c (Michal)

Signed-off-by: default avatarTomasz Lis <tomasz.lis@intel.com>
Reviewed-by: default avatarMichal Wajdeczko <michal.wajdeczko@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20241104213449.1455694-3-tomasz.lis@intel.com


Signed-off-by: default avatarMichal Wajdeczko <michal.wajdeczko@intel.com>
parent 6e6d7b41

drivers/gpu/drm/xe/xe_sriov_vf.c

+107 −0
Original line number Diff line number Diff line
@@ -12,6 +12,113 @@ 
#include "xe_sriov_printk.h"

#include "xe_sriov_vf.h"



/**

 * DOC: VF restore procedure in PF KMD and VF KMD

 *

 * Restoring previously saved state of a VF is one of core features of

 * SR-IOV. All major VM Management applications allow saving and restoring

 * the VM state, and doing that to a VM which uses SRIOV VF as one of

 * the accessible devices requires support from KMD on both PF and VF side.

 * VMM initiates all required operations through VFIO module, which then

 * translates them into PF KMD calls. This description will focus on these

 * calls, leaving out the module which initiates these steps (VFIO).

 *

 * In order to start the restore procedure, GuC needs to keep the VF in

 * proper state. The PF driver can ensure GuC set it to VF_READY state

 * by provisioning the VF, which in turn can be done after Function Level

 * Reset of said VF (or after it was freshly created - in that case FLR

 * is not needed). The FLR procedure ends with GuC sending message

 * `GUC_PF_NOTIFY_VF_FLR_DONE`, and then provisioning data is sent to GuC.

 * After the provisioning is completed, the VF needs to be paused, and

 * at that point the actual restore can begin.

 *

 * During VF Restore, state of several resources is restored. These may

 * include local memory content (system memory is restored by VMM itself),

 * values of MMIO registers, stateless compression metadata and others.

 * The final resource which also needs restoring is state of the VF

 * submission maintained within GuC. For that, `GUC_PF_OPCODE_VF_RESTORE`

 * message is used, with reference to the state blob to be consumed by

 * GuC.

 *

 * Next, when VFIO is asked to set the VM into running state, the PF driver

 * sends `GUC_PF_TRIGGER_VF_RESUME` to GuC. When sent after restore, this

 * changes VF state within GuC to `VF_RESFIX_BLOCKED` rather than the

 * usual `VF_RUNNING`. At this point GuC triggers an interrupt to inform

 * the VF KMD within the VM that it was migrated.

 *

 * As soon as Virtual GPU of the VM starts, the VF driver within receives

 * the MIGRATED interrupt and schedules post-migration recovery worker.

 * That worker queries GuC for new provisioning (using MMIO communication),

 * and applies fixups to any non-virtualized resources used by the VF.

 *

 * When the VF driver is ready to continue operation on the newly connected

 * hardware, it sends `VF2GUC_NOTIFY_RESFIX_DONE` which causes it to

 * enter the long awaited `VF_RUNNING` state, and therefore start handling

 * CTB messages and scheduling workloads from the VF::

 *

 *      PF                             GuC                              VF

 *     [ ]                              |                               |

 *     [ ] PF2GUC_VF_CONTROL(pause)     |                               |

 *     [ ]---------------------------> [ ]                              |

 *     [ ]                             [ ]  GuC sets new VF state to    |

 *     [ ]                             [ ]------- VF_READY_PAUSED       |

 *     [ ]                             [ ]      |                       |

 *     [ ]                             [ ] <-----                       |

 *     [ ] success                     [ ]                              |

 *     [ ] <---------------------------[ ]                              |

 *     [ ]                              |                               |

 *     [ ] PF loads resources from the  |                               |

 *     [ ]------- saved image supplied  |                               |

 *     [ ]      |                       |                               |

 *     [ ] <-----                       |                               |

 *     [ ]                              |                               |

 *     [ ] GUC_PF_OPCODE_VF_RESTORE     |                               |

 *     [ ]---------------------------> [ ]                              |

 *     [ ]                             [ ]  GuC loads contexts and CTB  |

 *     [ ]                             [ ]------- state from image      |

 *     [ ]                             [ ]      |                       |

 *     [ ]                             [ ] <-----                       |

 *     [ ]                             [ ]                              |

 *     [ ]                             [ ]  GuC sets new VF state to    |

 *     [ ]                             [ ]------- VF_RESFIX_PAUSED      |

 *     [ ]                             [ ]      |                       |

 *     [ ] success                     [ ] <-----                       |

 *     [ ] <---------------------------[ ]                              |

 *     [ ]                              |                               |

 *     [ ] GUC_PF_TRIGGER_VF_RESUME     |                               |

 *     [ ]---------------------------> [ ]                              |

 *     [ ]                             [ ]  GuC sets new VF state to    |

 *     [ ]                             [ ]------- VF_RESFIX_BLOCKED     |

 *     [ ]                             [ ]      |                       |

 *     [ ]                             [ ] <-----                       |

 *     [ ]                             [ ]                              |

 *     [ ]                             [ ] GUC_INTR_SW_INT_0            |

 *     [ ] success                     [ ]---------------------------> [ ]

 *     [ ] <---------------------------[ ]                             [ ]

 *      |                               |      VF2GUC_QUERY_SINGLE_KLV [ ]

 *      |                              [ ] <---------------------------[ ]

 *      |                              [ ]                             [ ]

 *      |                              [ ]        new VF provisioning  [ ]

 *      |                              [ ]---------------------------> [ ]

 *      |                               |                              [ ]

 *      |                               |       VF driver applies post [ ]

 *      |                               |      migration fixups -------[ ]

 *      |                               |                       |      [ ]

 *      |                               |                       -----> [ ]

 *      |                               |                              [ ]

 *      |                               |    VF2GUC_NOTIFY_RESFIX_DONE [ ]

 *      |                              [ ] <---------------------------[ ]

 *      |                              [ ]                             [ ]

 *      |                              [ ]  GuC sets new VF state to   [ ]

 *      |                              [ ]------- VF_RUNNING           [ ]

 *      |                              [ ]      |                      [ ]

 *      |                              [ ] <-----                      [ ]

 *      |                              [ ]                     success [ ]

 *      |                              [ ]---------------------------> [ ]

 *      |                               |                               |

 *      |                               |                               |

 */



static void migration_worker_func(struct work_struct *w);



/**