.. _GVT-G-porting: GVT-G Enabling and Porting Guide ################################ Introduction ************ GVT-g is Intel's Open Source GPU virtualization solution, up-streamed to the Linux kernel. Its implementation over KVM is named KVMGT, over Xen is named XenGT, and over ACRN is named ACRN-GT. GVT-g can export multiple virtual-GPU (vGPU) instances for virtual machine (VM) system. A VM can be assigned one instance of a vGPU. The guest OS graphic driver needs only minor modifications to drive the vGPU adapter in a VM. Every vGPU instance adopts the full HW GPU's acceleration capability for media, 3D renderingi, and display. ACRN-GT refers to the glue layer between the ACRN hypervisor and GVT-g core device model. It works as the agent of hypervisor-related services. It is the only layer that needs to be rewritten when porting GVT-g to other specific hypervisors. For simplicity, in the rest of this document, the term GVT is used to refer to the core device model component of GVT-g, specifically corresponding to ``gvt.ko`` when build as a module. Purpose of this document ************************ This document explains the relationship between components of GVT-g in the ACRN hypervisor, shows how to enable GVT-g on ACRN, and guides developers porting GVT-g to work on other hypervisors. This document describes: - the overall components of GVT-g - interaction interface of each components - core interaction scenarios APIs of each component interface can be found in the :ref:`GVT-g_api` documentation. Overall Components ****************** For the GVT-g solution for the ACRN hypervisor, there are two key modules: ACRN-GT and GVT: ACRN-GT module Compiled from ``drivers/gpu/drm/i915/gvt/acrn_gvt.c``, the ACRN-GT module acts as a glue layer between the ACRN hypervisor and the interface to the ACRN-DM in user space. ACRN-GT is the agent of hypervisor related services, including I/O trap request, IRQ injection, address translation, VM controls, etc. It also listens to ACRN hypervisor in ``acrngt_emulation_thread``, and informs GVT module of I/O traps. It calls into the GVT module's :ref:`intel_gvt_ops_interface` to invoke Device Model's routines, and receives request from GVT module through the :ref:`MPT_interface`. User space programs, such as ACRN-DM, communicate with ACRN-GT through the :ref:`sysfs_interface` by writing to sysfs node ``/sys/kernel/gvt/control/create_gvt_instance``. This is the only module that needs to be rewritten when porting to another embedded device hypervisor. GVT module This Device Model service is the central part of all GVT-g components. It receives workloads from each vGPU, shadows the workloads, and dispatches the workloads to Dom0's i915 module to deliver workloads to real hardware. It also emulates the virtual display to each VM. VHM module This is a kernel module that requires an interrupt (vIRQ) number, and exposes APIs to external kernel modules such as GVT-g, and the virtIO BE service running in kernel space. It exposes a char device node in user space, and only interacts with the DM. The DM routes I/O requests and responses between other modules to and from the VHM module via the char device. DM may use the VHM for hypervisor service (including remote memory map), and VHM may directly service the request such as for the remote memory map, or invoking hypercall. It also sends I/O responses to user space modules, notified by vIRQ injections. .. figure:: images/GVT-G-porting-image1.png :width: 700px :align: center :name: GVT-g_components GVT-g components and interfaces Core scenario interaction sequences *********************************** vGPU creation scenario ====================== In this scenario, ACRN-GT receives a create request from ACRN-DM. It calls GVT's :ref:`intel_gvt_ops_interface` to inform GVT of vGPU creation. This interface sets up all vGPU resources such as MMIO, GMA, PVINFO, GTT, DISPLAY, and Execlists, and calls back to the ACRN-GT module through the :ref:`MPT_interface` ``attach_vgpu``. Then, the ACRN-GT module sets up an I/O request server and asks to trap the PCI configure space of the vGPU (virtual device 0:2:0) via VHM's APIs. Finally, the ACRN-GT module launches a ACRN-GT emulation thread to listen to I/O trap notifications from HVM and ACRN hypervisor. vGPU destroy scenario ===================== In this scenario, ACRN-GT receives a destroy request from ACRN-DM. It calls GVT's :ref:`intel_gvt_ops_interface` to inform GVT of the vGPU destroy request, and cleans up all vGPU resources. vGPU pci configure space write scenario ======================================= ACRN traps the vGPU's PCI config space write, notifies ACRN-GT's ``acrngt_emulation_thread``, which calls ``acrngt_hvm_pio_emulation`` to handle all I/O trap notifications. This routine calls the GVT's :ref:`intel_gvt_ops_interface` ``emulate_cfg_write`` to emulate the vGPU PCI config space write: #. If it's BAR0 (GTTMMIO) write, turn on/off GTTMMIO trap, according to the write value. #. If it's BAR1 (Aperture) write, maps/unmaps vGPU's aperture to its corresponding part in the host's aperture. #. Otherwise, write to the virtual PCI configuration space of the vGPU. pci configure space read scenario ================================= Call sequence is almost the same as the write scenario above, but instead it calls the GVT's :ref:`intel_gvt_ops_interface` ``emulate_cfg_read`` to emulate the vGPU PCI config space read. GGTT read/write scenario ======================== GGTT's trap is set up in the pci configure space write scenario above. MMIO read/write scenario ======================== MMIO's trap is set up in the pci configure space write scenario above. PPGTT write protection page set/unset scenario ============================================== PPGTT write protection page is set by calling ``acrn_ioreq_add_iorange`` with range type as ``REQ_WP``, and trap its write to device model while allowing read without trap. PPGTT write protection page is unset by calling ``acrn_ioreq_del_range``. PPGTT write protection page write ================================= In the VHM module, ioreq for PPGTT WP and MMIO trap is the same. It will also be trapped into the routine ``intel_vgpu_emulate_mmio_write()``. API details *********** APIs of each component interface can be found in the :ref:`GVT-g_api` documentation.