diff --git a/doc/develop.rst b/doc/develop.rst index 6c2f8326a..ee57499ba 100644 --- a/doc/develop.rst +++ b/doc/develop.rst @@ -74,6 +74,7 @@ Advanced Features tutorials/nvmx_virtualization tutorials/vuart_configuration tutorials/rdt_configuration + tutorials/vcat_configuration tutorials/waag-secure-boot tutorials/enable_s5 tutorials/cpu_sharing diff --git a/doc/developer-guides/hld/hld-hypervisor.rst b/doc/developer-guides/hld/hld-hypervisor.rst index 2e0f8fca0..276c4f301 100644 --- a/doc/developer-guides/hld/hld-hypervisor.rst +++ b/doc/developer-guides/hld/hld-hypervisor.rst @@ -25,4 +25,5 @@ Hypervisor High-Level Design Hypercall / HSM upcall Compile-time configuration RDT support + vCAT support Split-locked Access handling diff --git a/doc/developer-guides/hld/hv-vcat.rst b/doc/developer-guides/hld/hv-vcat.rst new file mode 100644 index 000000000..0d707e7f2 --- /dev/null +++ b/doc/developer-guides/hld/hv-vcat.rst @@ -0,0 +1,139 @@ +.. _hv_vcat: + +Enable vCAT +########### + +vCAT refers to the virtualization of Cache Allocation Technology (CAT), one of the +RDT (Resource Director Technology) technologies. + +ACRN vCAT is built on top of ACRN RDT: ACRN RDT provides a number of physical CAT resources +(COS IDs + cache ways), ACRN vCAT exposes some number of virtual CAT resources to VMs +and then transparently map them to the assigned physical CAT resources in the ACRN hypervisor; +VM can take advantage of vCAT to prioritize and partition virtual cache ways for its own tasks. + +In current CAT implementation, one COS ID corresponds to one ``IA32_type_MASK_n`` (type: L2 or L3, +n ranges from 0 to ``MAX_CACHE_CLOS_NUM_ENTRIES`` - 1) MSR and a bit in a capacity bitmask (CBM) +corresponds to one cache way. + +On current generation systems, normally L3 cache is shared by all CPU cores on the same socket and +L2 cache is generally just shared by the hyperthreads on a core. But when dealing with ACRN +vCAT COS IDs assignment, it is currently assumed that all the L2/L3 caches (and therefore all COS IDs) +are system-wide caches shared by all cores in the system, this is done for convenience and to simplify +the vCAT configuration process. If vCAT is enabled for a VM (abbreviated as vCAT VM), there should not +be any COS ID overlap between a vCAT VM and any other VMs. e.g. the vCAT VM has exclusive use of the +assigned COS IDs. +When assigning cache ways, however, the VM can be given exclusive, shared, or mixed access to the cache +ways depending on particular performance needs. For example, use dedicated cache ways for RTVM, and use +shared cache ways between low priority VMs. + +In ACRN, the CAT resources allocated for vCAT VMs are determined in :ref:`vcat_configuration`. + +For further details on the RDT, refer to the ACRN RDT high-level design :ref:`hv_rdt`. + + +High Level ACRN vCAT Design +*************************** + +ACRN CAT virtualization support can be divided into two parts: + +- CAT Capability Exposure to Guest VM + +- CAT resources (COS IDs + cache ways) management + +The figure below shows high-level design of vCAT in ACRN: + + .. figure:: images/vcat-hld.png + :align: center + +CAT Capability Exposure to Guest VM +*********************************** +ACRN exposes CAT capability and resource to a Guest VM via vCPUID and vMSR, as explained +in the following sections. + +vCPUID +====== + +CPUID Leaf 07H +-------------- + +- CPUID.(EAX=07H, ECX=0).EBX.PQE[bit 15]: Supports RDT capability if 1. This bit will be set for a vCAT VM. + +CPUID Leaf 10H +-------------- + +**CAT Resource Type and Capability Enumeration** + +- CPUID.(EAX=10H, ECX=0):EBX[1]: If 1, indicate L3 CAT support for a vCAT VM. +- CPUID.(EAX=10H, ECX=0):EBX[2]: If 1, indicate L2 CAT support for a vCAT VM. +- CPUID.(EAX=10H, ECX=1): CAT capability enumeration sub-leaf for L3. Reports L3 COS_MAX and CBM_LEN to a vCAT VM +- CPUID.(EAX=10H, ECX=2): CAT capability enumeration sub-leaf for L2. Reports L2 COS_MAX and CBM_LEN to a vCAT VM + +vMSR +==== + +The following CAT MSRs will be virtualized for a vCAT VM: + +- IA32_PQR_ASSOC +- IA32_type_MASK_0 ~ IA32_type_MASK_n + +By default, after reset, all CPU cores are assigned to COS 0 and all IA32_type_MASK_n MSRs +are programmed to allow fill into all cache ways. + + +CAT resources (COS IDs + cache ways) management +************************************************ + +All accesses to the CAT MSRs are intercepted by vMSR and control is passed to vCAT, which will perform +the following actions: + +- Intercept IA32_PQR_ASSOC MSR to re-map virtual COS ID to physical COS ID. + Upon writes, store the re-mapped physical COS ID into its vCPU ``msr_store_area`` + data structure guest part. It will be loaded to physical IA32_PQR_ASSOC on each VM-Enter. + + +- Intercept IA32_type_MASK_n MSRs to re-map virtual CBM to physical CBM. Upon writes, + program re-mapped physical CBM into corresponding physical IA32_type_MASK_n MSR + + Several vCAT P2V (physical to virtual) and V2P (virtual to physical) + mappings exist, as illustrated in the following pseudocode: + +.. code-block:: none + + struct acrn_vm_config *vm_config = get_vm_config(vm_id) + + max_pcbm = vm_config->max_type_pcbm (type: l2 or l3) + mask_shift = ffs64(max_pcbm) + + vcosid = vmsr - MSR_IA32_type_MASK_0 + pcosid = vm_config->pclosids[vcosid] + + pmsr = MSR_IA32_type_MASK_0 + pcosid + pcbm = vcbm << mask_shift + vcbm = pcbm >> mask_shift + +Where + ``vm_config->pclosids[]``: array of physical COS IDs, where each corresponds to one ``vcpu_clos`` that + is defined in the scenario file + + ``max_pcbm``: a bitmask that selects all the physical cache ways assigned to the VM, corresponds to + the nth ``CLOS_MASK`` that is defined in scenario file, where n = the first physical COS ID assigned + = ``vm_config->pclosids[0]`` + + ``ffs64(max_pcbm)``: find the first (least significant) bit set in ``max_pcbm`` and return + the index of that bit. + + ``MSR_IA32_type_MASK_0``: 0xD10 for L2, 0xC90 for L3 + + ``vcosid``: virtual COS ID, always starts from 0 + + ``pcosid``: corresponding physical COS ID for a given ``vcosid`` + + ``vmsr``: virtual MSR address, passed to vCAT handlers by the + caller functions ``rdmsr_vmexit_handler()``/``wrmsr_vmexit_handler()`` + + ``pmsr``: physical MSR address + + ``vcbm``: virtual CBM, passed to vCAT handlers by the + caller functions ``rdmsr_vmexit_handler()``/``wrmsr_vmexit_handler()`` + + ``pcbm``: physical CBM diff --git a/doc/developer-guides/hld/images/vcat-hld.png b/doc/developer-guides/hld/images/vcat-hld.png new file mode 100644 index 000000000..75da1d7a3 Binary files /dev/null and b/doc/developer-guides/hld/images/vcat-hld.png differ diff --git a/doc/tutorials/vcat_configuration.rst b/doc/tutorials/vcat_configuration.rst new file mode 100644 index 000000000..1e333be54 --- /dev/null +++ b/doc/tutorials/vcat_configuration.rst @@ -0,0 +1,89 @@ +.. _vcat_configuration: + +Enable vCAT Configuration +######################### + +vCAT is built on top of RDT, so to use vCAT we must first enable RDT. +For details on enabling RDT configuration on ACRN, see :ref:`rdt_configuration`. +For details on ACRN vCAT high-level design, see :ref:`hv_vcat`. + +The vCAT feature is disabled by default in ACRN. You can enable vCAT via the UI, +the steps listed below serve as an FYI to show how those settings are translated +into XML in the scenario file: + +#. Configure system level features: + + - Edit :option:`hv.FEATURES.RDT.RDT_ENABLED` to `y` to enable RDT + + - Edit :option:`hv.FEATURES.RDT.CDP_ENABLED` to `n` to disable CDP. + Currently vCAT requires CDP to be disabled. + + - Edit :option:`hv.FEATURES.RDT.VCAT_ENABLED` to `y` to enable vCAT + + .. code-block:: xml + :emphasize-lines: 3,4,5 + + + + y + n + y + + + + +#. In each Guest VM configuration: + + - Edit :option:`vm.guest_flags.guest_flag` and add ``GUEST_FLAG_VCAT_ENABLED`` + to enable the vCAT feature on the VM. + + - Edit :option:`vm.clos.vcpu_clos` to assign COS IDs to the VM. + + If ``GUEST_FLAG_VCAT_ENABLED`` is not specified for a VM (abbreviated as RDT VM): + ``vcpu_clos`` is per CPU in a VM and it configures each CPU in a VM to a desired COS ID. + So the number of vcpu_closes is equal to the number of vCPUs assigned. + + If ``GUEST_FLAG_VCAT_ENABLED`` is specified for a VM (abbreviated as vCAT VM): + ``vcpu_clos`` is not per CPU anymore; instead, it specifies a list of physical COS IDs (minimum 2) + that are assigned to a vCAT VM. The number of vcpu_closes is not necessarily equal to + the number of vCPUs assigned, but may be not only greater than the number of vCPUs assigned but + less than this number. Each vcpu_clos will be mapped to a virtual COS ID, the first vcpu_clos + is mapped to virtual COS ID 0 and the second is mapped to virtual COS ID 1, etc. + + .. code-block:: xml + :emphasize-lines: 3,10,11,12,13 + + + + GUEST_FLAG_VCAT_ENABLED + + + 1 + 2 + + + 2 + 4 + 5 + 7 + + + + .. note:: + CLOS_MASK defined in scenario file is a capacity bitmask (CBM) starting + at bit position low (the lowest assigned physical cache way) and ending at position + high (the highest assigned physical cache way, inclusive). As CBM only allows + contiguous '1' combinations, so CLOS_MASK essentially is the maximum CBM that covers + all the physical cache ways assigned to a vCAT VM. + + The config tool imposes oversight to prevent any problems with invalid configuration data for vCAT VMs: + + * For a vCAT VM, its vcpu_closes cannot be set to 0, COS ID 0 is reserved to be used only by hypervisor + + * There should not be any COS ID overlap between a vCAT VM and any other VMs. e.g. the vCAT VM has exclusive use of the assigned COS IDs + + * For a vCAT VM, each vcpu_clos must be less than L2/L3 COS_MAX + + * For a vCAT VM, its vcpu_closes cannot contain duplicate values + +#. Follow instructions in :ref:`gsg` and build with this XML configuration.