doc: terminology cleanup in hv-dev-passthrough.rst
- Change UOS and SOS to User VM and Service VM, respectively - Change guest to VM or similar depending on context - Clean up some of the grammar Signed-off-by: Amy Reyes <amy.reyes@intel.com>
|
@ -21,10 +21,10 @@ discussed here.
|
|||
--------
|
||||
|
||||
In the ACRN project, device emulation means emulating all existing
|
||||
hardware resource through a software component device model running in
|
||||
the Service OS (SOS). Device emulation must maintain the same SW
|
||||
hardware resources through a software component device model running in
|
||||
the Service VM. Device emulation must maintain the same SW
|
||||
interface as a native device, providing transparency to the VM software
|
||||
stack. Passthrough implemented in hypervisor assigns a physical device
|
||||
stack. Passthrough implemented in the hypervisor assigns a physical device
|
||||
to a VM so the VM can access the hardware device directly with minimal
|
||||
(if any) VMM involvement.
|
||||
|
||||
|
@ -38,23 +38,23 @@ can't support device sharing.
|
|||
:align: center
|
||||
:name: emu-passthru-diff
|
||||
|
||||
Difference between Emulation and passthrough
|
||||
Difference between emulation and passthrough
|
||||
|
||||
Passthrough in the hypervisor provides the following functionalities to
|
||||
allow VM to access PCI devices directly:
|
||||
allow the VM to access PCI devices directly:
|
||||
|
||||
- VT-d DMA Remapping for PCI devices: hypervisor will setup DMA
|
||||
- VT-d DMA remapping for PCI devices: hypervisor will set up DMA
|
||||
remapping during VM initialization phase.
|
||||
- VT-d Interrupt-remapping for PCI devices: hypervisor will enable
|
||||
- VT-d interrupt-remapping for PCI devices: hypervisor will enable
|
||||
VT-d interrupt-remapping for PCI devices for security considerations.
|
||||
- MMIO Remapping between virtual and physical BAR
|
||||
- Device configuration Emulation
|
||||
- MMIO remapping between virtual and physical BAR
|
||||
- Device configuration emulation
|
||||
- Remapping interrupts for PCI devices
|
||||
- ACPI configuration Virtualization
|
||||
- ACPI configuration virtualization
|
||||
- GSI sharing violation check
|
||||
|
||||
The following diagram details passthrough initialization control flow in ACRN
|
||||
for post-launched VM:
|
||||
The following diagram details the passthrough initialization control flow in
|
||||
ACRN for a post-launched VM:
|
||||
|
||||
.. figure:: images/passthru-image22.png
|
||||
:align: center
|
||||
|
@ -70,59 +70,61 @@ passthrough, as detailed here:
|
|||
.. figure:: images/passthru-image77.png
|
||||
:align: center
|
||||
|
||||
Passthrough Device Status
|
||||
Passthrough device status
|
||||
|
||||
Owner of Passthrough Devices
|
||||
****************************
|
||||
|
||||
ACRN hypervisor will do PCI enumeration to discover the PCI devices on the platform.
|
||||
According to the hypervisor/VM configurations, the owner of these PCI devices can be
|
||||
one the following 4 cases:
|
||||
ACRN hypervisor will do PCI enumeration to discover the PCI devices on the
|
||||
platform. According to the hypervisor/VM configurations, the owner of these PCI
|
||||
devices can be one of the following 4 cases:
|
||||
|
||||
- **Hypervisor**: hypervisor uses UART device as the console in debug version for
|
||||
debug purpose, so the UART device is owned by hypervisor and is not visible
|
||||
to any VM. For now, UART is the only pci device could be owned by hypervisor.
|
||||
- **Pre-launched VM**: The passthrough devices will be used in a pre-launched VM is
|
||||
predefined in VM configuration. These passthrough devices are owned by the
|
||||
pre-launched VM after the VM is created. These devices will not be removed
|
||||
from the pre-launched VM. There could be pre-launched VM(s) in partitioned
|
||||
mode and hybrid mode.
|
||||
- **Service VM**: All the passthrough devices except these described above (owned by
|
||||
hypervisor or pre-launched VM(s)) are assigned to Service VM. And some of these devices
|
||||
can be assigned to a post-launched VM according to the passthrough device list
|
||||
specified in the parameters of the ACRN DM.
|
||||
- **Post-launched VM**: A list of passthrough devices can be specified in the parameters of
|
||||
the ACRN DM. When creating a post-launched VM, these specified devices will be moved
|
||||
from Service VM domain to the post-launched VM domain. After the post-launched VM is
|
||||
powered-off, these devices will be moved back to Service VM domain.
|
||||
- **Hypervisor**: Hypervisor uses a UART device as the console in debug version
|
||||
for debugging purposes, so the UART device is owned by the hypervisor and is
|
||||
not visible to any VM. For now, UART is the only PCI device that can be owned
|
||||
by the hypervisor.
|
||||
- **Pre-launched VM**: The passthrough devices that will be used in a
|
||||
pre-launched VM are predefined in the VM configuration. These passthrough
|
||||
devices are owned by the pre-launched VM after the VM is created. These
|
||||
devices will not be removed from the pre-launched VM. There can be
|
||||
pre-launched VMs in partitioned mode and hybrid mode.
|
||||
- **Service VM**: All the passthrough devices except those described above
|
||||
(owned by hypervisor or pre-launched VMs) are assigned to the Service VM. And
|
||||
some of these devices can be assigned to a post-launched VM according to the
|
||||
passthrough device list specified in the parameters of the ACRN Device Model.
|
||||
- **Post-launched VM**: A list of passthrough devices can be specified in the
|
||||
parameters of the ACRN Device Model. When creating a post-launched VM, these
|
||||
specified devices will be moved from the Service VM domain to the
|
||||
post-launched VM domain. After the post-launched VM is powered-off, these
|
||||
devices will be moved back to the Service VM domain.
|
||||
|
||||
|
||||
VT-d DMA Remapping
|
||||
******************
|
||||
|
||||
To enable passthrough, for VM DMA access the VM can only
|
||||
support GPA, while physical DMA requires HPA. One work-around
|
||||
support GPA, while a physical DMA requires HPA. One work-around
|
||||
is building identity mapping so that GPA is equal to HPA, but this
|
||||
is not recommended as some VM don't support relocation well. To
|
||||
is not recommended as some VMs don't support relocation well. To
|
||||
address this issue, Intel introduces VT-d in the chipset to add one
|
||||
remapping engine to translate GPA to HPA for DMA operations.
|
||||
|
||||
Each VT-d engine (DMAR Unit), maintains a remapping structure
|
||||
Each VT-d engine (DMAR Unit) maintains a remapping structure
|
||||
similar to a page table with device BDF (Bus/Dev/Func) as input and final
|
||||
page table for GPA/HPA translation as output. The GPA/HPA translation
|
||||
page table is similar to a normal multi-level page table.
|
||||
|
||||
VM DMA depends on Intel VT-d to do the translation from GPA to HPA, so we
|
||||
need to enable VT-d IOMMU engine in ACRN before we can passthrough any device. Service VM
|
||||
in ACRN is a VM running in non-root mode which also depends
|
||||
on VT-d to access a device. In Service VM DMA remapping
|
||||
engine settings, GPA is equal to HPA.
|
||||
VM DMA depends on Intel VT-d to do the translation from GPA to HPA, so we need
|
||||
to enable VT-d IOMMU engine in ACRN before we can passthrough any device. The
|
||||
Service VM in ACRN is a VM running in non-root mode which also depends on VT-d
|
||||
to access a device. In Service VM DMA remapping engine settings, GPA is equal to
|
||||
HPA.
|
||||
|
||||
ACRN hypervisor checks DMA-Remapping Hardware unit Definition (DRHD) in
|
||||
host DMAR ACPI table to get basic info, then sets up each DMAR unit. For
|
||||
simplicity, ACRN reuses EPT table as the translation table in DMAR
|
||||
unit for each passthrough device. The control flow of assigning and de-assigning
|
||||
a passthrough device to/from a post-launched VM is shown in the following figures:
|
||||
ACRN hypervisor checks DMA-Remapping Hardware unit Definition (DRHD) in the host
|
||||
DMAR ACPI table to get basic information, then sets up each DMAR unit. For
|
||||
simplicity, ACRN reuses the EPT table as the translation table in the DMAR unit
|
||||
for each passthrough device. The control flow of assigning and deassigning a
|
||||
passthrough device to/from a post-launched VM is shown in the following figures:
|
||||
|
||||
.. figure:: images/passthru-image86.png
|
||||
:align: center
|
||||
|
@ -132,7 +134,7 @@ a passthrough device to/from a post-launched VM is shown in the following figure
|
|||
.. figure:: images/passthru-image42.png
|
||||
:align: center
|
||||
|
||||
ptdev de-assignment control flow
|
||||
ptdev deassignment control flow
|
||||
|
||||
.. _vtd-posted-interrupt:
|
||||
|
||||
|
@ -140,47 +142,44 @@ a passthrough device to/from a post-launched VM is shown in the following figure
|
|||
VT-d Interrupt-Remapping
|
||||
************************
|
||||
|
||||
The VT-d interrupt-remapping architecture enables system software to
|
||||
control and censor external interrupt requests generated by all sources
|
||||
including those from interrupt controllers (I/OxAPICs), MSI/MSI-X capable
|
||||
devices including endpoints, root-ports and Root-Complex integrated
|
||||
end-points.
|
||||
ACRN forces to enabled VT-d interrupt-remapping feature for security reasons.
|
||||
If the VT-d hardware doesn't support interrupt-remapping, then ACRN will
|
||||
refuse to boot VMs.
|
||||
VT-d Interrupt-remapping is NOT related to the translation from physical
|
||||
The VT-d interrupt-remapping architecture enables system software to control and
|
||||
censor external interrupt requests generated by all sources including those from
|
||||
interrupt controllers (I/OxAPICs), MSI/MSI-X capable devices including
|
||||
endpoints, root-ports and Root-Complex integrated end-points. ACRN requires
|
||||
enabling the VT-d interrupt-remapping feature for security reasons. If the VT-d
|
||||
hardware doesn't support interrupt-remapping, then ACRN will refuse to boot VMs.
|
||||
VT-d interrupt-remapping is NOT related to the translation from physical
|
||||
interrupt to virtual interrupt or vice versa. The term VT-d interrupt-remapping
|
||||
remaps the interrupt index in the VT-d interrupt-remapping table to the physical
|
||||
interrupt vector after checking the external interrupt request is valid. Translation
|
||||
physical vector to virtual vector is still needed to be done by hypervisor, which is
|
||||
also described in the below section :ref:`interrupt-remapping`.
|
||||
interrupt vector after checking the external interrupt request is valid. The
|
||||
hypervisor still needs to translate the physical vector to the virtual vector,
|
||||
which is also described in the below section :ref:`interrupt-remapping`.
|
||||
|
||||
VT-d posted interrupt (PI) enables direct delivery of external interrupts from
|
||||
passthrough devices to VMs without having to exit to hypervisor, thereby improving
|
||||
interrupt performance. ACRN uses VT-d posted interrupts if the platform
|
||||
supports them. VT-d distinguishes between remapped
|
||||
and posted interrupt modes by bit 15 in the low 64-bit of the IRTE. If cleared the
|
||||
entry is remapped, if set it's posted.
|
||||
The idea for posted interrupt is to keep a Posted Interrupt Descriptor (PID) in memory.
|
||||
The PID is a 64-byte data structure that contains several fields:
|
||||
passthrough devices to VMs without having to exit to the hypervisor, thereby
|
||||
improving interrupt performance. ACRN uses VT-d posted interrupts if the
|
||||
platform supports them. VT-d distinguishes between remapped and posted interrupt
|
||||
modes by bit 15 in the low 64-bit of the interrupt-remapping table entry. If
|
||||
cleared, the entry is remapped. If set, it's posted. The idea is to keep a
|
||||
Posted Interrupt Descriptor (PID) in memory. The PID is a 64-byte data structure
|
||||
that contains several fields:
|
||||
|
||||
Posted Interrupt Request (PIR):
|
||||
a 256-bit field, one bit per request vector;
|
||||
this is where the interrupts are posted;
|
||||
this is where the interrupts are posted.
|
||||
|
||||
Suppress Notification (SN):
|
||||
determines whether to notify (``SN=0``) or not notify (``SN=1``)
|
||||
the CPU for non-urgent interrupts. For ACRN,
|
||||
all interrupts are treated as non-urgent. ACRN sets SN=0 during initialization
|
||||
and then never changes it at runtime;
|
||||
determines whether to notify (``SN=0``) or not notify (``SN=1``) the CPU for
|
||||
non-urgent interrupts. For ACRN, all interrupts are treated as non-urgent.
|
||||
ACRN sets SN=0 during initialization and then never changes it at runtime.
|
||||
|
||||
Notification Vector (NV):
|
||||
the CPU must be notified with an interrupt and this
|
||||
field specifies the vector for notification;
|
||||
field specifies the vector for notification.
|
||||
|
||||
Notification Destination (NDST):
|
||||
the physical APIC-ID of the destination.
|
||||
ACRN does not support vCPU migration, one vCPU always runs on the same pCPU,
|
||||
ACRN does not support vCPU migration. One vCPU always runs on the same pCPU,
|
||||
so for ACRN, NDST is never changed after initialization.
|
||||
|
||||
Outstanding Notification (ON):
|
||||
|
@ -188,10 +187,10 @@ Outstanding Notification (ON):
|
|||
|
||||
The ACRN scheduler supports vCPU scheduling, where two or more vCPUs can
|
||||
share the same pCPU using a time sharing technique. One issue emerges
|
||||
here for VT-d posted interrupt handling process, where IRQs could happen
|
||||
here for the VT-d posted interrupt handling process, where IRQs could happen
|
||||
when the target vCPU is in a halted state. We need to handle the case
|
||||
where the running vCPU disrupted by the external interrupt, is not the
|
||||
target vCPU that an external interrupt should be delivered.
|
||||
target vCPU that should have received the external interrupt.
|
||||
|
||||
Consider this scenario:
|
||||
|
||||
|
@ -206,7 +205,7 @@ allocate the same Activation Notification Vector (ANV) to all vCPUs.
|
|||
To circumvent this issue, ACRN allocates unique ANVs for each vCPU that
|
||||
belongs to the same pCPU. The ANVs need only be unique within each pCPU,
|
||||
not across all vCPUs. Since vCPU0's ANV is different from vCPU1's ANV,
|
||||
if a vCPU0 is in a halted state, external interrupts from an assigned
|
||||
if vCPU0 is in a halted state, external interrupts from an assigned
|
||||
device destined to vCPU0 delivered through the PID will not trigger the
|
||||
posted interrupt processing. Instead, a VMExit to ACRN happens that can
|
||||
then process the event such as waking up the halted vCPU0 and kick it
|
||||
|
@ -233,15 +232,15 @@ related vCPU array.
|
|||
An example to illustrate our solution:
|
||||
|
||||
.. figure:: images/passthru-image50.png
|
||||
:align: center
|
||||
:align: center
|
||||
|
||||
ACRN sets ``SN=0`` during initialization and then never change it at
|
||||
ACRN sets ``SN=0`` during initialization and then never changes it at
|
||||
runtime. This means posted interrupt notification is never suppressed.
|
||||
After posting the interrupt in Posted Interrupt Request (PIR), VT-d will
|
||||
always notify the CPU using the interrupt vector NV, in both root and
|
||||
non-root mode. With this scheme, if the target vCPU is running under
|
||||
VMX non-root mode, it will receive the interrupts coming from
|
||||
passed-through device without a VMExit (and therefore without any
|
||||
VMX non-root mode, it will receive the interrupts coming from the
|
||||
passthrough device without a VMExit (and therefore without any
|
||||
intervention of the ACRN hypervisor).
|
||||
|
||||
If the target vCPU is in a halted state (under VMX non-root mode), a
|
||||
|
@ -254,11 +253,11 @@ immediately.
|
|||
MMIO Remapping
|
||||
**************
|
||||
|
||||
For PCI MMIO BAR, hypervisor builds EPT mapping between virtual BAR and
|
||||
physical BAR, then VM can access MMIO directly.
|
||||
There is one exception, MSI-X table is also in a MMIO BAR. Hypervisor needs to trap the
|
||||
accesses to MSI-X table. So the page(s) having MSI-X table should not be accessed by guest
|
||||
directly. EPT mapping is not built for these pages having MSI-X table.
|
||||
For PCI MMIO BAR, the hypervisor builds EPT mapping between the virtual BAR and
|
||||
physical BAR, then the VM can access MMIO directly. There is one exception: an
|
||||
MSI-X table is also in a MMIO BAR. The hypervisor needs to trap the accesses to
|
||||
the MSI-X table. So the pages that have an MSI-X table should not be accessed by
|
||||
the VM directly. EPT mapping is not built for pages that have an MSI-X table.
|
||||
|
||||
Device Configuration Emulation
|
||||
******************************
|
||||
|
@ -266,25 +265,26 @@ Device Configuration Emulation
|
|||
The PCI configuration space can be accessed by a PCI-compatible
|
||||
Configuration Mechanism (IO port 0xCF8/CFC) and the PCI Express Enhanced
|
||||
Configuration Access Mechanism (PCI MMCONFIG). The ACRN hypervisor traps
|
||||
this PCI configuration space access and emulate it. Refer to :ref:`split-device-model` for details.
|
||||
this PCI configuration space access and emulates it. Refer to :ref:`split-device-model` for details.
|
||||
|
||||
MSI-X Table Emulation
|
||||
*********************
|
||||
|
||||
VM accesses to MSI-X table should be trapped so that hypervisor has the
|
||||
VM accesses to an MSI-X table should be trapped so that the hypervisor has the
|
||||
information to map the virtual vector and physical vector. EPT mapping should
|
||||
be skipped for the 4KB pages having MSI-X table.
|
||||
be skipped for the 4KB pages that have an MSI-X table.
|
||||
|
||||
There are three situations for the emulation of MSI-X table:
|
||||
There are three situations for the emulation of MSI-X tables:
|
||||
|
||||
- **Service VM**: accesses to MSI-X table are handled by HV MMIO handler (4KB adjusted up
|
||||
and down). HV will remap interrupts.
|
||||
- **Post-launched VM**: accesses to MSI-X Tables are handled by DM MMIO handler
|
||||
(4KB adjusted up and down) and when DM (Service VM) writes to the table, it will be
|
||||
intercepted by HV MMIO handler and HV will remap interrupts.
|
||||
- **Pre-launched VM**: Writes to MMIO region in MSI-X Table BAR handled by HV MMIO
|
||||
handler. If the offset falls within the MSI-X table (offset, offset+tables_size),
|
||||
HV remaps interrupts.
|
||||
- **Service VM**: Accesses to an MSI-X table are handled by the hypervisor MMIO
|
||||
handler (4KB adjusted up and down). The hypervisor remaps the interrupts.
|
||||
- **Post-launched VM**: Accesses to an MSI-X table are handled by the Device
|
||||
Model MMIO handler (4KB adjusted up and down). When the Device Model (Service
|
||||
VM) writes to the table, it will be intercepted by the hypervisor MMIO
|
||||
handler. The hypervisor remaps the interrupts.
|
||||
- **Pre-launched VM**: Writes to the MMIO region in an MSI-X table BAR are
|
||||
handled by the hypervisor MMIO handler. If the offset falls within the MSI-X
|
||||
table (offset, offset+tables_size), the hypervisor remaps the interrupts.
|
||||
|
||||
|
||||
.. _interrupt-remapping:
|
||||
|
@ -292,7 +292,7 @@ There are three situations for the emulation of MSI-X table:
|
|||
Interrupt Remapping
|
||||
*******************
|
||||
|
||||
When the physical interrupt of a passthrough device happens, hypervisor has
|
||||
When the physical interrupt of a passthrough device happens, the hypervisor has
|
||||
to distribute it to the relevant VM according to interrupt remapping
|
||||
relationships. The structure ``ptirq_remapping_info`` is used to define
|
||||
the subordination relation between physical interrupt and VM, the
|
||||
|
@ -303,10 +303,10 @@ virtual destination, etc. See the following figure for details:
|
|||
|
||||
Remapping of physical interrupts
|
||||
|
||||
There are two different types of interrupt source: IOAPIC and MSI.
|
||||
There are two different types of interrupt sources: IOAPIC and MSI.
|
||||
The hypervisor will record different information for interrupt
|
||||
distribution: physical and virtual IOAPIC pin for IOAPIC source,
|
||||
physical and virtual BDF and other info for MSI source.
|
||||
physical and virtual BDF and other information for MSI source.
|
||||
|
||||
Service VM passthrough is also in the scope of interrupt remapping which is
|
||||
done on-demand rather than on hypervisor initialization.
|
||||
|
@ -318,11 +318,12 @@ done on-demand rather than on hypervisor initialization.
|
|||
Initialization of remapping of virtual IOAPIC interrupts for Service VM
|
||||
|
||||
:numref:`init-remapping` above illustrates how remapping of (virtual) IOAPIC
|
||||
interrupts are remapped for Service VM. VM exit occurs whenever Service VM tries to
|
||||
unmask an interrupt in (virtual) IOAPIC by writing to the Redirection
|
||||
Table Entry (or RTE). The hypervisor then invokes the IOAPIC emulation
|
||||
handler (refer to :ref:`hld-io-emulation` for details on I/O emulation) which
|
||||
calls APIs to set up a remapping for the to-be-unmasked interrupt.
|
||||
interrupts are remapped for the Service VM. VM exit occurs whenever the Service
|
||||
VM tries to unmask an interrupt in (virtual) IOAPIC by writing to the
|
||||
Redirection Table Entry (or RTE). The hypervisor then invokes the IOAPIC
|
||||
emulation handler (refer to :ref:`hld-io-emulation` for details on I/O
|
||||
emulation) which calls APIs to set up a remapping for the to-be-unmasked
|
||||
interrupt.
|
||||
|
||||
Remapping of (virtual) MSI interrupts are set up in a similar sequence:
|
||||
|
||||
|
@ -331,67 +332,65 @@ Remapping of (virtual) MSI interrupts are set up in a similar sequence:
|
|||
|
||||
Initialization of remapping of virtual MSI for Service VM
|
||||
|
||||
This figure illustrates how mappings of MSI or MSI-X are set up for
|
||||
Service VM. Service VM is responsible for issuing a hypercall to notify the
|
||||
This figure illustrates how mappings of MSI or MSI-X are set up for the
|
||||
Service VM. The Service VM is responsible for issuing a hypercall to notify the
|
||||
hypervisor before it configures the PCI configuration space to enable an
|
||||
MSI. The hypervisor takes this opportunity to set up a remapping for the
|
||||
given MSI or MSI-X before it is actually enabled by Service VM.
|
||||
given MSI or MSI-X before it is actually enabled by the Service VM.
|
||||
|
||||
When the User VM needs to access the physical device by passthrough, it uses
|
||||
the following steps:
|
||||
|
||||
- User VM gets a virtual interrupt
|
||||
- User VM gets a virtual interrupt.
|
||||
- VM exit happens and the trapped vCPU is the target where the interrupt
|
||||
will be injected.
|
||||
- Hypervisor will handle the interrupt and translate the vector
|
||||
according to ptirq_remapping_info.
|
||||
- Hypervisor delivers the interrupt to User VM.
|
||||
- Hypervisor handles the interrupt and translates the vector
|
||||
according to ``ptirq_remapping_info``.
|
||||
- Hypervisor delivers the interrupt to the User VM.
|
||||
|
||||
When the Service VM needs to use the physical device, the passthrough is also
|
||||
active because the Service VM is the first VM. The detail steps are:
|
||||
|
||||
- Service VM get all physical interrupts. It assigns different interrupts for
|
||||
different VMs during initialization and reassign when a VM is created or
|
||||
- Service VM gets all physical interrupts. It assigns different interrupts for
|
||||
different VMs during initialization and reassigns when a VM is created or
|
||||
deleted.
|
||||
- When physical interrupt is trapped, an exception will happen after VMCS
|
||||
- When a physical interrupt is trapped, an exception will happen after VMCS
|
||||
has been set.
|
||||
- Hypervisor will handle the VM exit issue according to
|
||||
ptirq_remapping_info and translates the vector.
|
||||
- The interrupt will be injected the same as a virtual interrupt.
|
||||
- Hypervisor handles the VM exit issue according to
|
||||
``ptirq_remapping_info`` and translates the vector.
|
||||
- The interrupt is injected the same as a virtual interrupt.
|
||||
|
||||
ACPI Virtualization
|
||||
*******************
|
||||
|
||||
ACPI virtualization is designed in ACRN with these assumptions:
|
||||
|
||||
- HV has no knowledge of ACPI,
|
||||
- Hypervisor has no knowledge of ACPI,
|
||||
- Service VM owns all physical ACPI resources,
|
||||
- User VM sees virtual ACPI resources emulated by device model.
|
||||
- User VM sees virtual ACPI resources emulated by the Device Model.
|
||||
|
||||
Some passthrough devices require physical ACPI table entry for
|
||||
initialization. The device model will create such device entry based on
|
||||
the physical one according to vendor ID and device ID. Virtualization is
|
||||
implemented in Service VM device model and not in scope of the hypervisor.
|
||||
For pre-launched VM, ACRN hypervisor doesn't support the ACPI virtualization,
|
||||
so devices relying on ACPI table are not supported.
|
||||
Some passthrough devices require a physical ACPI table entry for initialization.
|
||||
The Device Model creates such device entry based on the physical one according
|
||||
to vendor ID and device ID. Virtualization is implemented in the Service VM
|
||||
Device Model and not in the scope of the hypervisor. For pre-launched VMs, the
|
||||
ACRN hypervisor doesn't support ACPI virtualization, so devices relying on ACPI
|
||||
tables are not supported.
|
||||
|
||||
GSI Sharing Violation Check
|
||||
***************************
|
||||
|
||||
All the PCI devices that are sharing the same GSI should be assigned to
|
||||
the same VM to avoid physical GSI sharing between multiple VMs.
|
||||
In partitioned mode or hybrid mode, the PCI devices assigned to
|
||||
pre-launched VM is statically predefined. Developers should take care not to
|
||||
violate the rule.
|
||||
For post-launched VM, devices that don't support MSI, ACRN DM puts the devices
|
||||
sharing the same GSI pin to a GSI
|
||||
sharing group. The devices in the same group should be assigned together to
|
||||
the current VM, otherwise, none of them should be assigned to the
|
||||
current VM. A device that violates the rule will be rejected to be
|
||||
passed-through. The checking logic is implemented in Device Model and not
|
||||
in scope of hypervisor.
|
||||
The platform specific GSI information shall be filled in devicemodel/hw/pci/platform_gsi_info.c
|
||||
for target platform to activate the checking of GSI sharing violation.
|
||||
All the PCI devices that share the same GSI should be assigned to the same
|
||||
VM to avoid physical GSI sharing between multiple VMs. In partitioned mode or
|
||||
hybrid mode, the PCI devices assigned to a pre-launched VM are statically
|
||||
predefined. Developers should take care not to violate the rule. For a
|
||||
post-launched VM, the ACRN Device Model puts the devices sharing the same GSI
|
||||
pin in a GSI sharing group (devices that don't support MSI). The devices in the
|
||||
same group should be assigned together to the current VM; otherwise, none of
|
||||
them should be assigned to the current VM. A device that violates the rule will
|
||||
be rejected to be passed-through. The checking logic is implemented in the
|
||||
Device Model and not in the scope of the hypervisor. The platform-specific GSI
|
||||
information shall be filled in ``devicemodel/hw/pci/platform_gsi_info.c`` for
|
||||
the target platform to activate the checking of GSI sharing violations.
|
||||
|
||||
.. _PCIe PTM implementation:
|
||||
|
||||
|
@ -408,14 +407,14 @@ further details on PTM, refer to the `PCIe specification
|
|||
<https://pcisig.com/specifications>`_.
|
||||
|
||||
ACRN adds PCIe root port emulation in the hypervisor to support the PTM feature
|
||||
and emulates a simple PTM hierarchy. ACRN enables PTM in a Guest VM if the user
|
||||
sets the ``enable_ptm`` option when passing through a device to a post-launched
|
||||
VM. When you enable PTM, the passthrough device is connected to a virtual
|
||||
root port instead of the host bridge.
|
||||
and emulates a simple PTM hierarchy. ACRN enables PTM in a post-launched VM if
|
||||
the user sets the ``enable_ptm`` option when passing through a device to the
|
||||
post-launched VM. When you enable PTM, the passthrough device is connected to a
|
||||
virtual root port instead of the host bridge.
|
||||
|
||||
By default, the :ref:`vm.PTM` option is disabled in ACRN VMs. Use the
|
||||
:ref:`ACRN configurator tool <acrn_configurator_tool>` to enable PTM
|
||||
in the scenario XML file that configures the Guest VM.
|
||||
:ref:`acrn_configurator_tool` to enable PTM
|
||||
in the scenario XML file that configures the VM.
|
||||
|
||||
Here is an example launch script that configures a supported Ethernet card for
|
||||
passthrough and enables PTM on it:
|
||||
|
@ -436,7 +435,7 @@ passthrough and enables PTM on it:
|
|||
echo ${passthru_bdf["ethptm"]} > /sys/bus/pci/drivers/pci-stub/bind
|
||||
|
||||
acrn-dm -A -m $mem_size -s 0:0,hostbridge \
|
||||
-s 3,virtio-blk,uos-test.img \
|
||||
-s 3,virtio-blk,user-vm-test.img \
|
||||
-s 4,virtio-net,tap0 \
|
||||
-s 5,virtio-console,@stdio:stdio_port \
|
||||
-s 6,passthru,a9/00/0,enable_ptm \
|
||||
|
@ -458,8 +457,8 @@ PTM Implementation Notes
|
|||
To simplify PTM support implementation, the virtual root port only supports the
|
||||
most basic PCIe configuration and operation, in addition to PTM capabilities.
|
||||
|
||||
In Guest VM post-launched scenarios, you enable PTM by setting the
|
||||
``enable_ptm`` option for the pass through device (as shown above).
|
||||
For a post-launched VM, you enable PTM by setting the
|
||||
``enable_ptm`` option for the passthrough device (as shown above).
|
||||
|
||||
.. figure:: images/PTM-hld-PTM-flow.png
|
||||
:align: center
|
||||
|
@ -469,49 +468,52 @@ In Guest VM post-launched scenarios, you enable PTM by setting the
|
|||
PTM-enabling workflow in post-launched VM
|
||||
|
||||
As shown in :numref:`ptm-flow`, PTM is enabled in the root port during the
|
||||
hypervisor startup. The Device Model (DM) then checks whether the pass-through device
|
||||
supports PTM requestor capabilities and whether the corresponding root port
|
||||
supports PTM root capabilities, as well as some other sanity checks. If an
|
||||
hypervisor startup. The Device Model (DM) then checks whether the passthrough
|
||||
device supports PTM requestor capabilities and whether the corresponding root
|
||||
port supports PTM root capabilities, as well as some other sanity checks. If an
|
||||
error is detected during these checks, the error will be reported and ACRN will
|
||||
not enable PTM in the Guest VM. This doesn't prevent the user from launching the Guest
|
||||
VM and passing through the device to the Guest VM. If no error is detected,
|
||||
the device model will use ``add_vdev`` hypercall to add a virtual root port (VRP),
|
||||
acting as the PTM root, to the Guest VM before passing through the device to the Guest VM.
|
||||
not enable PTM in the post-launched VM. This doesn't prevent the user from
|
||||
launching the post-launched VM and passing through the device to the VM. If no
|
||||
error is detected, the Device Model uses the ``add_vdev`` hypercall to add a
|
||||
virtual root port (VRP), acting as the PTM root, to the post-launched VM before
|
||||
passing through the device to the post-launched VM.
|
||||
|
||||
.. figure:: images/PTM-hld-PTM-passthru.png
|
||||
:align: center
|
||||
:width: 700
|
||||
:name: ptm-vrp
|
||||
|
||||
PTM-enabled PCI device pass-through to post-launched VM
|
||||
PTM-enabled PCI device passthrough to post-launched VM
|
||||
|
||||
:numref:`ptm-vrp` shows that, after enabling PTM, the passthru device connects to
|
||||
the virtual root port instead of the virtual host bridge.
|
||||
:numref:`ptm-vrp` shows that, after enabling PTM, the passthrough device
|
||||
connects to the virtual root port instead of the virtual host bridge.
|
||||
|
||||
To use PTM in a virtualized environment, you may want to first verify that PTM
|
||||
is supported by the device and is enabled on the bare metal machine.
|
||||
If supported, follow these steps to enable PTM in the post-launched guest VM:
|
||||
If supported, follow these steps to enable PTM in the post-launched VM:
|
||||
|
||||
1. Make sure that PTM is enabled in the guest kernel. In the Linux kernel, for example,
|
||||
set ``CONFIG_PCIE_PTM=y``.
|
||||
1. Make sure that PTM is enabled in the guest kernel. In the Linux kernel,
|
||||
for example, set ``CONFIG_PCIE_PTM=y``.
|
||||
2. Not every PCI device supports PTM. One example that does is the Intel I225-V
|
||||
Ethernet controller. If you passthrough this card to the guest VM, make sure the guest VM
|
||||
uses a version of the IGC driver that supports PTM.
|
||||
3. In the device model launch script, add the ``enable_ptm`` option to the
|
||||
Ethernet controller. If you passthrough this card to the post-launched VM,
|
||||
make sure the post-launched VM uses a version of the IGC driver that supports
|
||||
PTM.
|
||||
3. In the Device Model launch script, add the ``enable_ptm`` option to the
|
||||
passthrough device. For example:
|
||||
|
||||
.. code-block:: bash
|
||||
:emphasize-lines: 5
|
||||
|
||||
$ acrn-dm -A -m $mem_size -s 0:0,hostbridge \
|
||||
-s 3,virtio-blk,uos-test.img \
|
||||
-s 3,virtio-blk,user-vm-test.img \
|
||||
-s 4,virtio-net,tap0 \
|
||||
-s 5,virtio-console,@stdio:stdio_port \
|
||||
-s 6,passthru,a9/00/0,enable_ptm \
|
||||
--ovmf /usr/share/acrn/bios/OVMF.fd \
|
||||
|
||||
4. You can check that PTM is correctly enabled on guest by displaying the PCI
|
||||
bus hiearchy on the guest using the ``lspci`` command:
|
||||
4. You can check that PTM is correctly enabled on the post-launched VM by
|
||||
displaying the PCI bus hierarchy on the post-launched VM using the ``lspci``
|
||||
command:
|
||||
|
||||
.. code-block:: bash
|
||||
:emphasize-lines: 12,20
|
||||
|
@ -555,9 +557,10 @@ VMs:
|
|||
.. doxygenfunction:: ptirq_prepare_msix_remap
|
||||
:project: Project ACRN
|
||||
|
||||
Post-launched VM needs to pre-allocate interrupt entries during VM initialization.
|
||||
Post-launched VM needs to free interrupt entries during VM de-initialization.
|
||||
The following APIs are provided to pre-allocate/free interrupt entries for post-launched VM:
|
||||
Post-launched VMs need to pre-allocate interrupt entries during VM
|
||||
initialization. Post-launched VMs need to free interrupt entries during VM
|
||||
de-initialization. The following APIs are provided to pre-allocate/free
|
||||
interrupt entries for post-launched VMs:
|
||||
|
||||
.. doxygenfunction:: ptirq_add_intx_remapping
|
||||
:project: Project ACRN
|
||||
|
@ -568,12 +571,12 @@ The following APIs are provided to pre-allocate/free interrupt entries for post-
|
|||
.. doxygenfunction:: ptirq_remove_msix_remapping
|
||||
:project: Project ACRN
|
||||
|
||||
The following APIs are provided to acknowledge a virtual interrupt.
|
||||
The following APIs are provided to acknowledge a virtual interrupt:
|
||||
|
||||
.. doxygenfunction:: ptirq_intx_ack
|
||||
:project: Project ACRN
|
||||
|
||||
The following APIs are provided to handle ptdev interrupt:
|
||||
The following APIs are provided to handle a ptdev interrupt:
|
||||
|
||||
.. doxygenfunction:: ptdev_init
|
||||
:project: Project ACRN
|
||||
|
|
Before Width: | Height: | Size: 49 KiB After Width: | Height: | Size: 49 KiB |
Before Width: | Height: | Size: 21 KiB After Width: | Height: | Size: 27 KiB |
Before Width: | Height: | Size: 104 KiB After Width: | Height: | Size: 136 KiB |
Before Width: | Height: | Size: 27 KiB After Width: | Height: | Size: 22 KiB |
Before Width: | Height: | Size: 26 KiB After Width: | Height: | Size: 20 KiB |
Before Width: | Height: | Size: 26 KiB After Width: | Height: | Size: 22 KiB |
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 21 KiB |