cpr-transfer may lose a VFIO interrupt because the KVM instance is
destroyed and recreated. If an interrupt arrives in the middle, it is
dropped. To fix, stop pending new interrupts during cpr save, and pick
up the pieces. In more detail:
Stop the VCPUs. Call kvm_irqchip_remove_irqfd_notifier_gsi --> KVM_IRQFD to
deassign the irqfd gsi that routes interrupts directly to the VCPU and KVM.
After this call, interrupts fall back to the kernel vfio_msihandler, which
writes to QEMU's kvm_interrupt eventfd. CPR already preserves that
eventfd. When the route is re-established in new QEMU, the kernel tests
the eventfd and injects an interrupt to KVM if necessary.
Deassign INTx in a similar manner. For both MSI and INTx, remove the
eventfd handler so old QEMU does not consume an event.
If an interrupt was already pended to KVM prior to the completion of
kvm_irqchip_remove_irqfd_notifier_gsi, it will be recovered by the
subsequent call to cpu_synchronize_all_states, which pulls KVM interrupt
state to userland prior to saving it in vmstate.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Zhenzhong Duan <zhenzhong.duan@intel.com>
Link: https://lore.kernel.org/qemu-devel/1752689169-233452-3-git-send-email-steven.sistare@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
In commit a59d06305f ("vfio/pci: Introduce x-pci-class-code option"),
pci_register_vga() has been moved ouside of vfio_populate_vga(). As a
result, IGD VGA ranges are no longer properly exposed to guest.
To fix this, call pci_register_vga() after vfio_populate_vga() legacy
mode. A wrapper function vfio_pci_config_register_vga() is introduced
to handle it.
Fixes: a59d06305f ("vfio/pci: Introduce x-pci-class-code option")
Signed-off-by: Tomita Moeko <tomitamoeko@gmail.com>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20250723160906.44941-3-tomitamoeko@gmail.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Regions for sub-page BARs are normally mapped here, in response to the
guest writing to PCI config space:
vfio_pci_write_config()
pci_default_write_config()
pci_update_mappings()
memory_region_add_subregion()
vfio_sub_page_bar_update_mapping()
... vfio_dma_map()
However, after CPR, the guest does not reconfigure the device and the
code path above is not taken. To fix, in vfio_cpr_pci_post_load, call
vfio_sub_page_bar_update_mapping for each sub-page BAR with a valid
address.
Fixes: 7e9f214113 ("vfio/container: restore DMA vaddr")
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Zhenzhong Duan <zhenzhong.duan@intel.com>
Link: https://lore.kernel.org/qemu-devel/1752520890-223356-1-git-send-email-steven.sistare@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Commit 350785d41d ("ramfb: Add property to control if load the
romfile") introduced the `use-legacy-x86-rom` property for the
`vfio-pci-nohotplug` device, allowing control over VGA BIOS ROM
loading. However, the property compatibility setting was incorrectly
applied to the `vfio-pci` device instead, which causes all `vfio-pci`
devices to fail to load. This change fixes the issue by ensuring the
property is set on the correct device.
Fixes: d5fcf0d960 ("hw/i386: Add the ramfb romfile compatibility")
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Shaoqin Huang <shahuang@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20250723062714.1245826-1-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
ramfb is a sysbus device so it can only used for machine types where it
is explicitly enabled:
# git grep machine_class_allow_dynamic_sysbus_dev.*TYPE_RAMFB_DEVICE
hw/arm/virt.c: machine_class_allow_dynamic_sysbus_dev(mc,
TYPE_RAMFB_DEVICE);
hw/i386/microvm.c: machine_class_allow_dynamic_sysbus_dev(mc,
TYPE_RAMFB_DEVICE);
hw/i386/pc_piix.c: machine_class_allow_dynamic_sysbus_dev(m,
TYPE_RAMFB_DEVICE);
hw/i386/pc_q35.c: machine_class_allow_dynamic_sysbus_dev(m,
TYPE_RAMFB_DEVICE);
hw/loongarch/virt.c: machine_class_allow_dynamic_sysbus_dev(mc,
TYPE_RAMFB_DEVICE);
hw/riscv/virt.c: machine_class_allow_dynamic_sysbus_dev(mc,
TYPE_RAMFB_DEVICE);
So these six are the only machine types we have to worry about.
The three x86 machine types (pc, q35, microvm) will actually use the rom
(when booting with seabios).
For arm/riscv/loongarch virt we want to disable the rom.
This patch sets ramfb romfile option to false by default, except for x86
machines types (pc, q35, microvm) which need the rom file when booting
with seabios and machine types <= 10.0 (handling the case of arm virt,
for compat reasons).
At the same time, set the "use-legacy-x86-rom" property to true on those
historical versioned machine types in order to avoid the memory layout
being changed.
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Shaoqin Huang <shahuang@redhat.com>
Message-ID: <20250717100941.2230408-4-shahuang@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Move the TYPE_* to a new file hw/vfio/types.h because the
TYPE_VFIO_PCI will be used in later patch, but directly include the
hw/vfio/pci.h can cause some compilation error when cross build the
windows version.
The hw/vfio/types.h can be included to mitigate that problem.
Signed-off-by: Shaoqin Huang <shahuang@redhat.com>
Message-ID: <20250717100941.2230408-3-shahuang@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Currently the ramfb device loads the vgabios-ramfb.bin unconditionally,
but only the x86 need the vgabios-ramfb.bin, this can cause that when
use the release package on arm64 it can't find the vgabios-ramfb.bin.
Because only seabios will use the vgabios-ramfb.bin, load the rom logic
is x86-specific. For other !x86 platforms, the edk2 ships an EFI driver
for ramfb, so they don't need to load the romfile.
So add a new property use-legacy-x86-rom in both ramfb and vfio_pci
device, because the vfio display also use the ramfb_setup() to load
the vgabios-ramfb.bin file.
After have this property, the machine type can set the compatibility to
not load the vgabios-ramfb.bin if the arch doesn't need it.
For now the default value is true but it will be turned off by default
in subsequent patch when compats get properly handled.
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Shaoqin Huang <shahuang@redhat.com>
Message-ID: <20250717100941.2230408-2-shahuang@redhat.com>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Allow capping the maximum total size of in-flight VFIO device state buffers
queued at the destination, otherwise a malicious QEMU source could
theoretically cause the target QEMU to allocate unlimited amounts of memory
for buffers-in-flight.
Since this is not expected to be a realistic threat in most of VFIO live
migration use cases and the right value depends on the particular setup
disable this limit by default by setting it to UINT64_MAX.
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Avihai Horon <avihaih@nvidia.com>
Signed-off-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>
Link: https://lore.kernel.org/qemu-devel/4f7cad490988288f58e36b162d7a888ed7e7fd17.1752589295.git.maciej.szmigiero@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Introduce x-pci-class-code option to allow users to override PCI class
code of a device, similar to the existing x-pci-vendor-id option. Only
the lower 24 bits of this option are used, though a uint32 is used here
for determining whether the value is valid and set by user.
Additionally, to ensure VGA ranges are only exposed on VGA devices,
pci_register_vga() is now called in vfio_pci_config_setup(), after
the class code override is completed.
This is mainly intended for IGD devices that expose themselves either
as VGA controller (primary display) or Display controller (non-primary
display). The UEFI GOP driver depends on the device reporting a VGA
controller class code (0x030000).
Signed-off-by: Tomita Moeko <tomitamoeko@gmail.com>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20250708145211.6179-1-tomitamoeko@gmail.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Now after merging the precopy and postcopy version of complete() hook,
rename the precopy version from save_live_complete_precopy() to
save_complete().
Dropping the "live" when at it, because it's in most cases not live when
happening (in precopy).
No functional change intended.
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20250613140801.474264-7-peterx@redhat.com
[peterx: squash the fixup that covers a few more doc spots, per Juraj]
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reconstruct userland device state after CPR. During vfio_realize, skip all
ioctls that configure the device, as it was already configured in old QEMU.
Skip bind, and use the devid from CPR state.
Skip allocation of, and attachment to, ioas_id. Recover ioas_id from CPR
state, and use it to find a matching container, if any, before creating a
new one.
This reconstruction is not complete. hwpt_id is handled in a subsequent
patch.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Zhenzhong Duan <zhenzhong.duan@intel.com>
Link: https://lore.kernel.org/qemu-devel/1751493538-202042-17-git-send-email-steven.sistare@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
cpr-transfer will use the device name as a key to find the value
of the device descriptor in new QEMU. However, if the descriptor
number is specified by a command-line fd parameter, then
vfio_device_get_name creates a name that includes the fd number.
This causes a chicken-and-egg problem: new QEMU must know the fd
number to construct a name to find the fd number.
To fix, create an invariant name based on the id command-line parameter,
if id is defined. The user will need to provide such an id to use CPR.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Zhenzhong Duan <zhenzhong.duan@intel.com>
Link: https://lore.kernel.org/qemu-devel/1751493538-202042-10-git-send-email-steven.sistare@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Preserve vfio INTx state across cpr-transfer. Preserve VFIOINTx fields as
follows:
pin : Recover this from the vfio config in kernel space
interrupt : Preserve its eventfd descriptor across exec.
unmask : Ditto
route.irq : This could perhaps be recovered in vfio_pci_post_load by
calling pci_device_route_intx_to_irq(pin), whose implementation reads
config space for a bridge device such as ich9. However, there is no
guarantee that the bridge vmstate is read before vfio vmstate. Rather
than fiddling with MigrationPriority for vmstate handlers, explicitly
save route.irq in vfio vmstate.
pending : save in vfio vmstate.
mmap_timeout, mmap_timer : Re-initialize
bool kvm_accel : Re-initialize
In vfio_realize, defer calling vfio_intx_enable until the vmstate
is available, in vfio_pci_post_load. Modify vfio_intx_enable and
vfio_intx_kvm_enable to skip vfio initialization, but still perform
kvm initialization.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Link: https://lore.kernel.org/qemu-devel/1751493538-202042-3-git-send-email-steven.sistare@oracle.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
