Ubuntu 16.04 + VGA Passthrough

228 半夜停電結果原本的 server 掛了一顆硬碟申請到經費後不但買了新硬碟也把 server 整個升級還順便買了張新顯卡 XDD

新的電腦配置如下

CPU : Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
RAM：16GB
GPU: MSI Radeon RX 480 GAMING X 8G [1002:67df]
OS：Ubuntu 16.04

現在要設定 VGA Passthrough 簡直簡單到不行不過還是簡單記錄一下撞到的牆

使用 virt-manager 建立新 vm 時得要記得選成 UEFI 不要用 BIOS (不需要選 Q35 直接用預設的 I440 就可以了)
我有原版的 Windows 10 專業版裡面附的是一隻安裝 USB 我用這個 USB dd 出來的 image 在安裝時會一直要我裝驅動程式明明 virtio driver 都已經掛上也都裝完了但是卻一直沒辦法進行下一步把所有的磁碟都改成 IDE 也一樣最後就去 MS 官網下載 win10 iso 這樣才順利安裝
安裝時的磁碟機原本選 virtio 安裝時也裝了 virtio driver 但是安裝完重開時卻會找不到硬碟沒辦法開機得要改成 IDE 開然後再裝一次 virtio 的驅動程式
Windows 整個裝完後再從 virt-manager 那邊將顯卡跟音效卡從 PCI 那邊選進來原本的 QXL 顯卡可以先留著繼續用 virt-manager 裝顯卡但是最後要開始使用時記得將 QXL 移調預設會用這張卡沒辦法 disable 掉

其他的設定大概就這些

不知道為什麼之前泛用的 id 設定不會讓顯卡上的音效裝置載入 vfio-pci 所以我就寫死了
cat /etc/modprobe.d/vfio.conf
1
options vfio-pci ids=1002:67df,1002:aaf0
開機的 kernel command line 大部分都拿掉了不過多了個 video=efifb:off 一但啟動 vm dmesg 會吐一大堆錯誤訊息(我沒記錄下來是什麼 :p) 得要加上這個參數才行
cat /proc/cmdline
1
BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.4.0-64-generic root=ZFS=rpool/ROOT/ubuntu ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 kvm.ignore_msrs=1 video=efifb:off vt.handoff=7

Host blacklist radeon

blacklist radeon

1	sudo sh -c "echo \"blacklist radeon\"" >> /etc/modprobe.d/blacklist.conf

另外我有從舊 server 上抓了些設定檔過來不是很確定是不是需要

$ sudo tail -n17 /etc/libvirt/qemu.conf

nvram = [
   "/var/vm/ovmf-x64/OVMF_CODE-pure-efi.fd:/var/vm/ovmf-x64/OVMF_VARS-pure-efi.fd",
   "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd",
   "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd"
]

user = "root"
group = "root"
cgroup_device_acl = [
  "/dev/null", "/dev/full", "/dev/zero",
  "/dev/random", "/dev/urandom",
  "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
  "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
  "/dev/vfio/1"
]
clear_emulator_capabilities = 0

最後 vm 的 xml 長這樣

$ virsh dumpxml win10

<domain type='kvm' id='53'>
  <name>win10</name>
  <uuid>1ac4335d-e696-442a-bf9b-c75c455377e8</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/var/vm/win10.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/vm/iso/virtio-win-0.1.132.iso'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:92:0a:af'/>
      <source dev='enp0s31f6' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-1ac4335d-e696-442a-bf9b-c75c455377e8</label>
    <imagelabel>libvirt-1ac4335d-e696-442a-bf9b-c75c455377e8</imagelabel>
  </seclabel>
</domain>