My System:
- Intel Core i7 6700 So.1151
- Gigabyte GA-Z170X-UD5 TH Intel Z170 So.1151
- 1x Palit GeForce GTX 960
- 1x Zotac GeForce GTX 650 Ti
first slot: GTX 650 Ti (used by the host)
second slot: GTX 960 (used by the vm)
(the mb always uses the first slot as primary gpu)
---
sum up:
* install Ubuntu 16.04
* install kvm, quemu and so on
* let vfio-pci grab the GTX 960 (see [1] and [2])
* install nvidia driver for the GTX 650 Ti
* build patched kernel (see [3])
* start virtual machine manager
* create a vm with windows 7
* adding passthrough gpu from slot 2 results in (details see [4]):
* vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver
---
### 1
/etc/modprobe.d/vfio.conf
10de:0fba: GTX 960
10de:1401: GTX 960 audio
```
options vfio-pci ids=10de:0fba,10de:1401
```
### 2
output of: lspci -nnk
```
[...]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] [10de:11c6] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. GK106 [GeForce GTX 650 Ti] [19da:4263]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_340
01:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. GK106 HDMI Audio Controller [19da:4263]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
Subsystem: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia_340
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1401]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
[...]
```
output of: dmesg | grep iommu
```
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.4.0-23-generic root=/dev/mapper/ubuntu--vg-root ro quiet noplymouth intel_iommu=on pcie_acs_override=downstream rd.modules-load=vfio-pci
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.4.0-23-generic root=/dev/mapper/ubuntu--vg-root ro quiet noplymouth intel_iommu=on pcie_acs_override=downstream rd.modules-load=vfio-pci
[ 0.615463] iommu: Adding device 0000:00:00.0 to group 0
[ 0.615474] iommu: Adding device 0000:00:01.0 to group 1
[ 0.615480] iommu: Adding device 0000:00:01.1 to group 1
[...]
[ 0.615592] iommu: Adding device 0000:01:00.0 to group 1
[ 0.615595] iommu: Adding device 0000:01:00.1 to group 1
[ 0.615598] iommu: Adding device 0000:02:00.0 to group 1
[ 0.615601] iommu: Adding device 0000:02:00.1 to group 1
```
### 3
/etc/default/grub
(and after saving update-grub)
```
[...]
GRUB_CMDLINE_LINUX_DEFAULT="quiet noplymouth intel_iommu=on pcie_acs_override=downstream rd.modules-load=vfio-pci"
[...]
```
patched kernel with ACS quirk
```
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git
cd ubuntu-xenial
first patch: https://patchwork.ozlabs.org/patch/604533/
second patch: https://patchwork.ozlabs.org/patch/604532/
patch -p1 < first.patch
patch -p1 < second.patch
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic
reboot
```
### 4
Creating a vm using PCI 0:2:0.0 with the Virtual Machine Manager results into:
```
Unable to complete install: 'internal error: process exited while connecting to monitor: 2016-06-08T17:00:35.910505Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2016-06-08T17:00:35.910523Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 1
2016-06-08T17:00:35.910530Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed'
```