Window Smoke is a run of the mill Endpoint managed within Arroyo Systems Management, with a small wrinkle: it has an Nvidia GTX 1060 graphics card attached to it and a 2tb SATA SSD with Windows 10. For the longest time, this machine has primarily booted to Windows and served as a gaming machine.

Well, I want to put the CPU cycles to use in more ways especially now that I am building the Git edge-version of Emacs within Arroyo Emacs for both amd64 and aarch64 every time I bump my Nix Version Pins. I have also been having some somewhat annoying reliability issues cropping up with Virtuous Cassette so having a backup environment will feel alright.

This page isn't an Arroyo Module per se, but a NixOS morph role which is included in to Window Smoke, my desktop tower.

My desktop setup is very much inspired by my good friend kalessin's: A long USB cable, a long display cable, and long power cord connect a monitor and USB hub attached to a sit/stand desktop on casters to my desktop tower sitting in the corner of the room. The desk is spacious and easy to move while still having a full size display and a powerful machine attached to it. This has served me really well, especially considering that my monitor, a Dell U3818DW wide-screen display, has a USB-C input with alt-mode Display Port support. I can plug Virtuous Cassette in to the display and it swaps over my display and the USB hub attached to it. This plug-and-play setup has served me really well for the last few years.

El 🅱lan

This is disabled right now because vfio-pci was not robust enough. See Okay maybe I was wrong about 6.x

How much farther could I take it though?

The last time I visited kalessin he showed me something that I thought was quite remarkable: his desktop would virtualize Windows 10 with his graphics card directly forwarded to the VM so that his entire Steam library would run without having to consider any Proton shenanigans¹. I thought that was pretty impressive and it's got my brain going in this direction. His machine is an AMD threadripper and AMD GPU and the host runs Arch, but in theory this should work well enough with Intel, Nvidia, and NixOS, right? And if it is good, maybe I'll finally get around to refreshing my desktop with a bit more ram and a bit better GPU… As long as I don't have to deal with the Nvidia X11 drivers or Nouveau ever again.

skinparam linetype polyline
package "Tower" {
        [Intel iGPU]
        [RTX 1060]

        cloud "NixOS" {
                [libvirtd windows10 domain] --> "gamer mode"
                [plasma desktop] --> "goblin mode"
        }

        [Motherboard USB Controller] -> [plasma desktop]
        [Intel iGPU] -> [plasma desktop]
        [PCIe USB Controller] -> [libvirtd windows10 domain]
        [RTX 1060] -> [libvirtd windows10 domain]
        [1tb NVMe] -> [plasma desktop]
        [2tb SATA SSD] -> [libvirtd windows10 domain]
}

package "Desk" {
        node "Dell U3818DW" {
                interface HDMI2
                interface "USB in 1"
                interface DP
                interface "USB in 2"
                interface "Type-C DP"

                "USB out" --> "USB in 1" : when HDMI2 active
                "USB out" --> "USB in 2" : when DP active
        }

        "Type-C DP" --> [laptop] : forwards USB out when active

        "USB out" <-- [keyboard, mouse, etc] 
        HDMI2 --> [Intel iGPU]
        DP --> [RTX 1060]
        "USB in 1" --> [Motherboard USB Controller]
        "USB in 2" --> [PCIe USB Controller]

}

' hidden lines for node ordering/positioning
[keyboard, mouse, etc] -[hidden]d- [Motherboard USB Controller]
[keyboard, mouse, etc] -[hidden]d- "USB out"
[Type-C DP] -[hidden]l- [laptop]
[RTX 1060] -[hidden]d- [libvirtd windows10 domain]
[PCIe USB Controller] -[hidden]d- [libvirtd windows10 domain]
[Motherboard USB Controller] -[hidden]d- [plasma desktop]
[Intel iGPU] -[hidden]d- [plasma desktop]
[1tb NVMe] -[hidden]d- [plasma desktop]
[2tb SATA SSD] -[hidden]d- [plasma desktop]
' [HDMI2] -[hidden]l- [USB out]
[Intel iGPU] -[hidden]r- [1tb NVMe]

The four cables between the Tower and Desk are cable-tied together with a power cable which drives the monitor, peripherals, etc. With proper nudging, I can move my Windows 10 installation in to QEMU-KVM and have access to gamer mode and goblin mode side by side, rather than dual-booting and dealing with Nvidia-X11.

Setting up VFIO PCI Passthrough for virtualizing GPU accelerated Windows

This is disabled right now because vfio-pci was not robust enough. See Okay maybe I was wrong about 6.x

I started by reading the Arch Linux docs as well as Gentoo's on setting this up.

Most of the work here came in enabling VT-d and VT-x in the BIOS, and collating the devices properly. VT-x is the general "Intel hardware virtualization" support which lets qemu-kvm work, a bunch of CPU instructions and support for entering in to code that thinks it is in ring 0 when it is not, it's really clear in whether this works or not, and whether it's enabled or not. VT-d is the Intel-specific technical name for an virtualization-aware "IOMMU", basically allowing the OS to re-map hardware memory to main-memory addresses and in to the VMs' space. Your motherboard has to support it too, but if they do your VM can have PCI devices attached to it. My Intel i7-7700k and Asus 270AR motherboard both claimed to support VT-d but it was a bear to find that it was under a different firmware configuration menu than the VT-x feature was.² I spent quite some time trying to debug this after mistaking the generic "intel virtualization" option to only by VT-x, looking in ARK whether my motherboard chipset and CPU supported VT-d, spelunking the menu, eating dinner, getting lost in the woods, etc etc.

but with that enabled, it should be possible to see which IOMMU groups your hardware is set in to, hopefully the groupings make sense. qemu-kvm can only forward an entire IOMMU group, so maybe move some stuff around in the chassis if they group oddly. Here we see IOMMU group 1 has my Nvidia card in it, and the PCIe Controller it's plugged in to.

nix-shell -p pciutils
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V);
do
    echo "IOMMU Group ${g##*/}:";
    for d in $g/devices/*;
    do
        echo -e "\t$(lspci -k -nns ${d##*/})";
    done;
done | grep -A15 "IOMMU Group 1:"

You can see that I have already instructed the system to use vfio-pci as the kernel driver to use for them. This is done below.

With this terminal output, the following NixOS configuration can be created. It will 1) tell the Linux kernel to refuse loading the nouveau graphics driver for my X11 session 2) enable the IOMMU in the kernel 3) configure the "stub" vfio-pci module to attach to the devices I want forwarded to the VM 4) modify my initrd to load vfio-pci and attach it to my Nvidia driver early in the boot process before UDEV can auto-configure anything. I only needed to do this last part on the graphics card, the ethernet controller and USB controller do not need this treatment. You'll notice these IDs in the IOMMU Groups script-let or in the output of lspci -k -nn

Because of a really sick issue in kernel 6.x involving vfio-pci freezing the framebuffer, we used 5.15 on the desktop. We'll use 5.15 until it's sent out to pasture..

boot.kernelParams = [
  "intel_iommu=on"
  # nvidia gtx 1060, realtek pcie ethernet controller
  "vfio-pci.ids=10de:1c03,10de:10f1,10ec:8168,1b73:1100"
  "efifb=off"
];
boot.initrd.availableKernelModules = [ "vfio-pci" ];
boot.initrd.preDeviceCommands = ''
  DEVS="0000:00:02.0 0000:01:00.0 0000:01:00.1 0000:06:00.0"
  for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
  done
  modprobe -i vfio-pci
'';
# boot.kernelPackages = pkgs.linuxPackages_5_15;

And better make sure this thing is disallowed for good:

boot.blacklistedKernelModules = [
  "nouveau"
];

With that stuff in place, running libvirtd is just some pretty bog-standard stuff. Instruct NixOS to start the service, install virt-manager to set up the VM by hand … but I would like to make this declarative.

{ conifg, pkgs, ... }:

{
#  <<vfio>>
  <<nouveau>>
  virtualisation.libvirtd = {
    enable = true;
    onBoot = "start";
    onShutdown = "shutdown";
  };
  environment.systemPackages = with pkgs; [ virt-manager ];
}

This is essentially what is happening in Alexander Bakker's Notes on PCI Passthrough on NixOS using QEMU and VFIO blog post/literate configuration. I'll have to consider setting up Looking Glass and Scream soon, though it's not a big concern once I have the USB routing done.

Running Windows 10 in libvirtd

Instruct it to use /dev/sda as the "system image" but be careful if you delete the domain that you don't let libvirtd try to rm /dev/sda … it offered to, i'm not sure it actually would have, but that is spooky action.

UEFI boot support via OVMF is installed by default in the virtualisation.libvirt module, but the virt-manager GUI will BIOS boot by default. Change the boot mode in "overview configuration" in the install/setup phase, it's not editable once the VM is created in libvirtd..

Then attach PCI devices for the Nvidia card, a USB-3 PCIe card, an Ethernet PCIe card. The virtualized Windows will have its own USB ports exposed on the back of my case, it won't need to rely on bridge networking but will be connected directly to my switch, and the desktop will be displayed in front of me through the GPU and not the SPICE virtual display.

Forwarding USB devices one by one is fine during setup and fiddling around, but the USB controller will allow me to run another USB cable to my display and use the display as a KVM+USB switch between the host OS and the guest OS. I am waiting on the cables and controller to be delivered, it's been snowy this month.

NEXT integrate usb controller to vfio rules and libvirt domain

NEXT record video of boot + swapover + art of rally @ 60FPS + swap back

NEXT Investigate declarative libvirtd domains

NEXT what else should only be installed here?

• CUDA toolkit
• kind of want to play with stable diffusion on local GPU

Footnotes