PCI Passthrough for virtualized access to 2.5GbE NICs

I have spent a lot of my time earlier this year upgrading my home lab setup.  One thing that bothered me was the seeming inability to reach full spec thoughput in my LAN.  And after doing a bit of digging online, I found that my setup, with virtualizing NICs was adding significant load and latency from the virtualization overhead.

This write up details the steps I took to take back the bandwidth and latency back without giving up my virtual router setup.

We'll go over the steps needed to set up and configure the 2.5 GB Ethernet passthrough feature in Proxmox VMs for optimal performance. The steps are detailed as they include the specific hardware support requirements that are necessary, such as checking if the BIOS version is compatible with passthrough through of the 2.5 GB Ethernet connections.  The end result is maximized performance within virtualized environments using 2.5 or 10 GbE connections.

Introduction

PCI passthrough allows you to use physical PCI devices like graphics cards and network cards inside a virtualized environment with equivalent performance and CPU overhead compared to non virtualized environments. This write up will walk you through the basic requirements, steps, and troubleshooting for passing through 2.5GbE or 10GbE NICs in your home router.

The assumption is that you are using an X86 or AMD64 based PC with a Proxmox hypervisor and a dedicated VM for routing on a *nix based router/firewall distro. The distro I use and recommend currently is Opnsense as a good balance between features, open source compatible licensing, and  ease of use.

If you already have a system running, enabling PCI passthrough will be a revelation.

First things first

So, the first step involves enabling a feature called IOMMU (e.g., VT-d on Intel systems), provided both the CPU and hardware are compatible with it.

Check if your system supports the requirements

1. AMD: Ensure your system supports IOMMU (Input/Output Memory Management Unit), which is generally present in modern platforms but might be missing in older ones.
2. Intel: Check if your CPU and platform support the specific feature required for IOMMU, like Intel Virtualization Technology for Directed I/O (VT-d).

Setup the boot flags

Configure IOMMU with GRUB for Optimal Performance

1. Open the GRUB configuration file using the nano text editor:

sudo vi /etc/default/grub

2. For Intel CPUs, add the following line to enable IOMMU:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

3. If you have an AMD CPU, add the equivalent line for AMD support:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

Save your changes and exit the nano editor.

Now, update GRUB to apply the new settings:

sudo update-grub

By following these steps, you'll ensure that your system's IOMMU is properly configured using GRUB for efficient operation with passthrough devices.

My home router runs on an Intel CPU with 2.5GbE Intel i226-V NICs

Passing a PCI NIC

Steps for passing through a network card to a VM

1. Use lspci to get the NIC details

Setup the NIC in Proxmox

2. Go to VM settings -> Hardware -> Add -> Network Device

3. Select your network card in Network Dropdown

4. My home router has 4 Intel NICs.  I keep two on the host and pass through two to my OpnSense VM (hostpci0, hostpci1)

If all went well, reboot and you will notice a significant throughput increase and reduced CPU usage when pushing the network hard.

We're done now.  What's in it for me?

Here are some reasons why PCI passthrough can be useful for building a home router on Proxmox:

1. Performance - Passthrough allows the router VM direct access to the NIC hardware for maximum throughput and minimal overhead. This removes the performance penalty of virtualized networking.
2. Reliability - With passthrough, the NIC is not shared with the host or other VMs, providing dedicated bandwidth for routing tasks. This prevents other VMs from interfering with or slowing down the router VM.
3. Advanced networking - Many enterprise/gaming router features like packet filtering, QoS, traffic shaping, etc require low-level NIC access only possible via passthrough.
4. Security - Keeping the router on a separate VM improves security isolation between the router and other server workloads. Passthrough ensures strong network separation.
5. Flexibility - The router VM can be easily migrated between Proxmox nodes if needed, taking the passthrough device with it. Much easier than migrating physical router boxes.
6. Portability - Backing up and restoring the router VM is trivial with Proxmox. This makes transferring the router config to new hardware or recovery from disasters easier.

Conclusion

In summary, PCI passthrough allows you to leverage Proxmox's virtualization capabilities to build a powerful, flexible router with near bare-metal performance. The router VM gains many benefits compared to running on dedicated hardware.

The steps may sound daunting, but this guide makes approachable and a bit easier to setup.

I have not tried GPU passthrough yet.  It would be a fun project to look into over a quiet weekend.