High latency to and from Hyper-V guests with VMQ

Are you experiencing high response times to / from guests on your Hyper-V enabled host or Hyper-V Server 2008 R2? Are the NICs on the host Broadcom/Q-Logic?

Lots of chatter around the net already document this issue but I want to contribute a solution for Hyper-V Server 2008 R2 with PowerShell 2.0. All case scenarios were of admins running 2012 R2 with either a GUI available or a newer version of PowerShell. Hyper-V 2008 R2 has PowerShell 2.0 with a limited number of cmdlets.

The problem is having Virtual Machine Queues enabled in the NIC’s driver settings. VMQs is:

“a feature available to computers running Windows Server 2008 R2 with the Hyper-V server role installed, that have VMQ-capable network hardware. VMQ uses hardware packet filtering to deliver packet data from an external virtual machine network directly to virtual machines, which reduces the overhead of routing packets and copying them from the management operating system to the virtual machine.” – https://technet.microsoft.com/en-us/library/gg162704(v=ws.10).aspx

A solution that worked for me is to disable VMQs. Apparently, around April / May 2015 manufacturers released driver patches for this so perhaps give those a shot first.

The problem I faced was to disable it with just PowerShell 2.0, without a GUI and no devmgmt.msc.

Fix with registry

Open up regedit.exe and make your way to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\

You will see several subkeys named along the lines of “{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}” – these are GUID values of the devices. Each of these holds a value “Default” of each hardware device. Look for a Default value with data named “Network adapters“.

When found, expand that subkey which will then list more subkeys along the naming scheme of “0000“, “0001“, “0002” etc. Each of these subkeys resemble each network adapter. The “Driver desc” value will give you an indication as to which subkey you need, named something like “Broadcom NetXtreme Gigabit Ethernet“.

reg2.1

Have multiple Broadcom NICs? Run the below command to get GUIDs for all network adapters.

get-wmiobject win32_networkadapter | select name, GUID

Expand each subkey that has the Broadcom data value for “DriveDesc” and select the “Linkage” subkey – the values in this subkey show this device’s GUID.

Once identified the NIC(s) you want to disable VMQ on, look for the value named “*VMQ“. Alter its value from 1 to 0 and the changes will take effect after reboot.

reg3

Reports of these bugs are still occurring on new servers. Tom reported in May that his new R730 with a Broadcom/Q-Logic NIC still suffers from this. Microsoft suggest that this only impacts 2012 R2 servers and a select few NIC models, but my experiences suggest the impact is wider.

Leave a Reply

Your email address will not be published. Required fields are marked *