Intel vt-d for home vm use

I'm putting together a new system and one of the things I want to do on it is run a VM. When choosing CPUs I came across Intel's vpro technology. I'm a comp-sci student and last semester we looked at VMs, hardware virtualization, how VM device use is slowed because it has to go through the native OS and how hardware virtualization would help alleviate this. Naturally, vt-d looked like a good idea so I've been trying to build a system around vpro support that includes vt-d. This is easier said than done though! I've chosen the i5 2500 and so far (for lack of better alternatives that I've found) the Q67 chipset (haven't got to motherboards yet).

I think I'm getting ahead of myself though. Before I continue banging my head against the wall, trying to dig up all this vpro stuff and worrying about compatibility and which alternative is better, I would like to find out if it's even worth it, whether it will do what I want or not. I'm choosing this stuff to make my VM run closer to native performance. Does anyone have experience with vt-d? Is it worth all of this headache?

Message edited by Anto_di_Pretoro on 02-04-2012 at 05:20:01 AM

Reply to Anto_di_Pretoro

Well since you are a comp sci I would assume you would at least like to try it out. It's not as difficult as you think, for hardware, you just need a capable mobo and cpu.

Here is my low priced suggestions: xeon E3-1230 and Intel DQ67SW. Although the 1230 doesn't have igpu so you may want to get the 1235. I suggest this over the i5 because it has HT.
http://www.newegg.com/Product/Prod [...] 6819115083
http://www.newegg.com/Product/Prod [...] 6813121515

Reply to k1114

Thanks! I'll definitely look into those.

You've also answered another question I had, which was whether or not the devices had to be 'vt-d compatible'. I didn't think they did but I wanted to be sure. Based on how it seems to work it would make sense that anything can be used. It looks like the vt-d is implemented by giving the OSs different address mappings for the devices or something along those lines (I haven't read the whole page yet but the section after the introduction talks a bit about it).
http://software.intel.com/en-us/ar [...] o-devices/

I have another question related to vpro-compatible motherboards, but I'll post that in the 'chipsets and bios' section of the forum.

Thanks again!

Message edited by Anto_di_Pretoro on 02-05-2012 at 06:22:42 AM

Reply to Anto_di_Pretoro

What other plans do you have for this system? I mean, are you planning to run the free ESXi Hypervisor 5.0 or something else on it? What is the budget you're looking at for this home system?

Recently I've ordered several HP ProLiant ML110 G7 servers to run a single virtual machine within the free ESXi (both Windows 7 Professional and Windows Server 2008 R2) and have had wonderful luck with them. Granted, in all of them I added an HP SmartArray P410/256 MB RAID controller to enable full support for RAID arrays in the ESXi, but it has worked very well, and the performance has been great. These servers are quite affordable when you look at all the features you get with them, and from what I can see they have all your virtualization features with the Sandy Bridge Xeon processors.

Your only limitation with the ML110 G7 server is the maximum memory is 16 GB, but that's still quite a bit to throw at a single VM and well within the range to use the free ESXi Hypervisor.

Reply to choucove

That chipset and bios thread disappeared on me. It was closed when I marked best answer and now I can't find it. No big deal, I was just going to refer people back to this thread. I'll look into the ASRock boards, the workstation/server chipsets and their boards and compare them to the aforementioned Intel board. The Intel board seems great but it doesn't have many PCIe expansion slots which I thought may become an issue.

The plans for the system are to use it for homework (includes programming), internet and some media (blu-ray viewing, itunes/music library). It's replacing the family desktop (we've had it for five to ten years). No gaming (I prefer consoles) and not to make it a general use public server. It may be used as a home media server though. With regards to virtualization, I've only ever really used VMs to develop in Linux and to have a sandbox environment. Those are currently my plans for it.

I'm not familiar with the ESXi Hypervisor (I know it comes from VMWare but I've never experienced it nor know how it's different from the Player) but I'll be using freeware. I'm thinking to spend about a grand. What do you use your servers for choucove?

Thanks guys, looks like I have some researching to keep me busy a while.

Reply to Anto_di_Pretoro

When you select a best answer, they tend to close the threads so they don't get bumped. You can click your name and see all of your posts. http://www.tomshardware.com/forum/ [...] imply-bios

The asrock z68 extreme3 is near the same price as the q67. I just noticed xeons are on the support list too, that's rare for a company to do that on a desktop chipset. http://www.newegg.com/Product/Prod [...] 6813157271 Q67 and c206 have amt, the others do not so you should look into that. http://www.tomshardware.com/review [...] 003-7.html Other than that there isn't anything else affecting virtualization. This is the only c206 available. http://www.newegg.com/Product/Prod [...] 6813131725 What more research do you need to do? We could help with any questions you may have.

Message edited by k1114 on 02-09-2012 at 07:54:13 PM

Reply to k1114

These ML110 G7 servers have been used for various duties, primarily used as a simple storage server for some small offices here in town. We've done a few different configurations, basically with the ML110 G7 with a 3.3 Ghz Xeon quad-core processor, 8 GB of DDR3 1333 ECC memory, dual redundant 460 Watt power supplies, upgraded to HP SmartArray P410/256 MB RAID controller card, and put in two Western Digital Caviar Black or RE4 2 TB SATA hard drives in RAID 1. On some of the servers we have installed Windows Server 2008 R2 Standard and run Hyper-V with another instance of the virtual Windows Server 2008 R2 Standard to act as a file server/domain controller, but if a domain controller isn't necessary (which in most cases with these offices its not) then we just put a USB flash drive in the server, install ESXi Hypervisor, and run a virtual machine of Windows 7 Professional to do file sharing duties and manage automatic backups of the network. This setup would also work well for FreeNAS.

The problem with using ESXi Hypervisor in your particular situation is you would not be able to use the actual server as a standard desktop computer. What I mean is, when you install ESXi and it boots up from the flash drive, there is no standard desktop OS that you can go into to access email, install programs, play media, etc. etc. It's just a very stripped down basic GUI interface to allow you to change the IP address of the physical host server and change the administrative password. Everything else must be done and utilized through a completely different computer.

So, instead it seems what would be best for you would be VMWare's Player as it installs as an application on top of an existing operating system. This means that you will need a little more computational power since you have the host operating system, such as Windows 7, running as well as any virtual machines.

If you're looking to do virtual machines on a regular desktop system for testing and learning purposes, then about the only motherboard with the features you're looking for is the ASUS P8B WS http://www.newegg.com/Product/Prod [...] 6813131725 which supports ECC memory and Xeon processors. However, from what I can tell it does NOT support VT-d for some reason.

While the ML110 G7 itself offers the features you're looking for in virtualization, it would be a little difficult to convert to a desktop usage system perhaps. For instance, you would need to install a dedicated graphics card, and would have to go with a single slot graphics card as the only PCI-e x16 slot is at the very bottom of the case, and there is no additional power connectors for PCI-e devices. You would also need to purchase an add-in sound card to get audio. If you're wanting to play Blu-Ray you will also need to replace the included DVD drive, and then you still have to add your hard drives (which also require purchasing hard drive sleds for the drive cage.) So, in the end it would not be possible within your budget I don't believe.

Reply to choucove

I just came across this option perhaps, it is a microATX Q67 chipset motherboard from Tyan that does support VT-d (I had to look up in the manual to confirm) but it runs on normal Core i3/i5/i7 processors.

http://www.newegg.com/Product/Prod [...] 6813151254

Reply to choucove

The p8b ws DOES support vt-d in bios versions 0604 and after. http://support.asus.com/FAQ/detail [...] 9972F26277 This thread is about vt-d and I will not suggest a mobo that doesn't support it.

Message edited by k1114 on 02-09-2012 at 07:37:02 PM

Reply to k1114

Ah, thank you for that update, K1114 that is very good to hear as I had been disappointed before to hear the motherboard did not support VT-d.

Message edited by choucove on 02-09-2012 at 08:16:33 PM

Reply to choucove

I have P8B WS, just recently tried to install esxi 5.0 and received a compatibility error. It also seems esxi 4.1 doesn't support this motherboard either.. installed with errors along the way.. was able to get to esxi command console but i think there is some issues. (cpu is e3 1270

Reply to kttech

Bios version? os?

Message quoted 1 times

Reply to k1114

k1114 wrote :

Bios version? os?

latest bios 0900, installed esxi on physical.

received an error "This host has VT support however VT has not been enabled".
Thing is, I did enable it in the bios.

Reply to kttech

I'm not familiar with physical. You said you tried to install it and now you are saying you did install it. Is the error when installing or when executing.? Is there somewhere in the os or options you missed?

Message quoted 1 times
Message edited by k1114 on 02-15-2012 at 05:24:25 AM

Reply to k1114

k1114 wrote :

I'm not familiar with physical. You said you tried to install it and now you are saying you did install it. Is the error when installing or when executing.? Is there somewhere in the os or options you missed?

what i meant by physical is, non virtual. im not sure if i have the right terminology.
the beginning of the installation is when i received that error, but it still installed all the way through. not sure if esxi is utilizing the vt-d of the processor, due to that error.

Reply to kttech

The term is host which still doesn't tell me what os. But it's not a compatibility issue and it does say you can use vt. The error says it is supported by the host so your bios, os, hardware, everything is good to go. But you need to enable it in esxi. I'm guessing you are just learning esxi.

Reply to k1114

Also AMD have support for IOMMU and perhaps it easier to come by a decent configuration for a decent price from AMD than from Intel. Most 990FX motherboards support IOMMU and you basically can't go wrong with the CPU.

Also Xen supports passthrough via VT-d/IOMMU. If it works? You could check this youtube video for example:

Xen Dual VGA Passthrough

Another thing you may want to check is whether the hardware that you want to pass through supports function level reset (FLR). Nvidia does supply firmware patches for their (later generation) Geforce boards when contacting support (if they don't support it already). Another way to reset hardware is by using D3-D0 power states where you cut the Vcc power to the PCIe slot via ACPI. You may want to check whether Xen supports this type of reset before you try it out as I'm not sure. DirectPath supports it though.

Message edited by g00ey on 02-15-2012 at 09:45:41 AM

Reply to g00ey

From what I can tell the Q chipset is not necessary. I'm using an Asrock Z68 with a i5-2500 (not K not K not K). The VT-d's devices are :
- AMD 5850
- 2nd onboard USB2.0
- onboard audio (has issues)
- Creative X-Fi PCIe (not usable with native drivers)
- RTL8169 NIC

Failed:
- Creative Audigy 2 PCI
- Hercules Fortission IV (PCI VIA-based sound card)
- Geforce GTX260

Now, the question is whether the failed attemtps would have been better with Q chipset (since I found out that only they have "VT-d yes" on intel website).

Host: debian with Xen 4.1.2
Guest: Windows 7 (XP seems to not work right)

Just for the record, the 5850 is a secondary card to guest. The primary is the emulated one (which show a "!" once the 5850 is active). Same setup does not work for GTX260.
As for the issues for the sound cards, I would relate them to IRQ issues, because there are some pops, like when a sample arrives too late to the sound card. This cannot be tested/verified with any other device, as none are so time-sensitive (this is about ms-response times).

------------------------------ ATI HD5850+USB controller using Intel VT-d under Windows
i5-2500 on Asrock Z68 Extreme4,16GB RAM
128GB Kingston SSDNow V+
Xen 4.1.2+Linux host, Win7 HP guest OS
Reply to mathew7

It won't be better on q67, the intel site only states intel's own mobos not the mobos for other companies. As stated in his other thread which the link was posted for, the bios needs to have support for it. Either it supports it or it doesn't. The hardware (gpu, sound card, sata card, etc) also need to support vt, which is why all your hardware didn't work. Other people have no issues with the 260 so you may want to troubleshoot that.

Message edited by k1114 on 02-15-2012 at 07:06:12 PM

Reply to k1114

It seems that AMD has better support for virtualization than Intel at least on the desktop platform. Basically all current lines of AMD CPUs support AMD-v (which is the same as Intel VT-x) whereas many of Intel's low-budget line of CPU's don't have it. VT-x/AMD-v is useful but as I understand it, the thread is about VT-d, or IOMMU which is a more general term for this feature. AMD also calls it AMD-Vi.

In the AMD platform the support for IOMMU is dictated by the chipset and not the CPU as it is for Intel's VT-d. Apart from the chipset, the BIOS must have proper ACPI IVRS tables and support for it (the same goes for Intel's VT-d). There were some problems with it on 890FX motherboards but there should be no problems on 990FX motherboards (the 990X and 970 chipsets also support IOMMU but I don't know if it is always implemented in the BIOS). When that is settled all compatible AMD CPU's that I'm aware of will work with IOMMU and VT-x.

When it comes to peripheral I/O devices, there is no "special" virtualization feature that they need in order to work with PCI passthrough. What you need is to be able to reset them by software without resetting the entire system. This can mainly be done by function level reset (FLR) by which you can reset individual hardware components (or PCI devices, onboard components such as the NIC are also PCI devices) at the function level. The other way is to use the power save/management features in the ACPI where you can cut power to the PCI devices. A lot of hardware that are not specifically designed for virtualization or passthrough can be reset this way, at least in VMWare's VMDirectPath.

I don't know about Xen which is the virtualization software that comes to mind when one talks about advanced virtualization. In their VT-d HowTo page it explicitly says that only hardware with FLR is supported so it seems that Xen does not support hardware reset via ACPI, or d3d0 as it is also called (the power states D3 to D0 are covered in the Wikipedia page about ACPI). So this limitation is not in the hardware but in Xen, so the best thing to do is to point this out for the development team and hope they will implement it some day in the future.

There are other ways that can be used to reset but then you are moving into more dangerous territory. More information about these methods can be found here: http://www.vmware.com/files/pdf/te [...] h_host.pdf
Of course it would be desirable if peripheral manufacturers provided BIOS patches for their hardware that enabled FLR support and eventually supported it by default. After all it ain't no rocket science and it is the preferred way to initialize the hardware for PCI passthrough.

Then there is VGA passthrough which is an advanced form of PCI passthrough where you also need to get the VGA BIOS mappings sorted out in order for it to work. I don't know about ATI (or AMD) but Nvidia seems to be diligent to get it supported and they apparently supply firmware patches upon request for their Geforce cards. I've also read somewhere that their lines of Quadro and Tesla cards support FLR by default and are specifically designed to work with PCI/VGA passthrough.

Reply to threehosts

I don't know of any modern intel cpu that doesn't have vt-x. But vt-d is another story. Amd does make it a bit simpler by just supporting passthrough on all of their cpus. Although in a sense it still is dictated by the cpu, it's just all of them support it.

This thread has derailed after the hijack, I'm not even sure if the op is still here. Good info being posted but I think we should leave the thread to answering questions/resolving issues and I don't see any.

Reply to k1114

Yeah, I'm still here. Time's spread kind of thin is all. First off I'd like to thank everyone for their interest and input, it's generated good information.

I had considered AMD chips early in my research. I hadn't really looked too much into or compared the kind of support for the technologies between AMD and Intel. Reading around though, it seems that (for this generation of processors at least) Intel gives better overall performance.

I looked at a couple of Xeon E3s and a couple of the suggested motherboards. I'm leaning toward the E3 1235 with the ASUS P8B WS (looks like it has more PCIe 2.0 slots than the ASRock which, I believe, translates into being able to add more expansion cards). Looks like a winning combination but I'll put off ordering until I'm happy with a set of peripherals (got to make sure they fit on it). (Thanks to k1114 for the suggestion!)

To be clear on which virtualization tools can take advantage of vt-d (please correct me if I'm wrong), VMWare only offers support with ESXi, VirtualBox can't use this (yet, no estimated implementation date) and XEN implements it standard. Sounds like the winner is XEN (for my situation). I've never used it though, don't know what it's like. Apparently you can't take snapshots with it (a well-liked feature of VirtualBox), but there seems to be at least one (time-consuming) workaround for Linux guests
(http://searchservervirtualization.techtarget.com/tip/Creating-snapshots-in-Xen-with-Linux-commands).

Back to peripheral vt-d support, when researching the hardware will compatible parts say 'FLR implemented' or something in their description? Also, is FLR only needed for things that plug into PCI/PCIe slots and SATA plugs or, for example, does an external hard drive connected via USB need it too? I would think things that plug directly into the board (let's call them primary connectors) would need FLR and things that connect to those primary connectors don't. If that's the case then the external drive wouldn't need FLR because the USB microcontroller is the one that needs to have it.

Reply to Anto_di_Pretoro

FLR is in the PCI-SIG specification, USB and SATA is a different beast. IOMMU is for passing through PCI devices (even when you talk about PCIe or PCI-X you mean PCI also here as the software doesn't differentiate between PCI and PCIe).

When it comes to USB you could either pass through the entire USB controller with PCI passthrough or use USB passthrough via the hypervisor's (e.g. Xen or VirtualBox) software interface.

As for SATA, if you intend to run a ZFS storage pool on a virtual Solaris/OpenIndiana machine and/or expect the VM to use particularly heavy disk access then it is recommended that you PCI pass through the entire disk controller or host bus adapter, otherwise there are other layers (than IOMMU) through which you can pass through hard drive storage to the VMs.

Also KVM and libvirt have support for IOMMU and VirtualBox have added some IOMMU support on the Linux version. This functionality most likely comes from the KVM project. VMWare has only mentioned IOMMU support in their vSphere/ESXi products and VMDirectpath. I'm not aware of any other VMWare products that support IOMMU.

If you are looking for VGA passthrough, currently only Xen has support for it. The other hypervisors can only pass through regular single- and multifunction PCI devices.

A problem with most hardware today is that it generally doesn't say on the package or description whether it supports FLR or not. In Linux you can find out by the lspci -vv command; If the hardware has a "FLReset+" in the DevCap field then it supports FLR. If it has a "FLReset–" then it doesn't support FLR. Also, most hardware out there don't have FLR, it is actually pretty uncommon unfortunately. Perhaps matthew7 could provide some insights how he got around these hurdles. I'm impressed that he managed to pass through an AMD 5850.

Message edited by threehosts on 02-16-2012 at 07:49:25 AM

Reply to threehosts

Thank you very much threehosts! I think with the addition of that last piece of information I have what I need to give this project of mine some momentum. I've got in mind a cpu, chipset and board that support vt-d, I know to look for peripherals that support FLR ("function level reset" for people reading this as a summary) and I know of a couple of VMs that support vt-d/IOMMU. Please let me know if I'm overlooking anything...

When I finally build the computer I'll post my setup for people who are curious. From there I'll probably have questions setting up the software, but that's a little far off right now and I don't want to get ahead of myself...

Message quoted 1 times

Reply to Anto_di_Pretoro

Anto_di_Pretoro wrote :

Thank you very much threehosts! I think with the addition of that last piece of information I have what I need to give this project of mine some momentum. I've got in mind a cpu, chipset and board that support vt-d, I know to look for peripherals that support FLR ("function level reset" for people reading this as a summary) and I know of a couple of VMs that support vt-d/IOMMU. Please let me know if I'm overlooking anything...

When I finally build the computer I'll post my setup for people who are curious. From there I'll probably have questions setting up the software, but that's a little far off right now and I don't want to get ahead of myself...

What VM's do you have in mind that support vt-d(exept esxi-vsphere) ? Because the only one that I can find is parallel and even in that I can't get it to work!

Reply to saiva

Forum Business Computing : Virtualization and Unification Intel vt-d for home vm use

Cloud Computing

Virtualization and Unification

Security

Management